liteagents 2.15.2 → 2.17.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +196 -0
- package/README.md +8 -7
- package/package.json +2 -2
- package/packages/ampcode/AGENT.md +2 -2
- package/packages/ampcode/agents/quality-assurance.md +1 -1
- package/packages/ampcode/commands/docs-builder/docs-builder.cjs +2382 -0
- package/packages/ampcode/commands/docs-builder.md +787 -223
- package/packages/ampcode/commands/remember/AGENT_RULES.md +13 -43
- package/packages/ampcode/commands/remember/{friction.js → friction.cjs} +211 -99
- package/packages/ampcode/commands/remember.md +208 -29
- package/packages/claude/CLAUDE.md +3 -3
- package/packages/claude/commands/docs-builder/docs-builder.cjs +2382 -0
- package/packages/claude/commands/docs-builder.md +873 -0
- package/packages/claude/commands/remember/AGENT_RULES.md +10 -40
- package/packages/claude/commands/remember/{friction.js → friction.cjs} +208 -96
- package/packages/claude/commands/remember.md +208 -29
- package/packages/claude/plugins/live-canvas-marketplace/plugins/live-canvas-channel/package-lock.json +9 -9
- package/packages/droid/AGENTS.md +2 -2
- package/packages/droid/commands/docs-builder/docs-builder.cjs +2382 -0
- package/packages/droid/commands/docs-builder.md +787 -223
- package/packages/droid/commands/remember/AGENT_RULES.md +13 -43
- package/packages/droid/commands/remember/{friction.js → friction.cjs} +211 -99
- package/packages/droid/commands/remember.md +208 -29
- package/packages/droid/droids/quality-assurance.md +1 -1
- package/packages/opencode/AGENTS.md +2 -2
- package/packages/opencode/agent/quality-assurance.md +1 -1
- package/packages/opencode/command/docs-builder/docs-builder.cjs +2382 -0
- package/packages/opencode/command/docs-builder.md +787 -223
- package/packages/opencode/command/remember/AGENT_RULES.md +13 -43
- package/packages/opencode/command/remember/{friction.js → friction.cjs} +211 -99
- package/packages/opencode/command/remember.md +208 -29
- package/packages/opencode/opencode.jsonc +2 -6
- package/packages/subagentic-manual.md +31 -32
- package/packages/ampcode/commands/docs-builder/templates.md +0 -601
- package/packages/claude/skills/docs-builder/SKILL.md +0 -309
- package/packages/claude/skills/docs-builder/references/templates.md +0 -601
- package/packages/droid/commands/docs-builder/templates.md +0 -601
- package/packages/opencode/command/docs-builder/templates.md +0 -601
|
@@ -0,0 +1,873 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: docs-builder
|
|
3
|
+
description: Reorg a docs corpus, split an oversized doc, search it, keep pages current, index them
|
|
4
|
+
usage: /docs-builder [reorg | cleanup <file.md> | search <query words...>]
|
|
5
|
+
argument-hint: [reorg | cleanup <file.md> | search <query words...> — empty asks first run vs. drift]
|
|
6
|
+
allowed-tools: Read, Write, Edit, Grep, Glob, Task, AskUserQuestion, Bash(node:*), Bash(git:*), Bash(rg:*)
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
# docs-builder
|
|
10
|
+
|
|
11
|
+
Keep project docs **current, complete and findable**, and split a file when it outgrows
|
|
12
|
+
its row in `docs/README.md`.
|
|
13
|
+
|
|
14
|
+
> **The honest label: this does NOT make docs cheaper to read.**
|
|
15
|
+
> Measured four ways; best case is a tie with doing nothing. Cost tracks *findings*, not
|
|
16
|
+
> structure — better navigation raises how thorough an agent is willing to be, it does not
|
|
17
|
+
> cut reading. The winning arm wins because **synthesis caps cost**: the pages already did
|
|
18
|
+
> the reading. Never sell this as a token saving.
|
|
19
|
+
|
|
20
|
+
Every mechanical step is `docs-builder/docs-builder.cjs` (vanilla Node, zero deps). A model is used for
|
|
21
|
+
exactly **two** things: proposing themes, and writing pages. Bookkeeping done by a script is
|
|
22
|
+
100% correct; done by a model it was 27%.
|
|
23
|
+
|
|
24
|
+
### Model tiers — never hardcode a vendor model name
|
|
25
|
+
|
|
26
|
+
| step | tier | why |
|
|
27
|
+
|---|---|---|
|
|
28
|
+
| propose + assign themes | **cheapest tier** | structured labelling against a fixed list; no synthesis |
|
|
29
|
+
| write pages | **mid tier** | semantic synthesis, cheaper/faster than your top reasoning tier |
|
|
30
|
+
|
|
31
|
+
Use whatever your tool designates as that tier. The measured numbers below were taken on
|
|
32
|
+
Claude's Haiku 4.5 (cheap) and Sonnet 5 (mid) in August 2026 — the *ratios and shapes*
|
|
33
|
+
carry over, the absolute prices do not.
|
|
34
|
+
|
|
35
|
+
---
|
|
36
|
+
|
|
37
|
+
## Invocation
|
|
38
|
+
|
|
39
|
+
**With an argument** (`reorg`, `cleanup <file>`, or `search <query words...>`) — run that mode
|
|
40
|
+
directly, no question asked.
|
|
41
|
+
|
|
42
|
+
**Bare `/docs-builder`, no argument — ALWAYS ask, never auto-detect.** Run `due` first and
|
|
43
|
+
put its one-line verdict in the question text so the choice is informed. Then use
|
|
44
|
+
`AskUserQuestion`, one question, header `Mode`, exactly these two options:
|
|
45
|
+
|
|
46
|
+
> **Question: What should docs-builder do?**
|
|
47
|
+
>
|
|
48
|
+
> - **First run** — sort every `.md` in `docs/` into product / archive, then split anything
|
|
49
|
+
> too big into pages and index them. Use when docs are a pile of loose files, or
|
|
50
|
+
> docs-builder has never run here.
|
|
51
|
+
> - **Docs drift** — docs moved on since the last run: report what changed, rebuild the
|
|
52
|
+
> index, re-run lint. Nothing is restructured and nothing is split.
|
|
53
|
+
|
|
54
|
+
Do not offer a third option and do not recommend one. If `due` cannot run, say so plainly
|
|
55
|
+
and ask anyway — never guess the mode on the user's behalf.
|
|
56
|
+
|
|
57
|
+
Auto-detecting was considered and rejected: the two differ in cost (an unreviewed first-run
|
|
58
|
+
plan vs. a cheap drift check), and a wrong guess on the first one is expensive to unwind.
|
|
59
|
+
|
|
60
|
+
**`search` is a separate, explicit-argument-only mode — it is NOT a third bare-invocation
|
|
61
|
+
option.** The picker above stays at exactly two; do not add `search` to it. Typing
|
|
62
|
+
`/docs-builder search <query words...>` runs the mode directly (same rule as `reorg` and
|
|
63
|
+
`cleanup` above): it defaults the outline path to `docs/.docs-builder/outline.json` so the
|
|
64
|
+
user need only supply query words, and `N=` overrides the result count (default 10). It is
|
|
65
|
+
read-only — no model cost, no interview, nothing moves.
|
|
66
|
+
|
|
67
|
+
### What each option runs
|
|
68
|
+
|
|
69
|
+
**First run** — three steps, with the classification interview and a stop in between:
|
|
70
|
+
|
|
71
|
+
1. `discover` (Mode 0). Nothing moves. It writes `reorg-plan.json` — one row per file, each
|
|
72
|
+
carrying a mechanical **`suggested`** bucket + `reason` (a PRIOR, not a verdict), plus `h1`
|
|
73
|
+
and a short content `snip`, plus an `oversized` boolean (size no longer decides the
|
|
74
|
+
bucket). `bucket` itself starts **empty** on every row.
|
|
75
|
+
2. **The classification interview.** Feed the model the WHOLE plan table (file, h1, snip,
|
|
76
|
+
lines, suggested+reason) in one call and have it fill `bucket` for every row —
|
|
77
|
+
`product`/`logs`/`archive` — with a one-line reason: honour that a SHOUTED self-declared
|
|
78
|
+
status is near-conclusive for `archive` and that `suggested` is a prior, not an authority.
|
|
79
|
+
The model writes its answers straight into `reorg-plan.json`. Then show the user the full
|
|
80
|
+
resulting table via `AskUserQuestion` (approve all / correct specific rows / abort) — a
|
|
81
|
+
correction changes the plan file before anything moves.
|
|
82
|
+
3. `apply-reorg` moves every row, **oversized included** — size only decides whether a doc is
|
|
83
|
+
*splittable*, not whether it gets sorted. It refuses outright if any row's `bucket` is
|
|
84
|
+
still empty. Afterward it prints the oversized docs it just moved as a follow-up list,
|
|
85
|
+
`cleanup <NEW path> (N lines)`, `logs/` entries last. Show that list, then **ask which to
|
|
86
|
+
split** (any, all, none). Only then run `cleanup <file>` (Mode 1) on each chosen file —
|
|
87
|
+
`cleanup` itself prints the estimated split cost for that one file, then a mechanical
|
|
88
|
+
shape report, then stops for its own interview (Mode 1, step 1b) before anything else runs.
|
|
89
|
+
|
|
90
|
+
The two stops are deliberate and different. Step 2 guards *correctness* — the interview and
|
|
91
|
+
the user's approval, before a single file moves. Step 3's follow-up guards *cost* — splitting
|
|
92
|
+
is ~$0.39 per 1,000 source lines, and the user has seen neither the file list nor the number
|
|
93
|
+
when they pick "First run". Never split N files in one shot on an unseen list.
|
|
94
|
+
|
|
95
|
+
**Docs drift** — run bare `reorg` (Mode 2, below): its own `due`-style drift summary prints
|
|
96
|
+
first, if a ledger stamp exists, then it runs `discover`. If any row's `bucket` is still
|
|
97
|
+
empty (true on a genuine first run, or when new files appeared since the last classification),
|
|
98
|
+
`reorg` **stops right there** and prints what to do next — it never silently proceeds past an
|
|
99
|
+
unclassified plan. Once the plan is fully classified (an already-sorted corpus's re-run
|
|
100
|
+
carries its prior classifications forward automatically — see "Discover is idempotent"
|
|
101
|
+
below), `reorg` continues straight through `apply-reorg` → `lint`, no further stop, so
|
|
102
|
+
`index.md` and `lint.json` stay current. This is the common, cheap case for a corpus that is
|
|
103
|
+
already sorted: nothing new to classify, so the interview gate never fires.
|
|
104
|
+
|
|
105
|
+
---
|
|
106
|
+
|
|
107
|
+
## Modes
|
|
108
|
+
|
|
109
|
+
| Mode | Menu option | Does | Destructive |
|
|
110
|
+
|---|---|---|---|
|
|
111
|
+
| `/docs-builder reorg` (discover, classification interview, confirm, then apply-reorg) | *First run*, steps 1-3 | classify a WHOLE corpus into product/logs/archive | no (moves are `git mv`, plan classified and reviewed first) |
|
|
112
|
+
| `/docs-builder cleanup <file>` | *First run*, step 4 | measure ONE named oversized doc (cost, scan, heading shape) → **stops for the interview** | no (measure-only; original preserved) |
|
|
113
|
+
| `/docs-builder reorg` (bare `docs-builder.cjs reorg`) | *Docs drift* | due's drift summary (if a ledger stamp exists) + discover → (stops here if anything is still unclassified) → apply-reorg → lint, whole corpus | no |
|
|
114
|
+
| `/docs-builder search <query words...>` | *(none — explicit-argument mode only, never offered in the bare picker)* | BM25-rank sections of `docs/.docs-builder/outline.json` against the query, read-only | no |
|
|
115
|
+
|
|
116
|
+
`reorg` and `cleanup` solve different problems and compose: `reorg` sorts an entire messy
|
|
117
|
+
`docs/` tree into the four-bucket structure below in one pass and **never splits anything
|
|
118
|
+
itself**; an oversized file still moves into its bucket like everything else, but still needs
|
|
119
|
+
a human to run `cleanup <file>` individually (below), one named file per invocation, because
|
|
120
|
+
that step spends real model budget and should never fire without a look first. `cleanup` is
|
|
121
|
+
the ONLY entry point to the
|
|
122
|
+
split pipeline — it refuses more than one file at a time, refuses a missing/non-`.md`/
|
|
123
|
+
protected file, prints its cost estimate, then STOPS for an interview once the shape is
|
|
124
|
+
measured (Mode 1, step 1b); `cleanup-apply` (Mode 1, step 4) is the only door back in, and it
|
|
125
|
+
refuses to run until that interview has produced a `labels.json` with exactly one theme
|
|
126
|
+
marked `core: true`.
|
|
127
|
+
|
|
128
|
+
## Layout
|
|
129
|
+
|
|
130
|
+
```
|
|
131
|
+
docs/
|
|
132
|
+
README.md entry point, referenced from CLAUDE.md
|
|
133
|
+
index.md GENERATED by index-flat/apply-reorg/cleanup-apply. never hand-edited.
|
|
134
|
+
READER-FACING. The WHOLE-CORPUS map — the only file with a completeness
|
|
135
|
+
guarantee. ## Product, ## Logs, ## Archive.
|
|
136
|
+
log.md append-only: ## [DATE] operation | description — written by
|
|
137
|
+
`archive`, `apply-reorg`, `validate`, and `reorg`; NOT written by
|
|
138
|
+
read-only commands (`due`, `search`, `discover`).
|
|
139
|
+
product/ specs, designs, plans — the default. `apply-reorg` MOVES files here
|
|
140
|
+
(`git mv`); content is never rewritten.
|
|
141
|
+
logs/ pre-registrations, results, learnings, reports — historical, still
|
|
142
|
+
relevant. Same MOVE discipline as product/archive.
|
|
143
|
+
wiki/ synthesised pages, written by Mode 1 (`cleanup`)'s page writers.
|
|
144
|
+
archive/ what got cleaned up: self-declared dead. Originals are BYTE-FROZEN:
|
|
145
|
+
nothing under here is ever a rewrite target, so a doc lands byte-identical
|
|
146
|
+
to what it carried in (a clean R100 rename) and stays that way. Links
|
|
147
|
+
elsewhere POINTING AT it are still repaired. History via `git mv`.
|
|
148
|
+
Pruning is `git rm`, the user's own
|
|
149
|
+
call — nothing here does it automatically.
|
|
150
|
+
.docs-builder/ machine-only working state. Never hand-edited, never read by a human.
|
|
151
|
+
ledger.json last consolidation SHA + per-doc line counts
|
|
152
|
+
outline.json Layer 1 scan
|
|
153
|
+
cleanup-shape.json `cleanup`'s mechanical heading-shape report — the interview's proposal
|
|
154
|
+
is built from this, never a model guess at what a section is "about"
|
|
155
|
+
labels.json the model's theme assignment (`core: true` on exactly one theme)
|
|
156
|
+
reorg-plan.json `discover`'s plan (Mode 0): `suggested`+`reason` per row (the script's
|
|
157
|
+
mechanical PRIOR) plus `bucket` (empty until the classification
|
|
158
|
+
interview fills it — `apply-reorg` refuses to run while it's empty)
|
|
159
|
+
validate.json the gate's verdict
|
|
160
|
+
failures.json LIVE count of current `validate` gate failures, keyed
|
|
161
|
+
`<check>:<target>` — incremented on failure, DELETED the moment that
|
|
162
|
+
exact key passes again. Not a history; at 3+ recurrences `validate`
|
|
163
|
+
adds a STRUCTURAL warning line (message only, never the exit code).
|
|
164
|
+
lint.json latest lint proposals
|
|
165
|
+
tasks/ one task-<theme>.json per page
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
**`index.md` deliberately stays visible.** It is the thing a reader (or an agent) opens
|
|
169
|
+
first — the measured winning arm is *pages + a coarse index*. Hiding it under a dot-dir
|
|
170
|
+
would break the one mechanism that works. Only machine state goes in `.docs-builder/`.
|
|
171
|
+
|
|
172
|
+
**`index.md` has exactly one writer: `index-flat`** (called directly, or from
|
|
173
|
+
`apply-reorg`/`reorg`/`cleanup-apply`), and it is the corpus's ONLY index. This is
|
|
174
|
+
load-bearing, not a style choice — a real defect on bareloop is why: a second, themed
|
|
175
|
+
per-split index used to exist alongside it, and running a PRD split after a reorg silently
|
|
176
|
+
overwrote the 37-row whole-corpus map with that split's own 7-row view — 30 files vanished
|
|
177
|
+
from a file that still claimed completeness. Splitting the two apart into two files only
|
|
178
|
+
moved the problem (it then clobbered `outline.json` across concurrent splits instead), so
|
|
179
|
+
**the themed index was removed outright, 2026-08-24**. One index, rebuilt on every reorg and
|
|
180
|
+
after every split, is the whole design.
|
|
181
|
+
|
|
182
|
+
**Never moved — enforced in code, not just documented** (`PROTECTED_NAMES` / `walkMd`):
|
|
183
|
+
|
|
184
|
+
- **Files, at any depth:** `README.md`, `index.md`, `log.md`, `CHANGELOG.md`, `LICENSE.md`,
|
|
185
|
+
`CONTRIBUTING.md`, `CODE_OF_CONDUCT.md`, `SECURITY.md`, `CLAUDE.md`, `AGENTS.md`, `AGENT.md`.
|
|
186
|
+
Bare `LICENSE`/`NOTICE` have no `.md` extension, so the walker never sees them.
|
|
187
|
+
- **Directories:** every dot-dir (`.git/`, `.github/`, `.claude/`, `.factory/`, `.opencode/`,
|
|
188
|
+
`.amp/`, `.docs-builder/`) plus `node_modules/`, and the dirs reorg itself owns
|
|
189
|
+
(`product/`, `logs/`, `archive/`, `wiki/`) so a second run is idempotent.
|
|
190
|
+
|
|
191
|
+
---
|
|
192
|
+
|
|
193
|
+
## Mode 0 — reorg (a whole corpus, not one file)
|
|
194
|
+
|
|
195
|
+
The old skill (v1) did this job by handing an agent a file list and a prose rulebook
|
|
196
|
+
("KEEP/CONSOLIDATE/ARCHIVE", "when uncertain → ARCHIVE") and letting it read, judge and
|
|
197
|
+
`mv` every file itself — the exact shape that measured 27% correct on bookkeeping elsewhere
|
|
198
|
+
in this pipeline. `reorg` does the same JOB with the same discipline as everything else
|
|
199
|
+
here: classification is **mechanical and script-run**; nothing is guessed; nothing moves
|
|
200
|
+
until a plan has been written and reviewed.
|
|
201
|
+
|
|
202
|
+
**Not rebuilt:** v1's CONSOLIDATE (merging two docs' content into one). That rewrites
|
|
203
|
+
content, a different and higher-risk operation than anything measured so far. Descoped on
|
|
204
|
+
purpose, not silently dropped.
|
|
205
|
+
|
|
206
|
+
### 1. Discover (script) — enriches and PROPOSES, never classifies, never moves
|
|
207
|
+
|
|
208
|
+
```bash
|
|
209
|
+
REPO=<repo> node docs-builder/docs-builder.cjs discover # defaults to docs/
|
|
210
|
+
```
|
|
211
|
+
|
|
212
|
+
Recursively finds every `*.md` under the root (skipping `wiki/`, `logs/`, `archive/`,
|
|
213
|
+
`product/`, `.docs-builder/`, and the protected files), and for each one writes a row with:
|
|
214
|
+
|
|
215
|
+
- `h1` and a short `snip` (first ~200 chars of body, fence-masked) — reused straight from the
|
|
216
|
+
same `headings()`/`snippet()`/`fenceMask()` parsers `scan` uses, no second extraction path.
|
|
217
|
+
- `oversized` — a plain **boolean** (over the line ceiling, `OVERSIZED_LINES`, default 500 —
|
|
218
|
+
an UNMEASURED starting point). Size decides whether a doc is *splittable*, not whether it's
|
|
219
|
+
*sorted* — it is no longer a bucket.
|
|
220
|
+
- `suggested` + `reason` — a mechanical PRIOR, never a verdict:
|
|
221
|
+
|
|
222
|
+
| suggested | rule |
|
|
223
|
+
|---|---|
|
|
224
|
+
| `archive` | path already under `archive/old/reports/phases`, **or** the doc's own opening declares a SHOUTED status word (`CLOSED`, `DEPRECATED`, `SUPERSEDED`, `WITHDRAWN`, `RETRACTED`, `REFUTED`, `ARCHIVAL`, `ARCHIVED`), **or** the filename matches an archive-shaped prefix (`REPORT`, `STATUS`, `SUMMARY`, `FIX_`, `PHASE_`, `SPRINT_`, `DRAFT`, `WIP`, `OLD`, `TEMP` followed by `-` or `_`) |
|
|
225
|
+
| `logs` | filename carries an experiment-record token — `PREREG`, `LEARNINGS`, `REPORT`, `RESULTS`, `POSTMORTEM`, `RETRO` (case-sensitive, word-boundary, checked ONLY after the archive rules above, so a `REPORT-old.md` still reads as archive, not logs) |
|
|
226
|
+
| `product` | has an H1, no archive/logs signal — the default when nothing else applies |
|
|
227
|
+
| `product` | no H1, but an **include stub** — its whole non-blank content (≤3 lines) is nothing but include directives (mkdocs `--8<--`, `{% include %}`, `{{ .. }}`, `<!-- include -->`) and/or markdown links | a live pointer, not an unknown doc — real-world miss: uv's `docs/reference/contributing.md` |
|
|
228
|
+
| `product` | no H1 at all, and not an include stub — no strong signal either way; the interview decides, same as any other row |
|
|
229
|
+
|
|
230
|
+
- `bucket` — **empty on any row discover has not classified before** (see carry-forward
|
|
231
|
+
below; a re-run keeps a bucket the interview already set). This is the field the
|
|
232
|
+
classification interview (step 2) fills, and the ONLY field `apply-reorg` reads to decide
|
|
233
|
+
where a file goes. It is shown in the printed table, so a re-run confirms on screen that an
|
|
234
|
+
earlier classification is still in place.
|
|
235
|
+
|
|
236
|
+
**Discover is idempotent across re-runs, but not blind to prior work.** Re-running `discover`
|
|
237
|
+
carries an already-classified row's `bucket` FORWARD for any file it still sees at the same
|
|
238
|
+
path — it does not re-litigate a decision the interview already made. Only a file discover
|
|
239
|
+
has never classified before (new since the last run, or reappeared after a manual revert)
|
|
240
|
+
starts unclassified. `suggested`/`h1`/`snip`/`lines`/`oversized` are always freshly
|
|
241
|
+
recomputed, so the plan stays current even when `bucket` doesn't move. This is why bare
|
|
242
|
+
`reorg` (Mode 2) can compose discover with apply-reorg without a stop on an already-sorted
|
|
243
|
+
corpus: nothing new to classify, so its own gate never fires.
|
|
244
|
+
|
|
245
|
+
**Measured, why `logs` exists.** Run on bareloop's real `product/` (27 files): 11 were
|
|
246
|
+
experiment records (8 `*-PREREG`, 2 `*-LEARNINGS`, others) sitting alongside 14 actual specs
|
|
247
|
+
and designs — 41% of the bucket was run history, not product, which made the bucket useless
|
|
248
|
+
for finding specs. `logs/` is history that still matters (a prereg or a results doc), distinct
|
|
249
|
+
from `archive/`, which is history that is done.
|
|
250
|
+
|
|
251
|
+
**Why the status check requires SHOUTED caps, case-sensitively.** Tried case-insensitive
|
|
252
|
+
first, against a real, uncrafted corpus (not a fixture built to pass). It false-positived
|
|
253
|
+
three separate ways on real files: `"Supersedes **nothing**"` (negation), `"this rung
|
|
254
|
+
BUILDS three frozen records"` (an input being described, not the doc itself), `"archived
|
|
255
|
+
spines"` (data the doc references, not the doc). Same failure species as the lint fix
|
|
256
|
+
above — a word that means one thing in isolation matches unrelated prose. Restricting to
|
|
257
|
+
the ALL-CAPS form fixed every one of those, because this corpus's own writing convention
|
|
258
|
+
(observed, not designed around) SHOUTS a genuine self-declaration — `**Status: CLOSED**`,
|
|
259
|
+
`(ARCHIVAL 2026-07-25, before any number)` — while narrative mentions of the same word stay
|
|
260
|
+
lowercase or Title Case. `FROZEN` was in this list too, until 2026-08-23: measured against
|
|
261
|
+
bareloop's real docs corpus (37 files), 10 of its 12 `archive` calls were false positives,
|
|
262
|
+
all from `FROZEN` — e.g. `2026-08-01-layer-3-reuse-design.md` says "design FROZEN... build
|
|
263
|
+
follows this record" and `TYPES-PREREG.md` says "FROZEN before any model token is spent" —
|
|
264
|
+
in that corpus's own convention FROZEN means "locked, do not edit, still current," not
|
|
265
|
+
"retired." That's the one failure this design promises never to make, so the word was
|
|
266
|
+
dropped with no replacement heuristic. Traded away: 2 real misses (`"Frozen 2026-07-26"`,
|
|
267
|
+
`"job #4 ... (frozen)"`) — consistent with precision-over-recall. Neither miss is
|
|
268
|
+
dangerous: a miss just lands the doc in `product`, one bucket short of ideal, not
|
|
269
|
+
mis-archived.
|
|
270
|
+
|
|
271
|
+
**Also dropped from v1's own heuristics, on the same evidence standard:** "filename has a
|
|
272
|
+
date → likely stale." Tested against a real corpus and wrong — `2026-07-28-p-palette-
|
|
273
|
+
design.md` is a current, locked, actively-built spec, not a stale report. A dated filename
|
|
274
|
+
alone proves nothing.
|
|
275
|
+
|
|
276
|
+
Output: `docs/.docs-builder/reorg-plan.json`, plus a printed table. **Nothing has moved
|
|
277
|
+
yet, and nothing has been classified yet either.**
|
|
278
|
+
|
|
279
|
+
### 2. The classification interview — the model classifies, behind an approval gate
|
|
280
|
+
|
|
281
|
+
This is deliberately the model's job, not a rule's. The FROZEN incident (above) is usually
|
|
282
|
+
read as proof a model must not classify — that's the wrong lesson. FROZEN was a *mechanical
|
|
283
|
+
rule*, and it did damage precisely because it moved files with no gate at all. The failure
|
|
284
|
+
was the silent move, not the judgement.
|
|
285
|
+
|
|
286
|
+
1. Read `docs/.docs-builder/reorg-plan.json`. Feed the model the WHOLE table — `file`, `h1`,
|
|
287
|
+
`snip`, `lines`, `suggested`+`reason` — **in one call**, and have it fill `bucket` for
|
|
288
|
+
every row (`product`/`logs`/`archive`) with a one-line reason. `suggested` is a PRIOR the
|
|
289
|
+
model is shown, never an authority over it — but a SHOUTED self-declared status
|
|
290
|
+
(`**Status: CLOSED**`) is near-conclusive for `archive` regardless of what the mechanical
|
|
291
|
+
prior says.
|
|
292
|
+
2. Have the model write its answers straight into `reorg-plan.json`'s `bucket` fields.
|
|
293
|
+
3. Show the user the full resulting table via `AskUserQuestion` — approve all / correct
|
|
294
|
+
specific rows / abort. A correction changes the plan file.
|
|
295
|
+
|
|
296
|
+
**Show EXACTLY these four columns, in this order.** No extras, no prose padding — the
|
|
297
|
+
operator is scanning for a row that looks wrong, and every extra column hides it:
|
|
298
|
+
|
|
299
|
+
| file | lines | → destination | why |
|
|
300
|
+
|
|
301
|
+
- **`file`** — the doc's CURRENT path.
|
|
302
|
+
- **`lines`** — the plan's line count. Append `(oversized)` when the row is oversized.
|
|
303
|
+
- **`→ destination`** — the **full destination PATH** this row will move to, e.g.
|
|
304
|
+
`docs/archive/PRD.md` — **never the bare bucket word** (`archive`). A wrong destination
|
|
305
|
+
is obvious in a path and easy to skim past in a single word. This is the column the
|
|
306
|
+
operator is actually approving.
|
|
307
|
+
- **`why`** — the model's one-line reason, trimmed to one line.
|
|
308
|
+
|
|
309
|
+
**Sort the rows by destination**, so all `archive` rows sit together, then `logs`, then
|
|
310
|
+
`product`. A misfiled doc is easiest to spot against its neighbours; scattered through a
|
|
311
|
+
path-sorted list it reads as normal.
|
|
312
|
+
|
|
313
|
+
Only after approval does `apply-reorg` run. The approval gate, not the classifier's
|
|
314
|
+
mechanism, is the safety property here — and it is strictly stronger than a rule that moves
|
|
315
|
+
files with no gate at all.
|
|
316
|
+
|
|
317
|
+
### 3. Apply (script) — an ALREADY-CLASSIFIED plan, verified moves, survives a bad file
|
|
318
|
+
|
|
319
|
+
```bash
|
|
320
|
+
node docs-builder/docs-builder.cjs apply-reorg # defaults to the plan above
|
|
321
|
+
```
|
|
322
|
+
|
|
323
|
+
**Refuses outright if any row's `bucket` is still empty** — the interview-has-not-happened
|
|
324
|
+
message, not a crash — so nothing can move on an unreviewed plan. A plan from before this
|
|
325
|
+
version (`bucket: 'oversized'` or `'review'`, both gone from the schema) is refused too, with
|
|
326
|
+
a pointer to re-run `discover`.
|
|
327
|
+
|
|
328
|
+
- `product` → verified `git mv` to `docs/product/<basename>`
|
|
329
|
+
- `logs` → verified `git mv` to `docs/logs/<basename>`
|
|
330
|
+
- `archive` → verified `git mv` to `docs/archive/<basename>`
|
|
331
|
+
- **Oversized files move too — size decides splittable, not sorted.** No bucket is exempt.
|
|
332
|
+
After the move, every oversized row is printed as a follow-up list at its NEW path, one
|
|
333
|
+
`cleanup <path> (N lines)` line per file — run `cleanup` (Mode 1, below) on each, by hand,
|
|
334
|
+
one file at a time. The list is **ordered, `logs/` entries last**: a pre-registration is a
|
|
335
|
+
legitimate split target but rarely the best NEXT one — a prereg is a record of one
|
|
336
|
+
experiment, meant to be read whole. Auto-splitting N unknown files in one shot would spend
|
|
337
|
+
real model money with no confirmation; the pipeline never does that unprompted, and
|
|
338
|
+
`cleanup` itself refuses to run on more than one file.
|
|
339
|
+
- **After the scan, `apply-reorg` writes `docs/index.md` itself** — it calls `index-flat`
|
|
340
|
+
(see below) automatically, so a reorg-only corpus ends up indexed without a second command.
|
|
341
|
+
Runs every time, unconditionally.
|
|
342
|
+
- **`apply-reorg` also writes the docs pointer into `CLAUDE.md`** — a marker-wrapped
|
|
343
|
+
`<!-- DOCS_INDEX:START -->`/`<!-- DOCS_INDEX:END -->` block naming `docs/index.md` as a
|
|
344
|
+
**plain path, never an `@`-reference**: hot-loading a 100-row index into every session is
|
|
345
|
+
exactly what this avoids. The block also carries the `/docs-builder search` hint, and is
|
|
346
|
+
static — it never varies with row count, so a re-run rewrites identical bytes. Idempotent:
|
|
347
|
+
an existing block is replaced in place, never duplicated; other content is left alone.
|
|
348
|
+
The target is `CONFIG=` (default `CLAUDE.md`); this package uses `CONFIG=CLAUDE.md`.
|
|
349
|
+
- **The moves land STAGED in your git index — commit them promptly.** `git mv` stages each
|
|
350
|
+
rename immediately (that is what preserves history), so when `apply-reorg` returns the repo
|
|
351
|
+
is holding N staged renames. Any other session's `git add -A` or `git commit -a` will absorb
|
|
352
|
+
them into an unrelated commit — OBSERVED TWICE, in two different repos. `apply-reorg` prints
|
|
353
|
+
a closing advisory naming the counts and a copy-pasteable recipe. **Run that recipe
|
|
354
|
+
VERBATIM. Do NOT hand-edit it, and do NOT stage by hand instead.** If it looks incomplete
|
|
355
|
+
or names a path that errors, that is a BUG in the recipe — stop and report it to the user;
|
|
356
|
+
do not silently repair it and move on. OBSERVED, real (privcloud first field run): the
|
|
357
|
+
recipe omitted `docs/log.md`, the operator quietly added it by hand, and the bug only
|
|
358
|
+
surfaced because they were later asked for near-misses — a silent repair is a lost bug
|
|
359
|
+
report. Do NOT scope the commit to `docs` alone either: the renames are staged, but the
|
|
360
|
+
inbound-link rewrites are UNSTAGED and reach outside `docs/` (`src/`, `scripts/`, `tests/`,
|
|
361
|
+
`README.md`). Both belong in ONE commit, or you ship moved files whose links were never
|
|
362
|
+
repaired. The tool never auto-commits, by design.
|
|
363
|
+
- A basename collision (two files, same name, different original folders) is
|
|
364
|
+
disambiguated (`-2`, `-3`, …); a collision with a **file that already exists at the
|
|
365
|
+
destination** is skipped, logged, and does not stop the rest of the run.
|
|
366
|
+
- **Emptied source directories are removed.** Once a directory the run itself moved files
|
|
367
|
+
OUT of is genuinely empty (no stray files, no dotfiles), it's removed — depth-first, so a
|
|
368
|
+
nested empty (`docs/00-context/sub/`, then `docs/00-context/` once `sub/` is gone) collapses
|
|
369
|
+
in the same pass. A directory that still holds ANYTHING — even a file reorg never
|
|
370
|
+
touches — is never removed. Only directories THIS run emptied are candidates; a dir that
|
|
371
|
+
happened to already be empty before this run started is not this tool's to remove.
|
|
372
|
+
- **After every move, `apply-reorg` re-scans the whole corpus** — `docs/product/`,
|
|
373
|
+
`docs/logs/`, and `docs/archive/` all — straight into `outline.json`, the database `search`
|
|
374
|
+
reads. Not a hint, not opt-in: it runs every time, even when nothing moved this run (e.g.
|
|
375
|
+
re-running on a corpus already sorted from a previous pass). Measured bug this closes: on a
|
|
376
|
+
real 37-doc corpus, `outline.json` used to hold records for only the 12 files a split had
|
|
377
|
+
happened to touch — all 24 `docs/product/` files had zero records, so `search` was
|
|
378
|
+
structurally blind to them. Runs after the move, not before (moving changes paths, not
|
|
379
|
+
content, so a pre-move scan would just be redone), and reuses the same `scan` used
|
|
380
|
+
everywhere else in this pipeline — no second scanner, no second outline format.
|
|
381
|
+
|
|
382
|
+
**After each move it repairs the paths that move just broke** — the whole point of doing this
|
|
383
|
+
in a script. Both movers (`apply-reorg` and `archive`) go through ONE function, `moveDoc`, so
|
|
384
|
+
the follow-up list cannot be added to one and missed by the other; that exact miss shipped
|
|
385
|
+
three times. `moveDoc` prints nothing — each caller reports in its own format — and it throws
|
|
386
|
+
only if the MOVE failed. A failed follow-up is collected instead, so a moved file is never
|
|
387
|
+
reported as a file that needs re-moving. Two follow-ups:
|
|
388
|
+
|
|
389
|
+
1. **Pipeline artifacts** — `rewriteArchivedPath` syncs `outline.json` / `labels.json`
|
|
390
|
+
(`records[].file`, and the `<file> :: ` prefix inside every key). This is the same
|
|
391
|
+
function `archive` calls; `apply-reorg` used to bypass it, which silently invalidated
|
|
392
|
+
every key of every file it moved. Both now reach it through `moveDoc`.
|
|
393
|
+
2. **Inbound links** — every git-tracked `.md`/`.js`/`.cjs`/`.mjs`/`.json`/`.yml` file that
|
|
394
|
+
points at the old path (repo-rooted, e.g. `docs/GUIDE.md`) is rewritten to the new one.
|
|
395
|
+
In `.md` files specifically, a RELATIVE link is also caught: inside actual markdown link
|
|
396
|
+
syntax only (`[text](../concepts/x.md)` or a reference-style `[label]: ./tools.md`), never
|
|
397
|
+
bare prose, the target is resolved against the SCANNING file's own directory, and — if it
|
|
398
|
+
points at the file that just moved — rewritten to the correct relative path to its new
|
|
399
|
+
location, with `#fragment` preserved and a `./` prefix kept only if the original had one.
|
|
400
|
+
This is what fixes real corpora (e.g. astral-sh/uv) that cross-link with `../x.md`-style
|
|
401
|
+
paths instead of repo-rooted ones. The file that just moved also gets its OWN relative
|
|
402
|
+
links re-based from its new directory, so a link whose SOURCE and TARGET both move in the
|
|
403
|
+
same `apply-reorg` run still resolves regardless of which one moves first. Reported per
|
|
404
|
+
file, counted in the summary, and recorded in `log.md`.
|
|
405
|
+
|
|
406
|
+
**Why rewriting is safe here when the dangling-reference *lint* was cut outright.** That lint
|
|
407
|
+
had to **infer** whether `P95` was a reference (1/27 precision). This infers nothing:
|
|
408
|
+
`apply-reorg` is holding the old path and the new path in a variable at the instant it breaks
|
|
409
|
+
the link. The match is exact and anchored — a lookbehind rejects `xdocs/A.md` and
|
|
410
|
+
`./docs/A.md`, a lookahead rejects `docs/A.md.bak` and `docs/A.md-old`, while a sentence-final
|
|
411
|
+
`docs/A.md.` still matches. A plain substring replace corrupts all four.
|
|
412
|
+
|
|
413
|
+
**Never rewritten:** any file whose BASENAME is `CHANGELOG.md` or `log.md`, at ANY depth — not
|
|
414
|
+
only the root `CHANGELOG.md` / `docs/log.md`. Both are append-only history — a record of where
|
|
415
|
+
a file *was* is not a broken link.
|
|
416
|
+
|
|
417
|
+
**Archive is frozen — one rule, two directions.** Nothing resident under `docs/archive/` is
|
|
418
|
+
ever a rewrite TARGET (its bytes are never touched, stale links and all), but links ELSEWHERE
|
|
419
|
+
that POINT AT an archived file ARE still repaired. Same rationale as `CHANGELOG.md`/`log.md`,
|
|
420
|
+
one directory further: an archived doc is a historical record, and a record of where a file
|
|
421
|
+
*was* is not a broken link. This is evaluated against a row's **destination**, not its current
|
|
422
|
+
location — a reorg fills the archive, so a doc bound for `archive/` is exempt from the run's
|
|
423
|
+
FIRST rewrite, not from whenever it happens to move. Without that it would be edited by an
|
|
424
|
+
earlier row's sweep and carry the edit in with it (measured: it did). The check lives in one
|
|
425
|
+
predicate, `isRewriteExempt`, at one call site. `docs/.docs-builder/` is excluded too; item 1 owns it.
|
|
426
|
+
|
|
427
|
+
**A known, deliberate trade-off: this is a literal exact-path match over raw file bytes, not
|
|
428
|
+
fence-aware or context-aware.** It rewrites every exact, word-bounded occurrence of the old path
|
|
429
|
+
in every git-tracked `.md`/`.js`/`.cjs`/`.mjs`/`.json`/`.yml` file (except the two exemptions
|
|
430
|
+
above) — inside a code fence, inside a sentence describing history ("this used to live at
|
|
431
|
+
docs/OLD.md"), anywhere. A prose mention of where a file *used to be* WILL be rewritten to say
|
|
432
|
+
where it is now, changing what the sentence says. This is intentional, not an oversight: a dead
|
|
433
|
+
link is worse than a reworded sentence, the match is exact rather than inferred (unlike the
|
|
434
|
+
dangling-reference *lint*, which infers and was cut outright at 1/27 precision), and every
|
|
435
|
+
rewrite is printed per file so it is visible, never silent.
|
|
436
|
+
|
|
437
|
+
---
|
|
438
|
+
|
|
439
|
+
## Mode 1 — cleanup
|
|
440
|
+
|
|
441
|
+
**`cleanup <file.md>` is the ONLY entry point to the split pipeline, and it is a MEASURE step
|
|
442
|
+
only.** Settled 2026-08-23 (`docs-builder-v3-spec.md`, "cleanup"): the original always ends
|
|
443
|
+
up in `docs/archive/` via `git mv`, **byte-identical** — the archive is frozen and the
|
|
444
|
+
link rewriter never edits anything resident there (see "Archive is frozen" below) — and
|
|
445
|
+
everything the split
|
|
446
|
+
produces is a **new** file, including the core (the theme the document is mainly about, which
|
|
447
|
+
keeps the original's basename — see step 2a). Nothing in this mode ever rewrites a source
|
|
448
|
+
document in place.
|
|
449
|
+
|
|
450
|
+
`cleanup` takes exactly one named file — never zero, never more than one — and refuses
|
|
451
|
+
cleanly (exit non-zero) if the file doesn't exist, isn't a `.md`, or is one of the protected
|
|
452
|
+
entry-point docs (README, CLAUDE.md, etc.). It prints the file's line count and an estimated
|
|
453
|
+
write cost (the same cost law `plan` uses in step 4 below, priced as a 1-page floor since the
|
|
454
|
+
real page count isn't known until the model groups sections in step 2), runs step 1 (scan)
|
|
455
|
+
below for you, then measures the document's heading shape and **stops**:
|
|
456
|
+
|
|
457
|
+
```bash
|
|
458
|
+
REPO=<repo> node docs-builder/docs-builder.cjs cleanup docs/BIG.md
|
|
459
|
+
```
|
|
460
|
+
|
|
461
|
+
**Nothing past this command runs until a human has answered the interview (step 1b) below.**
|
|
462
|
+
Not the archive move, not a page, not a model call beyond the scan `cleanup` already ran. The
|
|
463
|
+
remaining steps — the interview, proposing/assigning themes, validating, planning, writing
|
|
464
|
+
pages, archiving the original, and indexing — are driven by this file, not by the script:
|
|
465
|
+
splitting spends real model budget, so nothing past the shape report runs without a human
|
|
466
|
+
choosing to continue, and confirming, the themes.
|
|
467
|
+
|
|
468
|
+
### 1. Scan (script) — run automatically by `cleanup`, shown here for what it produces
|
|
469
|
+
|
|
470
|
+
```bash
|
|
471
|
+
REPO=<repo> OUT=docs/.docs-builder/outline.json \
|
|
472
|
+
node docs-builder/docs-builder.cjs scan docs/BIG.md
|
|
473
|
+
```
|
|
474
|
+
|
|
475
|
+
One record per H2, each carrying the doc's H1 identity, a 2-line snippet, every H3 **with
|
|
476
|
+
its own start/end** (so a page writer can read a sub-section alone), and a `key`.
|
|
477
|
+
|
|
478
|
+
**Key format is always `<file> :: <heading>`, regardless of how many files were scanned
|
|
479
|
+
together.** It used to drop the `<file> ::` prefix when scanning a single file, so the same
|
|
480
|
+
heading keyed differently depending on scan batch size — a `labels.json` made from a
|
|
481
|
+
single-doc cleanup silently stopped matching once the file was rescanned as part of a
|
|
482
|
+
corpus-wide reorg. Fixed; this is a **one-time breaking change** — any `labels.json`
|
|
483
|
+
made under the old bare-key format (no `<file> ::` prefix) will no longer match and must be
|
|
484
|
+
regenerated.
|
|
485
|
+
|
|
486
|
+
### 1b. The interview — a proposal from a read, a verdict from the user
|
|
487
|
+
|
|
488
|
+
`cleanup` also writes `docs/.docs-builder/cleanup-shape.json` and prints it as a table: the
|
|
489
|
+
document's H2 sections grouped **mechanically** by shared heading shape (e.g. `86 sections:
|
|
490
|
+
"§N ..." — 11 sections, 193 lines (3%); "Addendum vN.N ..." — 75 sections, 5,476 lines
|
|
491
|
+
(97%)`). This is measurement, not a guess — the script groups on heading text alone, never on
|
|
492
|
+
what a section is *about*.
|
|
493
|
+
|
|
494
|
+
Now read the document yourself — the cheapest tier is enough, and the read is weighted to the
|
|
495
|
+
**opening**, where a document states its intent and what it is for, but covers enough of the
|
|
496
|
+
rest to name the other themes actually present. Then ask, via `AskUserQuestion`, one question,
|
|
497
|
+
in exactly this shape:
|
|
498
|
+
|
|
499
|
+
> **Question: Is this split right?**
|
|
500
|
+
>
|
|
501
|
+
> <the shape table `cleanup` printed>
|
|
502
|
+
>
|
|
503
|
+
> This document is mainly: **\<your read of the dominant theme>**.
|
|
504
|
+
> Other themes present: **\<theme>**, **\<theme>**, ...
|
|
505
|
+
>
|
|
506
|
+
> Is that right, and are those the themes you want split out?
|
|
507
|
+
|
|
508
|
+
At minimum two options: **Confirm** (proceed with the themes exactly as stated) and
|
|
509
|
+
**Correct** (the user names what the document is actually mainly about, and/or edits the
|
|
510
|
+
other-themes list). **This must not degenerate into auto-detect wearing a costume** — the
|
|
511
|
+
failure mode `docs-builder-v3-spec.md` names by name. A correction has to change what gets
|
|
512
|
+
built: if the user corrects the "mainly" answer, that theme — not your first guess — is the
|
|
513
|
+
one step 2a marks `core: true`; if they edit the other-themes list, that list — not your first
|
|
514
|
+
guess — is the fixed list step 2a proposes against. Do not run step 2a until this question is
|
|
515
|
+
answered.
|
|
516
|
+
|
|
517
|
+
### 2a. Propose + assign themes — **cheapest tier, ONE call over ALL headings**
|
|
518
|
+
|
|
519
|
+
Feed every `records[].key` plus its `snip`, together with the interview's confirmed-or-
|
|
520
|
+
corrected answer from step 1b. Ask for a fixed list of themes with a one-line gloss each —
|
|
521
|
+
the theme the interview named as "mainly" goes in as the **core** theme, everything else as
|
|
522
|
+
an ordinary theme. Aim for a list that leaves no theme holding more than ~30% of the lines.
|
|
523
|
+
|
|
524
|
+
**Exactly one theme is core**, marked `core: true` in `labels.json` (step 2b below). Its page
|
|
525
|
+
will carry the source file's own basename (e.g. splitting `docs/01-product/PRD.md` yields a
|
|
526
|
+
core page named `PRD.md`, not a slugified theme name) — `plan` (step 4) enforces this, and
|
|
527
|
+
rejects a `labels.json` that marks more than one theme core.
|
|
528
|
+
|
|
529
|
+
**This pass is load-bearing.** Skipping it and assigning directly gave 68 themes for 86
|
|
530
|
+
sections, 57 of them singletons — perfect key accuracy, useless grouping. Chunking fixes
|
|
531
|
+
misalignment; only *global sight* fixes convergence. They are two different jobs.
|
|
532
|
+
|
|
533
|
+
### 2b. Assign — **cheapest tier, chunks of ~20 sections**
|
|
534
|
+
|
|
535
|
+
Each section gets a theme **from that fixed list**. Five rules, all paid for:
|
|
536
|
+
|
|
537
|
+
1. **Never emit a positional index.** One call over 97 sections keyed on `{index, id}`
|
|
538
|
+
dropped 1 section, shifted 41 IDs from #55 on, and emitted one heading twice.
|
|
539
|
+
2. **Echo `records[].key` back verbatim** — and **delimit it explicitly in the prompt** so
|
|
540
|
+
the model can see where the key ends:
|
|
541
|
+
|
|
542
|
+
```
|
|
543
|
+
<<<KEY>>>the exact key text<<<END>>>
|
|
544
|
+
snippet text on the following lines
|
|
545
|
+
```
|
|
546
|
+
|
|
547
|
+
Measured the hard way: a bare `KEY: <text>` line followed by the snippet made the model
|
|
548
|
+
glue snippet text onto 6 of 86 keys, because a key truncated mid-sentence has no visible
|
|
549
|
+
end. The script also trims the key, because a truncation landing on a space produces a
|
|
550
|
+
trailing space no model will echo back (5 more failures). Never ask a model to reproduce
|
|
551
|
+
a boundary it cannot see.
|
|
552
|
+
3. **Chunk to ~20.** (40–50 may be cheaper — untested, see Open.)
|
|
553
|
+
4. **Validate before use** (step 3). Script, not model.
|
|
554
|
+
5. **Propose globally before assigning** (2a).
|
|
555
|
+
|
|
556
|
+
Write the result as `{ "themes": [{name, gloss, core?}], "labels": [{key, theme}] }` — `core:
|
|
557
|
+
true` on exactly the one theme step 1b's interview settled on as "mainly"; omit it (or leave
|
|
558
|
+
it `false`) on every other theme.
|
|
559
|
+
|
|
560
|
+
### 3. Validate (script) — **hard gate, exits 1 on failure**
|
|
561
|
+
|
|
562
|
+
```bash
|
|
563
|
+
REPO=<repo> node docs-builder/docs-builder.cjs validate \
|
|
564
|
+
docs/.docs-builder/{outline,labels}.json
|
|
565
|
+
```
|
|
566
|
+
|
|
567
|
+
`REPO=` must be the SAME repo `scan` used — the new `paths`/`links`/`citations` checks
|
|
568
|
+
resolve source files and `docs/wiki/` pages against it, so a mismatched `REPO` fails every
|
|
569
|
+
file at once. JSON artifacts stay cwd-relative, as everywhere else in the pipeline.
|
|
570
|
+
|
|
571
|
+
Checks every key exists, appears exactly once, none invented, none off-list, and reports
|
|
572
|
+
lines covered vs total. Also gates on: every outline record's source file still existing
|
|
573
|
+
(`paths`), every `wiki/*.md` link inside `index.md` resolving (`links`), and every page
|
|
574
|
+
citation landing inside its own task's source ranges (`citations`). **Uncited sections are
|
|
575
|
+
reported but never block** — flagged for a human, not a failure. **Do not proceed on FAIL**
|
|
576
|
+
— re-run the failing chunk.
|
|
577
|
+
|
|
578
|
+
The `links` check reads `INDEX` (default `docs/index.md`, the one index `index-flat`
|
|
579
|
+
writes) rather than a hardcoded path — set it if the index lives somewhere else in this
|
|
580
|
+
repo. `TASKS` (default `docs/.docs-builder/tasks`) likewise overrides where the `citations`
|
|
581
|
+
check looks for the per-page task files. It checks EVERY relative `.md` link in that index —
|
|
582
|
+
product/, logs/, archive/ and pages alike, resolved from `INDEX`'s own directory. (It used to
|
|
583
|
+
scope itself to `PAGES`-prefixed links only; that scoping existed for the themed per-split
|
|
584
|
+
index, which is gone — on the one whole-corpus index it would silently skip most rows.)
|
|
585
|
+
|
|
586
|
+
### 4. Plan + apply (script) — `cleanup-apply`, the door back in after the interview
|
|
587
|
+
|
|
588
|
+
```bash
|
|
589
|
+
REPO=<repo> node docs-builder/docs-builder.cjs cleanup-apply docs/BIG.md \
|
|
590
|
+
docs/.docs-builder/outline.json docs/.docs-builder/labels.json
|
|
591
|
+
```
|
|
592
|
+
|
|
593
|
+
This is the first script command allowed to run after step 1b, and it **refuses outright** —
|
|
594
|
+
before doing anything — if `labels.json` is missing, or has no theme marked `core: true`: in
|
|
595
|
+
both cases it prints that the interview has not happened yet and stops. Once that gate
|
|
596
|
+
passes, it runs `plan` (below), which writes `docs/.docs-builder/tasks/task-<theme>.json` per
|
|
597
|
+
page (the core theme's task file named after the original basename, per step 2a) and prints
|
|
598
|
+
an estimated write cost for the pages **still to write**.
|
|
599
|
+
|
|
600
|
+
If any page is still to write, `cleanup-apply` **stops there** — go to step 5 and write them,
|
|
601
|
+
then re-run the exact same `cleanup-apply` command. It is deliberately re-runnable: a human/
|
|
602
|
+
model page-writing step sits between planning and archiving that the script cannot run
|
|
603
|
+
itself, so nothing is auto-chained across that gap. Once **every** page exists, that same
|
|
604
|
+
re-run archives the original (step 6), relocates the core page into the original document's
|
|
605
|
+
own directory, and rebuilds the WHOLE-CORPUS map (step 8, `index-flat` → `docs/index.md`) —
|
|
606
|
+
all in one call, no separate `archive`/`index-flat` invocation needed, though both remain
|
|
607
|
+
runnable standalone (their own sections below still apply if you ever need to run either by
|
|
608
|
+
hand). The rebuild matters: archiving just moved a file, the core page just landed at the
|
|
609
|
+
original's path, and the new pages just appeared under `PAGES` — all three are corpus changes
|
|
610
|
+
`docs/index.md` must reflect.
|
|
611
|
+
|
|
612
|
+
`plan` is the underlying resume mechanism, unchanged: any theme whose page already exists in
|
|
613
|
+
`docs/wiki/` (override with `PAGES=`) is reported `done` and dropped from the estimate, so a
|
|
614
|
+
crash or an early `cleanup-apply` stop relaunches only what is missing. Each finished page is
|
|
615
|
+
the checkpoint; there is no separate state file to go stale. It is also still runnable on its
|
|
616
|
+
own:
|
|
617
|
+
|
|
618
|
+
```bash
|
|
619
|
+
REPO=<repo> OUT=docs/.docs-builder/tasks node docs-builder/docs-builder.cjs plan \
|
|
620
|
+
docs/.docs-builder/{outline,labels}.json
|
|
621
|
+
```
|
|
622
|
+
|
|
623
|
+
### 5. Write pages — **mid tier, one agent per page**
|
|
624
|
+
|
|
625
|
+
Each agent reads **only its own line ranges**. The value is context isolation.
|
|
626
|
+
|
|
627
|
+
- **250 lines is a ceiling, never a target.** Measured: every page came in under it
|
|
628
|
+
unprompted, and agents said padding would be filler.
|
|
629
|
+
- **Coherence decides grouping, not size.**
|
|
630
|
+
- **Every claim carries a line citation** back to the source file, and never to a line
|
|
631
|
+
outside the page's own ranges. Measured: 294 citations, 0 bad, 0 out of scope.
|
|
632
|
+
Cite as `(<file>:<start>-<end>)` — e.g. `(CYBERNETICS.md:262-268)`. `validate` checks this
|
|
633
|
+
(step 3).
|
|
634
|
+
- **A page counts as written only with YAML frontmatter and at least 10 lines.** This is a
|
|
635
|
+
mechanical gate, not a style note: `plan`/`cleanup-apply` call anything short of it
|
|
636
|
+
`PARTIAL` and rewrite it. MEASURED, real (bareagent field run): this criterion was
|
|
637
|
+
documented only in the lint section, never in the brief the page-writing agents actually
|
|
638
|
+
read — so the first wave produced PARTIAL pages and had to be redone. Put it in every
|
|
639
|
+
writer's prompt.
|
|
640
|
+
- **The CORE page goes back to the original document's own directory**, keeping the
|
|
641
|
+
original's basename — only non-core theme pages stay under `PAGES`. Write it under `PAGES`
|
|
642
|
+
like the rest (the original still occupies its final path until step 6 archives it);
|
|
643
|
+
`cleanup-apply` relocates it for you once the archive move frees that path.
|
|
644
|
+
- **Launch 3 at a time.** Each finished page in `docs/wiki/` is its own checkpoint — re-run
|
|
645
|
+
`plan` and it reports what is left. A cleanup that dies halfway and cannot resume is worse
|
|
646
|
+
than a slow one.
|
|
647
|
+
- **Exit condition is a command, not a judgement: re-run `plan` and read its output.** Step 5
|
|
648
|
+
is done ONLY when it prints `all pages written` with **zero** `WARN ... PARTIAL` lines. Any
|
|
649
|
+
PARTIAL page gets rewritten before moving on — do not proceed to `cleanup-apply` past a
|
|
650
|
+
PARTIAL warning, and do not decide by eye that a page "looks done". The criterion above is
|
|
651
|
+
enforced by `pageStatus()`; this re-run is how you invoke that enforcement.
|
|
652
|
+
|
|
653
|
+
### 6. Archive the original (script) — run for you by `cleanup-apply` once all pages exist
|
|
654
|
+
|
|
655
|
+
```bash
|
|
656
|
+
REPO=<repo> node docs-builder/docs-builder.cjs archive docs/BIG.md
|
|
657
|
+
```
|
|
658
|
+
|
|
659
|
+
A **verified move**, not a copy: hash → `git mv` (so history follows) → hash again →
|
|
660
|
+
confirm the old path is empty. The original is never rewritten and never edited, but it
|
|
661
|
+
does not stay where it was either — otherwise the same content sits in three places at
|
|
662
|
+
once (old path, archive, and the new pages), which is duplication, not cleanup.
|
|
663
|
+
|
|
664
|
+
Archiving is also what frees the original's path for the **core page**, which `cleanup-apply`
|
|
665
|
+
then relocates there from `PAGES` (settled 2026-08-24). Splitting is internal maintenance
|
|
666
|
+
bookkeeping: a reader who knows where "the PRD" lives should still find it there afterwards.
|
|
667
|
+
MEASURED, real (bareagent): before this, a split left `CLAUDE.md`'s canonical-PRD reference
|
|
668
|
+
pointing into `docs/archive/`, which reads as "the canonical spec is archived" — backwards.
|
|
669
|
+
|
|
670
|
+
**Exit codes are not interchangeable — they mean two opposite outcomes:**
|
|
671
|
+
|
|
672
|
+
| exit | meaning | what to do |
|
|
673
|
+
|---|---|---|
|
|
674
|
+
| `0` | move succeeded, all follow-ups (artifact sync, link rewrite) succeeded | nothing |
|
|
675
|
+
| `1` | the move itself failed — **nothing moved** | fix the problem and re-run `archive` |
|
|
676
|
+
| `2` | the move **succeeded** — the file IS at the new path — but a follow-up (syncing `outline.json`/`labels.json`, or rewriting inbound links) failed | fix the follow-up by hand; do **NOT** re-run `archive` for this file, it has already moved |
|
|
677
|
+
|
|
678
|
+
A caller branching on exit code must treat 1 and 2 as distinct — retrying `archive` on a `2` is
|
|
679
|
+
exactly the mistake the printed message warns against.
|
|
680
|
+
|
|
681
|
+
Pruning the archive is the user's own call — `git rm` — and nothing in this pipeline does it
|
|
682
|
+
automatically. Archiving never deletes.
|
|
683
|
+
|
|
684
|
+
### 7. Search (script) — look a section up instead of reading the corpus whole
|
|
685
|
+
|
|
686
|
+
Row count is the variable that decides whether an index helps or hurts: 16 rows fine, 97 rows
|
|
687
|
+
won, **364 rows lost**. Past a hundred-odd rows, don't read `docs/index.md` whole — look
|
|
688
|
+
sections up directly. (An H3-grain outline as a reader index was the worst arm tested, and
|
|
689
|
+
indexing a monolith is never an alternative to splitting it.) The human entry point is the
|
|
690
|
+
slash command:
|
|
691
|
+
|
|
692
|
+
```
|
|
693
|
+
/docs-builder search <query words...>
|
|
694
|
+
```
|
|
695
|
+
|
|
696
|
+
which defaults the outline path and takes only the query. The underlying script form still
|
|
697
|
+
works directly, and is what the slash command runs:
|
|
698
|
+
|
|
699
|
+
```bash
|
|
700
|
+
REPO=<repo> node docs-builder/docs-builder.cjs search docs/.docs-builder/outline.json <query words...>
|
|
701
|
+
```
|
|
702
|
+
|
|
703
|
+
BM25 over each section's real text (no deps, no separate index to build — it reads
|
|
704
|
+
`outline.json` and the source files `scan` already produced). Ranks and points at a
|
|
705
|
+
file/line range; it does not read the section for you. Result count is `N` (default 10).
|
|
706
|
+
|
|
707
|
+
`search` can only rank what has a record — a file `outline.json` never scanned scores nothing
|
|
708
|
+
and cannot be found, no matter how well its title matches the query. `apply-reorg` (step 3
|
|
709
|
+
above) covers this for you: it re-scans the whole corpus, `docs/product/`, `docs/logs/`, and
|
|
710
|
+
`docs/archive/` all, every time it runs, so `search` sees every doc that has gone through the
|
|
711
|
+
reorg, not only whichever ones a split happened to touch.
|
|
712
|
+
|
|
713
|
+
### 8. Index-flat (script) — the WHOLE-CORPUS map, the only writer of `docs/index.md`
|
|
714
|
+
|
|
715
|
+
`index-flat` writes the corpus's one index — `apply-reorg` (step 3 above) and `cleanup-apply`
|
|
716
|
+
(step 4) both call it for you; this is only for re-running it by hand (e.g. after `git rm`-ing
|
|
717
|
+
some archived docs).
|
|
718
|
+
|
|
719
|
+
```bash
|
|
720
|
+
REPO=<repo> node docs-builder/docs-builder.cjs index-flat
|
|
721
|
+
```
|
|
722
|
+
|
|
723
|
+
Writes **one** `docs/index.md` covering the whole corpus, in three sections: `## Product`
|
|
724
|
+
(one row per file under `docs/product/`, plus any pages under `PAGES` — default `docs/wiki/`
|
|
725
|
+
— if they exist, plus any doc still sitting in place elsewhere), `## Logs` (one row per file
|
|
726
|
+
under `docs/logs/`), and `## Archive` (one row per file under `docs/archive/`). Each row is
|
|
727
|
+
an H1 title, a line count, and a link. No theme grouping, no `labels.json`, no model call.
|
|
728
|
+
Default destination `docs/index.md` — **the only writer of that default path** in this whole
|
|
729
|
+
pipeline (nothing else writes an index at all).
|
|
730
|
+
`search` reads `outline.json`, never `index.md`. Prints the row counts and records a `log.md`
|
|
731
|
+
line.
|
|
732
|
+
|
|
733
|
+
**Archive growth flag.** Every `index-flat` run counts the `## Archive` rows and prints a
|
|
734
|
+
console-only `WARN` once the count crosses `ARCHIVE_WARN_ROWS` (default 100, a stated default,
|
|
735
|
+
not a measured one). It only ever warns — never prunes, never collapses the section, never
|
|
736
|
+
deletes. Review `docs/index.md`'s `## Archive` section and `git rm` what you no longer need —
|
|
737
|
+
that is the whole mechanism; nothing in this pipeline prunes the archive for you.
|
|
738
|
+
|
|
739
|
+
---
|
|
740
|
+
|
|
741
|
+
## Mode 2 — reorg (the single front door)
|
|
742
|
+
|
|
743
|
+
v3 folds the old `reconcile` and `due` commands into one: "first run" (nothing sorted yet) and
|
|
744
|
+
"since last time" (a ledger stamp already exists) are the same job with different starting
|
|
745
|
+
state, and two separate commands only made users guess which one to run.
|
|
746
|
+
|
|
747
|
+
```bash
|
|
748
|
+
REPO=<repo> node docs-builder/docs-builder.cjs reorg
|
|
749
|
+
```
|
|
750
|
+
|
|
751
|
+
If a ledger stamp exists (see "Knowing when reorg is due" below), its `due`-style drift
|
|
752
|
+
summary prints FIRST — against whatever the tree looked like coming in, before this run's own
|
|
753
|
+
moves can confuse it. It then runs `discover`. **If any row's `bucket` is still empty,
|
|
754
|
+
`reorg` STOPS right there** and prints what to do next (run the classification interview,
|
|
755
|
+
step 2 above) — it never silently proceeds past an unclassified plan; that would be the exact
|
|
756
|
+
failure the approval gate exists to prevent, just moved one layer up. Once the plan is fully
|
|
757
|
+
classified — an already-sorted corpus's re-run carries its prior classifications forward
|
|
758
|
+
automatically (discover is idempotent, see step 1 above), so this is a genuine no-op on the
|
|
759
|
+
common case — it continues straight through: `apply-reorg` (which re-scans the whole corpus
|
|
760
|
+
itself and writes `docs/index.md` via `index-flat`) → `lint` over that same whole corpus.
|
|
761
|
+
`OUT` is IGNORED here, loudly: `reorg` writes several different artifacts
|
|
762
|
+
(`reorg-plan.json`, `outline.json`, `index.md`, `lint.json`) and a single `OUT` would point
|
|
763
|
+
them all at one file — set it on an individual subcommand instead.
|
|
764
|
+
|
|
765
|
+
**What the old `reconcile`'s `validate`/`index` steps did has no home in `reorg`, and that is
|
|
766
|
+
not a loss.** Those two needed a theme assignment (`labels.json`) that only the model's
|
|
767
|
+
grouping step (2a, above) ever produces — and `reorg` never calls a model by default and never
|
|
768
|
+
splits anything (rule: splitting is opt-in, per file, via `cleanup` only). So it never has a
|
|
769
|
+
`labels.json` to work from. That capability didn't move; it stayed exactly where it already
|
|
770
|
+
lived — the standalone `validate` (step 3) and `index-flat` (step 8) subcommands, unchanged,
|
|
771
|
+
still runnable by hand once a `labels.json` exists.
|
|
772
|
+
|
|
773
|
+
`lint` is also runnable standalone, on any file list, not only as part of `reorg`:
|
|
774
|
+
|
|
775
|
+
```bash
|
|
776
|
+
REPO=<repo> node docs-builder/docs-builder.cjs lint <file.md...>
|
|
777
|
+
```
|
|
778
|
+
|
|
779
|
+
-> `lint.json`. Every check below is declared-only (see the governing rule further down) —
|
|
780
|
+
nothing is inferred from similarity.
|
|
781
|
+
|
|
782
|
+
| check | precision | how to treat it |
|
|
783
|
+
|---|---|---|
|
|
784
|
+
| `supersession` (declared in a HEADING) | **24/24 across 4 repos** | act on it |
|
|
785
|
+
| `supersessionInBody` | not measured | read, never act |
|
|
786
|
+
| `uncited` (repo-wide sweep) | fact, not a verdict | **propose only** |
|
|
787
|
+
| `redundant` (shared verbatim sentences) | 1/4 | **propose only** |
|
|
788
|
+
|
|
789
|
+
**The governing rule: observed beats inferred.** Lint only on what a doc *says about
|
|
790
|
+
itself*. Declared scored 100%; inferred scored 4–25%. Dangling-ID and duplicate-ID checks
|
|
791
|
+
are **cut entirely** — `P95` is a percentile, not a broken reference.
|
|
792
|
+
|
|
793
|
+
`uncited` must be **repo-wide**. Scoped to the doc corpus it flagged two live docs that are
|
|
794
|
+
cited from a logs file and `CHANGELOG.md`. And uncited ≠ deletable: bareloop's `O2`–`O4`
|
|
795
|
+
are genuinely uncited and must stay — they are the middle of a coherent `O1`–`O5` series.
|
|
796
|
+
|
|
797
|
+
Frontmatter is what makes a low-precision flag safe to ship: a flag is a **proposal**, and
|
|
798
|
+
confirmation is recorded in the file itself.
|
|
799
|
+
|
|
800
|
+
```yaml
|
|
801
|
+
type: reference # the only required field
|
|
802
|
+
title: ...
|
|
803
|
+
status: draft | stable | deprecated
|
|
804
|
+
sources: [...]
|
|
805
|
+
verified: {by: "human:<name>", at: 2026-08-21}
|
|
806
|
+
stale_after: 2026-12-01 # ONLY when asked and answered. never inferred.
|
|
807
|
+
```
|
|
808
|
+
|
|
809
|
+
### Knowing when reorg is due
|
|
810
|
+
|
|
811
|
+
git is the diff engine. The ledger stores only the one thing git cannot know — **when you
|
|
812
|
+
last consolidated** — so the two can never drift apart.
|
|
813
|
+
|
|
814
|
+
```bash
|
|
815
|
+
node docs-builder/docs-builder.cjs ledger # stamp the current state (run after a reorg)
|
|
816
|
+
node docs-builder/docs-builder.cjs due # what changed since, and by how much
|
|
817
|
+
```
|
|
818
|
+
|
|
819
|
+
`due` classifies every doc against the stamped SHA using `git diff --numstat -M`:
|
|
820
|
+
|
|
821
|
+
| kind | means |
|
|
822
|
+
|---|---|
|
|
823
|
+
| `new` | did not exist at the last consolidation |
|
|
824
|
+
| `moved` | same content, different path (`-M` rename detection) |
|
|
825
|
+
| `moved+changed` | renamed **and** edited, with the line delta |
|
|
826
|
+
| `changed` | `+added/-deleted of N lines (~X%)` — how much of the doc actually moved |
|
|
827
|
+
| `deleted` | was in the ledger, gone from the tree |
|
|
828
|
+
|
|
829
|
+
A reorg is **due at 5 changed docs**, the same threshold and the same derived-not-counted
|
|
830
|
+
shape `/stash` uses for its nudge. `due` only ever prints; it never runs `reorg` for you.
|
|
831
|
+
|
|
832
|
+
`/remember` calls `due` at the end of its run (its step 7), detect-only and crash-isolated.
|
|
833
|
+
Silent when the project has no `docs/`; **loud** if `docs/` exists but the check could not
|
|
834
|
+
run; one nudge line when it is due. `/remember` never consolidates — `/docs-builder` owns the
|
|
835
|
+
ledger, `/remember` only reads it.
|
|
836
|
+
|
|
837
|
+
---
|
|
838
|
+
|
|
839
|
+
## Cost
|
|
840
|
+
|
|
841
|
+
Measured end-to-end on a 5,669-line doc → 10 pages: **$2.20**, or **$0.39 per 1,000 source
|
|
842
|
+
lines**. Group step (cheap tier) $0.23; write step (mid tier) $1.97.
|
|
843
|
+
|
|
844
|
+
**Write cost law** (n=10, R² = 0.96): `$0.083 per page + $0.200 per 1,000 source lines`.
|
|
845
|
+
42% of the write bill is per-page fixed overhead, so page count matters as much as size —
|
|
846
|
+
but page count is set by coherence, not by cost.
|
|
847
|
+
|
|
848
|
+
The cheap-tier group calls were flat at 35–41K tokens regardless of input size. **That
|
|
849
|
+
flatness does not generalise** — the mid-tier write calls are ~58% input-driven. Do not
|
|
850
|
+
carry a per-step cost shape across steps.
|
|
851
|
+
|
|
852
|
+
## Open
|
|
853
|
+
|
|
854
|
+
1. **Chunk 40–50 vs 20** — untested. Now worth <5% of the bill; low priority.
|
|
855
|
+
2. **Concurrency cap of 3** is not tuned. Its original justification (10 parallel writers
|
|
856
|
+
"killed 9 of 10") turned out to describe lost *cost accounting*, not lost pages. It may
|
|
857
|
+
still be the right number for a different reason — smaller batches may produce better
|
|
858
|
+
pages — but that is a **quality** hypothesis and it is untested.
|
|
859
|
+
3. **Mechanical clustering is CUT** — tf-idf gave a 42% blob and a naive sequential chop
|
|
860
|
+
beat it. But that corpus was an append-only log, where a chop wins by construction. n=1.
|
|
861
|
+
4. **Page density is unjudged.** CLI-written pages ran ~half the prose of subagent-written
|
|
862
|
+
ones at the same citation count. Which reads better is untested.
|
|
863
|
+
5. **`reorg`'s 500-line oversized ceiling is a stated default, not a measured one.**
|
|
864
|
+
Classification quality (archive/product split) was validated against real, uncrafted
|
|
865
|
+
corpora; the size cutoff itself was not tuned against anything.
|
|
866
|
+
6. **`reorg` was validated on ONE real corpus family** (bareloop's docs/, ~35 files). Its
|
|
867
|
+
SHOUTED-caps status rule leans on that corpus's own writing convention; a repo that
|
|
868
|
+
never shouts status in caps will simply get fewer `archive` hits (a recall loss, not a
|
|
869
|
+
false-archive risk) rather than a wrong one — but that's inferred from the design, not
|
|
870
|
+
independently confirmed on a second differently-styled corpus. n=1.
|
|
871
|
+
7. **`reorg` has not been run on a repo it doesn't control the content of.** Every corpus
|
|
872
|
+
tested so far was one this session had full visibility into. No adversarial or
|
|
873
|
+
unusually-formatted real-world corpus has been thrown at it yet.
|