agent-docs-bootstrap 0.0.0-stage → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +58 -0
- package/LICENSE +21 -0
- package/README.md +202 -2
- package/THIRD_PARTY.md +35 -0
- package/agents/docs-reader.md +36 -0
- package/agents/docs-scout.md +64 -0
- package/install.mjs +198 -0
- package/package.json +11 -4
- package/skills/docs-bootstrap/SKILL.md +158 -0
- package/skills/docs-bootstrap/agents/openai.yaml +3 -0
- package/skills/docs-bootstrap/references/agents-md-format.md +70 -0
- package/skills/docs-bootstrap/references/c4-format.md +32 -0
- package/skills/docs-bootstrap/references/scout-briefs.md +76 -0
- package/skills/docs-bootstrap/references/signals.md +26 -0
- package/skills/docs-bootstrap/templates/AGENTS.starter.md +43 -0
- package/skills/docs-bootstrap/templates/bootstrap-state.md +30 -0
- package/skills/docs-bootstrap/templates/sources.md +19 -0
- package/skills/docs-bootstrap/vendor/diataxis-audit/LICENSE +21 -0
- package/skills/docs-bootstrap/vendor/diataxis-audit/diataxis-audit.md +180 -0
- package/skills/docs-bootstrap/vendor/doc-coauthoring/LICENSE +202 -0
- package/skills/docs-bootstrap/vendor/doc-coauthoring/doc-coauthoring.md +375 -0
- package/skills/docs-bootstrap/vendor/domain-modeling/ADR-FORMAT.md +47 -0
- package/skills/docs-bootstrap/vendor/domain-modeling/CONTEXT-FORMAT.md +60 -0
- package/skills/docs-bootstrap/vendor/domain-modeling/LICENSE +21 -0
- package/skills/docs-bootstrap/vendor/domain-modeling/agents/openai.yaml +3 -0
- package/skills/docs-bootstrap/vendor/domain-modeling/domain-modeling.md +74 -0
- package/skills/docs-bootstrap/vendor/grilling/LICENSE +21 -0
- package/skills/docs-bootstrap/vendor/grilling/agents/openai.yaml +3 -0
- package/skills/docs-bootstrap/vendor/grilling/grilling.md +28 -0
- package/skills/docs-bootstrap/vendor/writing-for-agents/LICENSE +21 -0
- package/skills/docs-bootstrap/vendor/writing-for-agents/SKILL-MECHANICS.md +22 -0
- package/skills/docs-bootstrap/vendor/writing-for-agents/agents/openai.yaml +3 -0
- package/skills/docs-bootstrap/vendor/writing-for-agents/writing-for-agents.md +81 -0
- package/skills/docs-check/SKILL.md +114 -0
- package/skills/docs-check/agents/openai.yaml +3 -0
- package/skills/docs-check/scripts/reader-run.mjs +188 -0
- package/skills/docs-check/templates/reader-tests.md +29 -0
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
# Skill mechanics
|
|
2
|
+
|
|
3
|
+
The skill-specific branch of [`writing-for-agents`](SKILL.md): what changes when the document is a skill (frontmatter, the invocation choice, and router skills). Everything else about writing it is the universal reference in `SKILL.md`.
|
|
4
|
+
|
|
5
|
+
## Invocation
|
|
6
|
+
|
|
7
|
+
Two choices, trading the two loads:
|
|
8
|
+
|
|
9
|
+
- A **model-invoked** skill keeps a `description`, so the agent can fire it autonomously, and other skills can reach it. You can still type its name: model-invocation always _includes_ user reach; a description only ever adds agent discovery, never removes the human's. The description is the skill's top-level context pointer, forced to stay loaded at all times: permanent context load in exchange for discoverability. A model-invoked skill whose content is all reference is also one home for shared reference: another skill can invoke it, so reference needed by several skills lives in one place. Mechanics: omit `disable-model-invocation`, and write a model-facing description carrying the trigger branches (the pointer-writing rules in `SKILL.md` apply in full).
|
|
10
|
+
- A **user-invoked** skill strips the description from the agent's reach: only the human typing its name can invoke it, and no other skill can. Zero context load, but it spends cognitive load: you are the index that must remember it exists. Mechanics: set `disable-model-invocation: true`; the `description` becomes human-facing: a one-line summary, trigger lists stripped.
|
|
11
|
+
|
|
12
|
+
Pick model-invocation only when the agent must reach the skill on its own, or another skill must. If it only ever fires by hand, make it user-invoked and pay no context load.
|
|
13
|
+
|
|
14
|
+
Shared reference that two user-invoked skills both need can live in neither: with no descriptions, neither can fire the other. Push it to a plain file outside the skill system: external reference any skill can point at.
|
|
15
|
+
|
|
16
|
+
## Splitting by invocation
|
|
17
|
+
|
|
18
|
+
The invocation cut of splitting (the sequence cut lives in `SKILL.md`): split off a model-invoked skill when you have a distinct leading word that should trigger it on its own (a trigger word you actually use in your prompts), or another skill must reach it. You pay context load for the new always-loaded description, so that independent reach has to be worth it.
|
|
19
|
+
|
|
20
|
+
## Router skills
|
|
21
|
+
|
|
22
|
+
When user-invoked skills multiply past what you can remember, that piled-up cognitive load is cured by a **router skill**: one user-invoked skill that names the others and when to reach for each, so the human has one skill to remember instead of many. It can only hint, never fire them: user-invoked skills have no description, so nothing but the human can reach them.
|
|
@@ -0,0 +1,81 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: writing-for-agents
|
|
3
|
+
description: Writing documents for agents. Use when creating or editing skills, or modifying AGENTS.md or CLAUDE.md.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
Reference for writing any document an agent consumes: a skill, an `AGENTS.md` / `CLAUDE.md`, a doc reached by a pointer. The packaging differs; the writing does not: the same levers make each one predictable, since the agent takes the same _process_ every run rather than producing the same output.
|
|
7
|
+
|
|
8
|
+
When the document you're writing is a skill, read [`SKILL-MECHANICS.md`](SKILL-MECHANICS.md) for frontmatter, invocation choice, and router skills.
|
|
9
|
+
|
|
10
|
+
## Context pointers
|
|
11
|
+
|
|
12
|
+
A **context pointer** is a reference held in the agent's context that names some out-of-context material and encodes the condition for reaching it. A skill's description is one; a line in `AGENTS.md` naming a doc is the same object. The pointer's _wording_, not its target, decides when the agent reaches the material, and how reliably. A must-have target behind a weakly worded pointer is a variance bug: sharpen the wording first, and inline the material only if sharpening fails.
|
|
13
|
+
|
|
14
|
+
A pointer does two jobs: state what the material is, and list the **branches** that should trigger reaching it (a branch is a distinct case the document handles, so different runs take different paths through it). Every word of an always-loaded pointer costs on every turn, so it earns even harder pruning than the body:
|
|
15
|
+
|
|
16
|
+
- **Front-load the leading word**: the pointer is where it does its triggering work.
|
|
17
|
+
- **One trigger per branch.** Synonyms that rename a single branch are one branch written twice; collapse them and keep only genuinely distinct branches.
|
|
18
|
+
- **Cut identity the body already carries.**
|
|
19
|
+
|
|
20
|
+
## The two loads
|
|
21
|
+
|
|
22
|
+
Every document and pointer you add spends one of two budgets:
|
|
23
|
+
|
|
24
|
+
- **Context load** is the cost of always-loaded material on the agent's window: an `AGENTS.md` line, a skill description, anything sitting in context every turn, spending tokens and attention whether or not it fires.
|
|
25
|
+
- **Cognitive load** is the cost on the human: which documents exist and when to reach for each. The human is the index. Not a cost to minimise: it is the price of human agency; spend it where human judgement matters, remove it where it does not.
|
|
26
|
+
|
|
27
|
+
Material reached only through a pointer escapes context load at the price of the pointer's own line; material with no pointer at all rides entirely on cognitive load.
|
|
28
|
+
|
|
29
|
+
## Information hierarchy
|
|
30
|
+
|
|
31
|
+
A document is built from two content types: **steps** (the ordered actions the agent performs) and **reference** (definitions, rules, facts consulted on demand). The two mix freely: all steps (a recipe), all reference (a review's rules, this skill), or both. The core decision is where each piece sits on the **information hierarchy**, a ladder ranked by how immediately the agent needs the material:
|
|
32
|
+
|
|
33
|
+
1. **In-file step** is the primary tier: what the agent does, in order.
|
|
34
|
+
2. **In-file reference** is consulted on demand. Often a legitimately flat peer-set (every rule of a review on one rung), which is a fine arrangement, not a smell.
|
|
35
|
+
3. **Disclosed reference** is pushed out into a separate file, reached by a context pointer, loaded only when the pointer fires. Spans a sibling file in the same folder through fully external reference that lives anywhere and any document can point at.
|
|
36
|
+
|
|
37
|
+
Push too little down and the top bloats; push too much and you hide material the agent actually needs. That tension is the whole decision.
|
|
38
|
+
|
|
39
|
+
**Progressive disclosure** is the move down the ladder (out of the main file and behind a pointer) so the top stays legible. Not primarily a token optimisation: it is how the hierarchy is protected. Branching is the cleanest disclosure test: inline what every branch needs, and push behind a pointer what only some branches reach. When a document has steps, in-file reference that should be disclosed buries them and turns attending to them into a coin-flip: a variance lever, not just a legibility one.
|
|
40
|
+
|
|
41
|
+
**Co-location** is the within-file companion: where the ladder decides _how far down_ a piece sits, co-location decides _what sits beside it_ once there. Keep a concept's definition, rules, and caveats under one heading rather than scattered, so reading one part brings its neighbours with it. The test: the document should read like documentation written for the agent. Grouped material reads that way; scattered material does not. (Distinct from duplication: that repeats one meaning in two places; scattering fragments one meaning across many.)
|
|
42
|
+
|
|
43
|
+
**Sprawl** is the failure mode here: a document simply too long, even when every line is live and unique. Attention thins across the excess, and every extra line is one more to keep relevant. The cure is the ladder: disclose reference behind pointers, and split by branch or sequence so each path carries only what it needs.
|
|
44
|
+
|
|
45
|
+
## Steps and completion criteria
|
|
46
|
+
|
|
47
|
+
Every step ends on a **completion criterion**, the condition that tells the agent the work is done. Two properties make it a lever:
|
|
48
|
+
|
|
49
|
+
- **Clarity**: can the agent tell done from not-done? A vague bound ("understanding reached") invites **premature completion**: ending the step before it is genuinely done, attention slipping to _being done_. The visible steps still ahead (the **post-completion steps**) supply the pull; the criterion's clarity is the resistance. Defend in order: **sharpen the bound first** (local and cheap); only if it is irreducibly fuzzy _and_ you observe the rush, hide the later steps by splitting the sequence. Hiding only works across a real context boundary (a hand-off or a subagent dispatch; an inline call leaves the later steps in context and clears nothing).
|
|
50
|
+
- **Demand**: how much it requires. "Every modified model accounted for" forces thorough work where "produce a change list" does not. Demand drives **legwork** (the digging the agent does within the work, latent in the wording rather than written as its own step), and it is not step-bound: "every rule applied" binds a body of flat reference just as "every step done" binds a sequence, which is how an all-reference document still carries an exhaustiveness bar.
|
|
51
|
+
|
|
52
|
+
The strongest criteria are both checkable and exhaustive.
|
|
53
|
+
|
|
54
|
+
## When to split
|
|
55
|
+
|
|
56
|
+
Splitting one document into two spends one of the two loads, so split only when the cut earns it:
|
|
57
|
+
|
|
58
|
+
- **By sequence**: split a run of steps where the post-completion steps tempt the agent to rush the one in front of it. Keeping them out of view drives more legwork on the current task. Beware the reverse: merging sequences exposes each step's later steps to what follows, inviting premature completion.
|
|
59
|
+
- **By invocation**, skill-specific: see [`SKILL-MECHANICS.md`](SKILL-MECHANICS.md).
|
|
60
|
+
|
|
61
|
+
## Leading words
|
|
62
|
+
|
|
63
|
+
A **leading word** is a compact concept already living in the model's pretraining that the agent thinks with while running the document (_lesson_, _fog of war_, _tracer bullets_). Repeated as a token, never as a sentence, it accumulates a distributed definition and anchors a whole region of behaviour in the fewest tokens, by recruiting priors the model already holds. Coining your own works if you define it clearly, but a made-up word recruits no priors: you pay in definition tokens what a pretrained word gives free; reach for an existing word first.
|
|
64
|
+
|
|
65
|
+
It anchors twice. In the body, _execution_: the agent reaches for the same behaviour every time the word appears, and inside flat reference it focuses attention on a class of thing to look for. In a pointer, _invocation_: when the same word lives in your prompts, your docs, and your codebase, the agent links that shared language to the material and reaches it more reliably.
|
|
66
|
+
|
|
67
|
+
Hunt for opportunities to refactor with leading words. A triad spelled out at three sites, a pointer spending a sentence to gesture at one idea. Each is a passage begging to collapse into a single token:
|
|
68
|
+
|
|
69
|
+
- "fast, deterministic, low-overhead" → _tight_ (a _tight_ loop).
|
|
70
|
+
- "a loop you believe in" → _red_, turning a fuzzy gate into a binary observable state (the loop goes _red_ on the bug, or it doesn't).
|
|
71
|
+
|
|
72
|
+
You win twice: fewer tokens, and a sharper hook for the agent to hang its thinking on. Assume every document is carrying restatements that leading words retire. Go find them.
|
|
73
|
+
|
|
74
|
+
**Negation** is the failure mode beside this lever: steering by prohibition drags the forbidden behaviour into context and makes it _more_ available, not less. _Don't think of an elephant_, and the elephant is all there is; the negation is a weak modifier the strongly-activated concept overruns, so the ban half-reads as an instruction to do the thing. Prompt the **positive**: state the target behaviour ("write one-line comments") so the banned one is never spoken. A prohibition earns its place only as a hard guardrail you cannot phrase positively; even then, pair it with the positive target so attention lands on what to do.
|
|
75
|
+
|
|
76
|
+
## Pruning
|
|
77
|
+
|
|
78
|
+
- Keep each meaning in a **single source of truth**: one authoritative place, so changing the behaviour is a one-place edit. **Duplication** (the same meaning in more than one place) costs maintenance and tokens, and inflates a meaning's prominence on the ladder past its real rank. (The accidental inverse of a leading word, which repeats a token on purpose, never the meaning.)
|
|
79
|
+
- The **environment** is a source of truth too (`package.json` scripts, config files, the directory layout, `--help` output), and a document that restates it is a **cache**: a copy of a lookup, earning its load only when the lookup is expensive. Cache what the agent cannot find by looking: the unwritten convention, the reason behind a choice, the gotcha no config confesses. Leave the one-file, one-command lookups to the environment, where they cannot go stale.
|
|
80
|
+
- Check every line for **relevance**: does it still bear on what the document does? A line loses relevance by never bearing on the task (mere exposition, or a branch that should be disclosed) or by going stale as the behaviour or world it describes changes. Shorter documents are easier to keep relevant. Without a pruning discipline the default fate is **sediment**: stale layers that settle because adding feels safe and removing feels risky, until you must core down through them to find what is still live.
|
|
81
|
+
- Hunt **no-ops** sentence by sentence: an instruction the model already obeys by default pays load to say nothing. The test (does it change behaviour versus the default?) is model-relative, not reader-relative: two people disagreeing about a no-op disagree about the default, and settle it by running the document, not by debate. When a sentence fails, delete the whole sentence rather than trim words from it. The test also grades leading words: a word too weak to beat the default (_be thorough_ when the agent is already thorough-ish) is a no-op, and the fix is a stronger word (_relentless_), not a different technique.
|
|
@@ -0,0 +1,114 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: docs-check
|
|
3
|
+
description: Optional reader test for project documentation — a fresh agent, started as a separate process, plans a task on a history-free code snapshot with the current docs. Two modes - history (real past tasks, the plan is compared with what the real commit changed, gaps become one list of doc fixes) and own task (the human gives a task and reads the plan). Use only when the user asks to check, test or measure a project's docs or AGENTS.md; docs-bootstrap does not run it.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# docs-check
|
|
7
|
+
|
|
8
|
+
Checks whether an agent can work from the project's documentation. It is optional: run it only when the human asks. It costs a reader process per task and several minutes each, and it diagnoses where a fresh agent stumbles rather than proving the documentation good.
|
|
9
|
+
|
|
10
|
+
Two modes. Ask once which one, unless the human already said or gave a task with the command:
|
|
11
|
+
|
|
12
|
+
- **History** — three real past tasks; the reader's plans are compared with what the real commits changed, and the gaps become one list of doc fixes. Steps 1–5 below.
|
|
13
|
+
- **Own task** — the human gives a task, for example a current ticket; the reader plans it from the documents, and the human reads the plan. See [Own task](#own-task).
|
|
14
|
+
|
|
15
|
+
You organise the test; you do not judge whether the documentation is good and do not defend it. In the history mode the reference is what the real commit changed, not your opinion.
|
|
16
|
+
|
|
17
|
+
No `docs/agents/reader-tests.md` → create it from this skill's template (`templates/reader-tests.md`). No `docs/agents/scratch/` line in the project's `.gitignore` → add it. Hand-written documentation can be checked too, not only documentation built with `docs-bootstrap`.
|
|
18
|
+
|
|
19
|
+
## Step 1. Tasks
|
|
20
|
+
|
|
21
|
+
Three tasks with status `ready` in `reader-tests.md` → leak check below, then step 2. A task marked `used` was the source of earlier fixes: the documents now hold its answer, so it tests nothing. Replace it.
|
|
22
|
+
|
|
23
|
+
Missing tasks → start the `docs-scout` subagent with the brief "Reader test task candidates" from `docs-bootstrap/references/scout-briefs.md`. Pick three from different areas yourself, preferring candidates with a clear commit message, a test in the diff and no unrelated files in `--stat`. Prefer commits made after the documents were written (the dates in `docs/agents/bootstrap-state.md`, or the documents' first commits): the documents could not know their solutions. Fewer than three such commits → say so in the report: older tasks may share material with the documents. Do not stop to ask the human: they see the tasks with the results in step 5 and can strike out what came from a poor commit.
|
|
24
|
+
|
|
25
|
+
Record in `reader-tests.md`: commit hash, parent commit, task wording, reference, status `ready`.
|
|
26
|
+
|
|
27
|
+
**Wording.** What a developer would understand without the ticket: what to do and where the user sees it. No file names, hashes, ticket numbers or tracker links. The reader gets it word for word.
|
|
28
|
+
|
|
29
|
+
**Reference.** Files and areas of code and tests from `git show --stat <hash>`, without unrelated changes in the commit. Verification commands only if the reader can find them in the repository (package scripts, CI jobs that run locally). No documentation files, CI scans or deploys: the reader cannot name them.
|
|
30
|
+
|
|
31
|
+
**Leak check.** Before every run, take from `git show <hash>` the names the solution introduced: new files, classes, functions, endpoints, keys. Search for them:
|
|
32
|
+
|
|
33
|
+
- in the parent's code (`git grep <name> <parent>`) → found: the solution is already in the snapshot. Mark the task `replaced` and pick another;
|
|
34
|
+
- in the current documents (`AGENTS.md`, `CLAUDE.md`, `CONTEXT*.md`, `docs/`) → found: the documents already know the answer and the task tests nothing. Mark it `used` and pick another.
|
|
35
|
+
|
|
36
|
+
## Steps 2–3. Snapshot and reader
|
|
37
|
+
|
|
38
|
+
The reader must not see the solution, the history, your session's instructions or the project's own repository. So it runs as a **separate process** of your harness, in a snapshot **outside the repository**, never as a subagent of this session: a subagent shares the session's working directory, where the solution sits in the current code. The kit's script builds the snapshot and starts the reader the same way in bash, PowerShell and cmd.
|
|
39
|
+
|
|
40
|
+
1. Write the run's tasks to `docs/agents/scratch/reader-runs/<date>/tasks.json`:
|
|
41
|
+
|
|
42
|
+
```json
|
|
43
|
+
{ "tasks": [ { "n": 1, "parent": "<parent>", "task": "<wording from reader-tests.md>" } ] }
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
2. Run the script once per task from the project root, with `--harness` set to the harness you are running in (`claude`, `codex` or `opencode`). The three runs are independent: start them in parallel if your harness can, one at a time only if the model provider rate-limits parallel requests. A reader takes several minutes: if your shell tool has a timeout, set it to 15 minutes.
|
|
47
|
+
|
|
48
|
+
```
|
|
49
|
+
node <skills>/docs-check/scripts/reader-run.mjs docs/agents/scratch/reader-runs/<date> <N> --harness <harness>
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
`<skills>` is where this skill is installed: `.claude/skills` or `.agents/skills`. Write the command exactly like this, with forward slashes and without `cd`, `&&`, pipes or redirects, so it works in any shell. For a global install, add `--reader <path to the installed docs-reader>`.
|
|
53
|
+
|
|
54
|
+
What the script does:
|
|
55
|
+
|
|
56
|
+
- the snapshot is the parent's code without the documents and the kit's installed files, with the current documents from the working tree on top, committed or not (git-ignored and deleted ones are skipped, `docs/agents/` never goes in). Do not keep unfinished hand edits in the documents during a run: the reader gets them as they are. `README` and other files stay as they were in the parent commit, like the code: today's `README` may already describe the solution;
|
|
57
|
+
- the snapshot gets an empty `git init`, which stops the harness from loading `AGENTS.md` or `CLAUDE.md` from parent directories. No installed dependencies: the reader builds a plan, it does not execute it;
|
|
58
|
+
- the reader: Claude Code `claude -p --restricted` with only `Read`, `Grep`, `Glob` and no MCP servers; Codex `codex exec -s read-only` with every MCP server, plugin and app turned off; OpenCode `opencode run --pure` with the `docs-reader` agent, whose permissions deny everything but reading, MCP tools included. The prompt is the installed `docs-reader`, the task goes through stdin;
|
|
59
|
+
- it writes `rt-<N>-plan.md` and `rt-<N>-log.md` to the run directory and prints a three-line summary.
|
|
60
|
+
|
|
61
|
+
Non-zero exit or "plan EMPTY" → show the human the summary and `rt-<N>-log.md`; do not guess. A plan that cites files outside the snapshot, the history, a tracker or external docs is invalid: note why in `reader-tests.md` and rerun.
|
|
62
|
+
|
|
63
|
+
The user's global instructions may still reach the reader: Codex loads `~/.codex/AGENTS.md`, OpenCode `~/.config/opencode/AGENTS.md`; whether `claude -p --restricted` loads `~/.claude/CLAUDE.md` is not documented. If they are about this project, ask the human to move them aside for the test.
|
|
64
|
+
|
|
65
|
+
Your harness is not `claude`, `codex` or `opencode`, its CLI is not in PATH, or there is no `node` → do not play the reader yourself: you have seen the docs and the history. Run the script with `--snapshot-only` (it prints the snapshot path) and ask the human to start a new agent session with its working directory set to that path, paste the `docs-reader` prompt and the task, and save the answer to `rt-<N>-plan.md`.
|
|
66
|
+
|
|
67
|
+
## Step 4. Gaps
|
|
68
|
+
|
|
69
|
+
Compare each plan with the reference in `reader-tests.md`. Each of these is a gap, written down as is, without interpretation:
|
|
70
|
+
|
|
71
|
+
- an expected file or area is not named in the plan (a named directory of the area counts);
|
|
72
|
+
- an expected verification command is not named;
|
|
73
|
+
- a term listed under `_Avoid_` in the glossary is used (if `CONTEXT.md` exists);
|
|
74
|
+
- a directory outside the expected areas is proposed for change;
|
|
75
|
+
- a solution rejected in an ADR is proposed (if `docs/adr/` exists);
|
|
76
|
+
- an item under "had to search" whose answer is or should be in the documents.
|
|
77
|
+
|
|
78
|
+
Not gaps, but kept:
|
|
79
|
+
|
|
80
|
+
- "had to search" items only a human can answer (product clarifications) — candidates for how-tos and `AGENTS.md`;
|
|
81
|
+
- **snapshot artifacts** — the documents describe a newer layout than the parent's code (a moved directory, a new test runner). Many of them mean the task is too old: replace it next time.
|
|
82
|
+
|
|
83
|
+
An item where the reader says it found the answer in the documents is not a "had to search" item.
|
|
84
|
+
|
|
85
|
+
## Step 5. Fixes
|
|
86
|
+
|
|
87
|
+
**One list, no rerun loop.**
|
|
88
|
+
|
|
89
|
+
1. Group the gaps by the document that should hold the answer (`AGENTS.md`, glossary, a how-to, an ADR). For each fix, the gap it closes and the task it came from. Write a fix as general project knowledge ("new pages are registered in …"), not as this task's solution. A fix never contains passwords, tokens or keys: only where to get them. Product questions go to a separate block as open questions, not fixes.
|
|
90
|
+
2. Show the list with the three tasks and their commits. Say that some gaps may come from the commit rather than the docs (a commit the team would not repeat today, unrelated changes), and let the human strike those out. This is the only stop in the check.
|
|
91
|
+
3. Apply the approved fixes. Mark each task the documents were fixed for as `used`: they now hold its answer.
|
|
92
|
+
4. Do not rerun the same task to "confirm" the fix: it will pass because the answer is now in the documents, which proves nothing. The next check on new tasks shows whether the docs got better.
|
|
93
|
+
|
|
94
|
+
Add a row to the runs table in `reader-tests.md`: date, task, gaps found, snapshot artifacts, fixes applied.
|
|
95
|
+
|
|
96
|
+
Report to the human: per task, the gaps and the applied fixes, the open questions, and when to check again — after a few weeks of team work or notable changes, on new tasks.
|
|
97
|
+
|
|
98
|
+
## Own task
|
|
99
|
+
|
|
100
|
+
The human gives a task in their own words, for example a ticket they are about to start. There is no reference commit, so there is no leak check, no gap list and no task status: the human reads the plan and decides what the documents lack.
|
|
101
|
+
|
|
102
|
+
1. Take the wording as the human gave it. Remove file names, hashes and tracker links only with their consent: they hand the reader the answer. Record the task in the "Own tasks" table of `reader-tests.md` with the commit it runs on.
|
|
103
|
+
2. The snapshot is the current commit: write `tasks.json` as in steps 2–3 with `"parent"` set to the hash from `git rev-parse HEAD`, and run the script the same way. The code comes from that commit and the documents from the working tree, so uncommitted code changes are not in the snapshot: tell the human if `git status` shows any.
|
|
104
|
+
3. Show the plan as it is, then two lists from it: what the reader had to search the code for or guess, and its questions to the team. Do not grade the plan or argue with it.
|
|
105
|
+
4. The human says what the documents should have answered. Group those items by document as in step 5 and apply what they approve; secrets are never written, only where to get them. Add a row to the runs table.
|
|
106
|
+
|
|
107
|
+
Do not reuse an own task for the history mode later: its plan has already shaped the documents.
|
|
108
|
+
|
|
109
|
+
## Never
|
|
110
|
+
|
|
111
|
+
- Ask any agent to judge the plans or the documentation instead of comparing with the reference: an LLM judge prefers the longer, more confident plan.
|
|
112
|
+
- Start the reader as a subagent of this session, from inside the repository, or with your own commands instead of `reader-run.mjs`.
|
|
113
|
+
- Show the reader the project's git history or a tracker link, or mention the history. The empty repository that `reader-run.mjs` creates in the snapshot holds no history.
|
|
114
|
+
- Rerun a task after fixing the documents for it.
|
|
@@ -0,0 +1,188 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
// docs-check steps 2–3: a history-free snapshot outside the repository and a fresh reader started as a
|
|
3
|
+
// separate process of your harness. Works the same from bash, PowerShell and cmd; needs git and Node 18+.
|
|
4
|
+
//
|
|
5
|
+
// node <skills>/docs-check/scripts/reader-run.mjs <run dir> <N> --harness claude|codex|opencode [--reader <path>] [--timeout-min <m>]
|
|
6
|
+
// node <skills>/docs-check/scripts/reader-run.mjs <run dir> <N> --snapshot-only
|
|
7
|
+
//
|
|
8
|
+
// <run dir> holds tasks.json: { "tasks": [ { "n": 1, "parent": "<hash>", "task": "<wording>" } ] }.
|
|
9
|
+
// Writes rt-<N>-plan.md (the reader's answer) and rt-<N>-log.md there; prints a short summary.
|
|
10
|
+
// --timeout-min stops the reader after that many minutes (default 12).
|
|
11
|
+
|
|
12
|
+
import fs from 'node:fs';
|
|
13
|
+
import path from 'node:path';
|
|
14
|
+
import os from 'node:os';
|
|
15
|
+
import { execFileSync, spawn } from 'node:child_process';
|
|
16
|
+
import { createHash } from 'node:crypto';
|
|
17
|
+
|
|
18
|
+
const a = parseArgs(process.argv.slice(2));
|
|
19
|
+
if (a.help || a._.length < 2) {
|
|
20
|
+
const src = fs.readFileSync(new URL(import.meta.url), 'utf8').split('\n').slice(1);
|
|
21
|
+
console.log(src.slice(0, src.findIndex(l => !l.startsWith('//'))).map(l => l.replace(/^\/\/ ?/, '')).join('\n'));
|
|
22
|
+
process.exit(a.help ? 0 : 1);
|
|
23
|
+
}
|
|
24
|
+
|
|
25
|
+
const ROOT = git(['rev-parse', '--show-toplevel'], process.cwd()).trim();
|
|
26
|
+
const RUN = path.resolve(a._[0]);
|
|
27
|
+
const N = Number(a._[1]);
|
|
28
|
+
const spec = JSON.parse(read(path.join(RUN, 'tasks.json'), 'no tasks.json in the run directory'));
|
|
29
|
+
const task = (spec.tasks || []).find(t => Number(t.n) === N) || die(`tasks.json has no task n=${N}`);
|
|
30
|
+
if (!task.parent || !task.task) die(`task ${N} has no parent or task`);
|
|
31
|
+
|
|
32
|
+
// The path hash keeps two repositories with the same folder name apart.
|
|
33
|
+
const SNAPS = path.join(os.tmpdir(), `agent-docs-check-${path.basename(ROOT)}-${createHash('sha1').update(ROOT).digest('hex').slice(0, 8)}`);
|
|
34
|
+
const SNAP = path.join(SNAPS, `rt-${N}`);
|
|
35
|
+
const TIMEOUT_MS = Number(a['timeout-min'] || 12) * 60 * 1000;
|
|
36
|
+
const win = process.platform === 'win32';
|
|
37
|
+
|
|
38
|
+
// Documents the kit builds; kit-installed paths and its run files never reach the reader. The marker is committed
|
|
39
|
+
// and could hold any line, so only lines shaped like an installed path count (a line `docs` would hide every document).
|
|
40
|
+
const KIT_DIRS = ['.claude/skills', '.claude/agents', '.agents/skills', '.codex/agents', '.opencode/agents', '.opencode/commands'];
|
|
41
|
+
const kitPaths = read(path.join(ROOT, '.agent-docs-bootstrap'), '', true).split('\n').map(l => l.trim())
|
|
42
|
+
.filter(l => KIT_DIRS.includes(path.posix.dirname(l)) && /^[a-z0-9-]+(\.md|\.toml)?$/.test(path.posix.basename(l)));
|
|
43
|
+
const isKit = (p) => p === '.agent-docs-bootstrap' || p.startsWith('docs/agents/') || kitPaths.some(k => p === k || p.startsWith(k + '/'));
|
|
44
|
+
// README and other files stay as they were in the parent commit, like the code: they are the project as it was,
|
|
45
|
+
// not the documents under test, and today's README may already describe the solution.
|
|
46
|
+
const isDoc = (p) => /(^|\/)(AGENTS\.md|CLAUDE\.md|CONTEXT[^/]*\.md)$/.test(p) || p.startsWith('docs/');
|
|
47
|
+
|
|
48
|
+
// ---------- snapshot: the parent's code without documents, today's documents from the working tree on top
|
|
49
|
+
fs.rmSync(SNAP, { recursive: true, force: true });
|
|
50
|
+
fs.mkdirSync(SNAP, { recursive: true });
|
|
51
|
+
exportParent(p => !isKit(p) && !isDoc(p));
|
|
52
|
+
// Tracked and new documents, committed or not; git-ignored and deleted ones are skipped.
|
|
53
|
+
const docs = git(['ls-files', '-z', '--cached', '--others', '--exclude-standard'], ROOT).split('\0')
|
|
54
|
+
.filter(p => p && isDoc(p) && !isKit(p) && fs.existsSync(path.join(ROOT, p)));
|
|
55
|
+
for (const p of docs) {
|
|
56
|
+
fs.mkdirSync(path.dirname(path.join(SNAP, p)), { recursive: true });
|
|
57
|
+
fs.copyFileSync(path.join(ROOT, p), path.join(SNAP, p));
|
|
58
|
+
}
|
|
59
|
+
// An empty repository stops harnesses from loading AGENTS.md / CLAUDE.md from parent directories.
|
|
60
|
+
git(['init', '-q', SNAP], SNAP);
|
|
61
|
+
|
|
62
|
+
const files = walk(SNAP).filter(p => !p.startsWith('.git/'));
|
|
63
|
+
const leaked = files.filter(isKit);
|
|
64
|
+
if (leaked.length) die(`kit files in the snapshot: ${leaked.slice(0, 5).join(', ')}`);
|
|
65
|
+
if (a['snapshot-only']) { console.log(SNAP); process.exit(0); }
|
|
66
|
+
|
|
67
|
+
// ---------- reader
|
|
68
|
+
const harness = a.harness || die('pass --harness claude|codex|opencode: the harness you are running in');
|
|
69
|
+
const defaults = { claude: '.claude/agents/docs-reader.md', codex: '.codex/agents/docs-reader.toml', opencode: '.opencode/agents/docs-reader.md' };
|
|
70
|
+
if (!defaults[harness]) die(`unknown harness '${harness}'`);
|
|
71
|
+
const readerFile = path.resolve(ROOT, a.reader || defaults[harness]);
|
|
72
|
+
const reader = parseReader(read(readerFile, `no docs-reader for ${harness} (installed with --target ${harness}? pass --reader <path>)`));
|
|
73
|
+
const ask = `Task: ${task.task.trim()}\n\nMake a plan: files, changes, verification commands. Do not edit anything. Separately list everything you had to search the code for or guess, and your questions to the team.`;
|
|
74
|
+
const plan = path.join(RUN, `rt-${N}-plan.md`);
|
|
75
|
+
fs.mkdirSync(RUN, { recursive: true });
|
|
76
|
+
|
|
77
|
+
let cmd, args, env = { ...process.env }, input, fromFile = false;
|
|
78
|
+
if (harness === 'claude') {
|
|
79
|
+
// --restricted: no shell tools, file tools confined to the snapshot, no settings files; no MCP servers.
|
|
80
|
+
cmd = 'claude';
|
|
81
|
+
args = ['-p', '--restricted', '--strict-mcp-config', '--no-session-persistence', '--output-format', 'text',
|
|
82
|
+
...(reader.model ? ['--model', reader.model] : []), '--tools', 'Read,Grep,Glob'];
|
|
83
|
+
input = `${reader.prompt}\n\n${ask}`;
|
|
84
|
+
} else if (harness === 'codex') {
|
|
85
|
+
cmd = 'codex';
|
|
86
|
+
args = ['exec', '-C', SNAP, '-s', 'read-only', '--skip-git-repo-check', '--ephemeral', ...codexMcpOff(),
|
|
87
|
+
...(reader.model ? ['-m', reader.model] : []), '-o', plan, '-'];
|
|
88
|
+
input = `${reader.prompt}\n\n${ask}`;
|
|
89
|
+
fromFile = true;
|
|
90
|
+
} else {
|
|
91
|
+
// The agent definition goes to a config dir outside the snapshot; its permissions do the confining.
|
|
92
|
+
const cfg = path.join(SNAPS, 'opencode');
|
|
93
|
+
fs.mkdirSync(path.join(cfg, 'agents'), { recursive: true });
|
|
94
|
+
fs.copyFileSync(readerFile, path.join(cfg, 'agents', 'docs-reader.md'));
|
|
95
|
+
env = { ...env, OPENCODE_CONFIG_DIR: cfg, OPENCODE_DISABLE_CLAUDE_CODE: '1' };
|
|
96
|
+
cmd = 'opencode';
|
|
97
|
+
args = ['run', '--pure', '--dir', SNAP, '--agent', 'docs-reader'];
|
|
98
|
+
input = ask;
|
|
99
|
+
}
|
|
100
|
+
|
|
101
|
+
// On Windows the CLIs are often .cmd shims that only a shell can start. The task goes through stdin,
|
|
102
|
+
// so quotes and non-ASCII text never pass through the shell.
|
|
103
|
+
const child = spawn(win ? [cmd, ...args].map(q).join(' ') : cmd, win ? [] : args,
|
|
104
|
+
{ cwd: SNAP, env, shell: win, stdio: ['pipe', 'pipe', 'pipe'] });
|
|
105
|
+
child.stdin.end(input);
|
|
106
|
+
let out = '', err = '';
|
|
107
|
+
child.stdout.on('data', d => { out += d; });
|
|
108
|
+
child.stderr.on('data', d => { err += d; });
|
|
109
|
+
// With shell: true on Windows, child.kill() stops only cmd.exe; taskkill /T takes the CLI with it.
|
|
110
|
+
const stop = () => win ? spawn('taskkill', ['/pid', String(child.pid), '/T', '/F'], { stdio: 'ignore' }) : child.kill();
|
|
111
|
+
const timer = setTimeout(() => { err += `\nstopped after ${TIMEOUT_MS / 60000} min`; stop(); }, TIMEOUT_MS);
|
|
112
|
+
child.on('error', e => die(`${cmd} did not start: ${e.message}`));
|
|
113
|
+
child.on('close', code => {
|
|
114
|
+
clearTimeout(timer);
|
|
115
|
+
if (!fromFile || !fs.existsSync(plan)) fs.writeFileSync(plan, out);
|
|
116
|
+
const text = read(plan, '', true).trim();
|
|
117
|
+
fs.writeFileSync(path.join(RUN, `rt-${N}-log.md`), `# rt-${N}: ${harness} reader output\n\n## stdout\n\n\`\`\`\n${out.trim()}\n\`\`\`\n\n## stderr\n\n\`\`\`\n${err.trim()}\n\`\`\`\n`);
|
|
118
|
+
console.log([
|
|
119
|
+
`task ${N}, parent ${task.parent}, harness ${harness}`,
|
|
120
|
+
`snapshot: ${SNAP} (${files.length} files, ${docs.length} documents)`,
|
|
121
|
+
`reader: exit ${code}, plan ${text ? 'written' : 'EMPTY'}: ${path.relative(ROOT, plan)}`,
|
|
122
|
+
].join('\n'));
|
|
123
|
+
process.exit(code === 0 && text ? 0 : 1);
|
|
124
|
+
});
|
|
125
|
+
|
|
126
|
+
// ---------- helpers
|
|
127
|
+
function exportParent(keep) {
|
|
128
|
+
const list = git(['ls-tree', '-r', '-z', '--name-only', task.parent], ROOT).split('\0').filter(Boolean).filter(keep);
|
|
129
|
+
if (!list.length) return;
|
|
130
|
+
const index = path.join(SNAPS, `index-${N}`);
|
|
131
|
+
const genv = { ...process.env, GIT_INDEX_FILE: index };
|
|
132
|
+
try {
|
|
133
|
+
// A temporary index: the repository's own index and working tree are untouched, no tar needed.
|
|
134
|
+
execFileSync('git', ['read-tree', task.parent], { cwd: ROOT, env: genv, stdio: ['ignore', 'ignore', 'pipe'] });
|
|
135
|
+
execFileSync('git', ['checkout-index', '-f', '-z', '--stdin', `--prefix=${SNAP.split(path.sep).join('/')}/`],
|
|
136
|
+
{ cwd: ROOT, env: genv, input: list.join('\0'), stdio: ['pipe', 'ignore', 'pipe'] });
|
|
137
|
+
} finally { fs.rmSync(index, { force: true }); }
|
|
138
|
+
}
|
|
139
|
+
// `-c mcp_servers={}` is merged into the user's table and disables nothing. Plugin and app servers go with
|
|
140
|
+
// their features; every server from config.toml is switched off by name. A name with a dot cannot be
|
|
141
|
+
// addressed in a `-c` path.
|
|
142
|
+
function codexMcpOff() {
|
|
143
|
+
const off = ['--disable', 'plugins', '--disable', 'apps'];
|
|
144
|
+
const ls = [...off, 'mcp', 'list', '--json'];
|
|
145
|
+
let list;
|
|
146
|
+
try {
|
|
147
|
+
const out = execFileSync(win ? ['codex', ...ls].join(' ') : 'codex', win ? [] : ls,
|
|
148
|
+
{ cwd: SNAP, encoding: 'utf8', shell: win, stdio: ['ignore', 'pipe', 'pipe'] });
|
|
149
|
+
list = JSON.parse(out);
|
|
150
|
+
} catch (e) { die(`codex mcp list --json failed, cannot turn MCP servers off: ${(e.stderr || e.message).toString().trim()}`); }
|
|
151
|
+
const names = list.filter(s => s.enabled !== false).map(s => s.name);
|
|
152
|
+
const dotted = names.filter(n => n.includes('.'));
|
|
153
|
+
if (dotted.length) die(`MCP server names with a dot cannot be turned off for the reader: ${dotted.join(', ')}`);
|
|
154
|
+
return [...off, ...names.flatMap(n => ['-c', `mcp_servers.${n}.enabled=false`])];
|
|
155
|
+
}
|
|
156
|
+
// The installed docs-reader: Markdown with frontmatter (Claude Code, OpenCode) or TOML (Codex).
|
|
157
|
+
function parseReader(text) {
|
|
158
|
+
if (readerFile.endsWith('.toml')) {
|
|
159
|
+
return { prompt: (text.match(/developer_instructions = '''\n?([\s\S]*?)'''/) || die(`${readerFile}: no developer_instructions`))[1].trim(),
|
|
160
|
+
model: (text.match(/^model = "(.*)"$/m) || [])[1] };
|
|
161
|
+
}
|
|
162
|
+
const m = text.match(/^---\n([\s\S]*?)\n---\n([\s\S]*)$/) || die(`${readerFile}: no frontmatter`);
|
|
163
|
+
return { prompt: m[2].trim(), model: (m[1].match(/^model: (.+)$/m) || [])[1] };
|
|
164
|
+
}
|
|
165
|
+
function git(args, cwd) {
|
|
166
|
+
try { return execFileSync('git', args, { cwd, encoding: 'utf8', maxBuffer: 256 * 1024 * 1024, stdio: ['ignore', 'pipe', 'pipe'] }); }
|
|
167
|
+
catch (e) { die(`git ${args.join(' ')}: ${(e.stderr || e.message).toString().trim()}`); }
|
|
168
|
+
}
|
|
169
|
+
function walk(dir, base = dir) {
|
|
170
|
+
return fs.readdirSync(dir, { withFileTypes: true }).flatMap(e => {
|
|
171
|
+
const p = path.join(dir, e.name);
|
|
172
|
+
return e.isDirectory() ? walk(p, base) : [path.relative(base, p).split(path.sep).join('/')];
|
|
173
|
+
});
|
|
174
|
+
}
|
|
175
|
+
function q(s) { return /^[\w./:\\=,-]+$/.test(s) ? s : `"${s}"`; }
|
|
176
|
+
function read(p, msg, optional) { try { return fs.readFileSync(p, 'utf8').replace(/\r\n/g, '\n'); } catch { return optional ? '' : die(`${msg}: ${p}`); } }
|
|
177
|
+
function die(msg) { console.error(`reader-run: ${msg}`); process.exit(1); }
|
|
178
|
+
function parseArgs(argv) {
|
|
179
|
+
const o = { _: [] };
|
|
180
|
+
for (let i = 0; i < argv.length; i++) {
|
|
181
|
+
const k = argv[i];
|
|
182
|
+
if (k === '-h' || k === '--help') o.help = true;
|
|
183
|
+
else if (k === '--snapshot-only') o['snapshot-only'] = true;
|
|
184
|
+
else if (k.startsWith('--')) o[k.slice(2)] = argv[++i];
|
|
185
|
+
else o._.push(k);
|
|
186
|
+
}
|
|
187
|
+
return o;
|
|
188
|
+
}
|
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
# Reader test tasks
|
|
2
|
+
|
|
3
|
+
Tasks for the `docs-check` skill, an optional check the team runs when it wants to. History tasks are three typical project tasks from git history, picked by the agent; own tasks are given by a human.
|
|
4
|
+
|
|
5
|
+
Wording — what a developer would understand without the ticket: no file names, hashes or tracker links. Reference — files and areas of code and tests from `git show --stat`, without unrelated changes; commands only those the reader can find in the repository. Documentation, CI scans and deploys are not part of the reference.
|
|
6
|
+
|
|
7
|
+
Status: `ready` — goes into the next check; `used` — the documents hold its answer (they were fixed for it or mention the solution), replace it; `replaced` — the solution was already in the parent's code, or the task was swapped.
|
|
8
|
+
|
|
9
|
+
## History tasks
|
|
10
|
+
|
|
11
|
+
| # | Commit (hash) | Parent | Task (wording for the reader) | Expected files/areas | Expected commands | Area | Status |
|
|
12
|
+
|---|---|---|---|---|---|---|---|
|
|
13
|
+
| 1 | | | | | | | ready |
|
|
14
|
+
| 2 | | | | | | | ready |
|
|
15
|
+
| 3 | | | | | | | ready |
|
|
16
|
+
|
|
17
|
+
## Own tasks
|
|
18
|
+
|
|
19
|
+
| Date | Commit the snapshot ran on | Task (wording for the reader) | Given by |
|
|
20
|
+
|---|---|---|---|
|
|
21
|
+
| | | | |
|
|
22
|
+
|
|
23
|
+
## Runs
|
|
24
|
+
|
|
25
|
+
Plans and reader logs are in `docs/agents/scratch/reader-runs/<date>/`. For an own task, the gaps are what the human said the documents should have answered; snapshot artifacts do not apply.
|
|
26
|
+
|
|
27
|
+
| Date | Task | Gaps found | Snapshot artifacts | Fixes applied |
|
|
28
|
+
|---|---|---|---|---|
|
|
29
|
+
| | | | | |
|