project-spec 0.8.0 → 0.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +23 -2
- package/bin/project-spec.mjs +6 -2
- package/package.json +1 -1
- package/skills/project-knowledge/SKILL.md +27 -8
- package/skills/project-knowledge/assets/viewer.html +86 -15
- package/skills/project-knowledge/references/format.md +10 -5
- package/skills/project-knowledge/references/gaps.md +1 -1
- package/skills/project-knowledge/scripts/coverage.py +200 -0
- package/skills/project-knowledge/scripts/knowledge.py +62 -19
- package/skills/project-knowledge/scripts/test_coverage.py +128 -0
- package/skills/project-knowledge/scripts/test_knowledge.py +75 -0
- package/skills/spec-change/references/gaps.md +1 -1
package/README.md
CHANGED
|
@@ -51,7 +51,7 @@ It is made for one loop:
|
|
|
51
51
|
| --- | --- |
|
|
52
52
|
|  |  |
|
|
53
53
|
| **Recent changes** | **Gap queue** |
|
|
54
|
-
|  |  |
|
|
54
|
+
|  |  |
|
|
55
55
|
|
|
56
56
|

|
|
57
57
|
|
|
@@ -104,6 +104,7 @@ Without the package the command is `python3 <skill>/scripts/knowledge.py build -
|
|
|
104
104
|
| `--out FILE` | Where to write the page. It must be outside the vault: nothing is generated into the vault. |
|
|
105
105
|
| `--source-url URL` | Where a cited file can be read. The file's path in the repository is added to this address, and `{commit}` becomes the commit the page was built from. Without it, cited files are shown as names. |
|
|
106
106
|
| `--with-sources` | Put the text of the cited files into the page, as `serve` does, so they open inside the viewer. Leave it out when people who may not read the repository can open the page. |
|
|
107
|
+
| `--allow-missing-sources` | Do not stop when a cited file is not there. For a build that sees only part of the repository, such as a Docker build whose `.dockerignore` leaves files out. The command lists the files it did not find. |
|
|
107
108
|
|
|
108
109
|
The spec describes how the product works inside, and with `--with-sources` it carries source code. So treat the file like the repository:
|
|
109
110
|
|
|
@@ -137,7 +138,7 @@ export async function GET() {
|
|
|
137
138
|
}
|
|
138
139
|
```
|
|
139
140
|
|
|
140
|
-
The build needs Python
|
|
141
|
+
The build needs Python, so run it where the application is built (a CI job or a build stage), not inside a running container that has none. Without Git, or without the `.git` folder (a Docker build usually leaves it out), the page is built all the same, with an empty history and no commit to put into `{commit}`. The **Recent changes** page comes from the Git history: in a shallow clone it starts at the first commit that was fetched in full, and the command says so. In GitHub Actions, `fetch-depth: 0` on the checkout step gives it the whole history.
|
|
141
142
|
|
|
142
143
|
## What a feature page says
|
|
143
144
|
|
|
@@ -175,6 +176,25 @@ A rule's number is its name. It stays the same when the rule's text changes, a n
|
|
|
175
176
|
|
|
176
177
|
Things the code registers in one place and no single feature owns (background jobs, emails, routes, roles, plan limits) are listed in reference pages: one table with every item. A page cites the one to five files that decide its behavior. When one of them changes, the page is marked "Code changed since the last check" until someone re-reads it.
|
|
177
178
|
|
|
179
|
+
### Big features and several apps
|
|
180
|
+
|
|
181
|
+
A feature that is too big for one page (more than about twenty rules, or several kinds of use, such as direct, group and support chats) has **parts**. The vault then has three levels: the area, the feature, and its parts. The feature's page says what it is for and lists its parts; each part is a page of the same shape with its own rules. The viewer shows the parts under their feature.
|
|
182
|
+
|
|
183
|
+
When the product has several clients, such as a web app and a mobile app, a feature page has one more section, `## Where the apps differ`: a table with what each client does, one row per difference.
|
|
184
|
+
|
|
185
|
+
### How deep the first map goes
|
|
186
|
+
|
|
187
|
+
The first map starts with an inventory: every page or screen, every server procedure, every job and email, counted from the place that registers them. The agent then takes one feature at a time and finishes it before starting the next. A page whose feature was not read to the end is marked **Partial** and says what is left. A first map of a large project is therefore a number of finished features and a list of the ones not started, not a thin page about everything.
|
|
188
|
+
|
|
189
|
+
One command shows what is still missing, by reading the code and the pages:
|
|
190
|
+
|
|
191
|
+
```sh
|
|
192
|
+
project-spec coverage # every folder, the least covered first
|
|
193
|
+
project-spec coverage --path src/features/chat # one feature: each message and limit no page mentions
|
|
194
|
+
```
|
|
195
|
+
|
|
196
|
+
Without the package: `python3 <skill>/scripts/coverage.py --repo .`. For each folder it prints how many files a page cites, and how many messages a person can see and how many named limits are on no page. It is a list of places to read, not a score: some of what it lists is left out of the spec on purpose.
|
|
197
|
+
|
|
178
198
|
## Why use this?
|
|
179
199
|
|
|
180
200
|
- **Onboarding:** follow a workflow across UI, API, policy, persistence and tests without rediscovering the same paths every session.
|
|
@@ -197,6 +217,7 @@ The package is [`project-spec`](https://www.npmjs.com/package/project-spec) on t
|
|
|
197
217
|
npx project-spec install --claude # or --codex, or --dir knowledge/tooling
|
|
198
218
|
npx project-spec serve # the viewer for the project in the current folder
|
|
199
219
|
npx project-spec build --out FILE # the same viewer as one file, to show on a server
|
|
220
|
+
npx project-spec coverage # what the code has that no spec page mentions yet
|
|
200
221
|
```
|
|
201
222
|
|
|
202
223
|
Then ask your agent to use the `project-knowledge` skill ([example prompts](#example-prompts)).
|
package/bin/project-spec.mjs
CHANGED
|
@@ -15,6 +15,7 @@ const MARKER = '.project-spec.json'; // written into every copy: which version i
|
|
|
15
15
|
const SKIP = new Set(['__pycache__', '.DS_Store', MARKER]);
|
|
16
16
|
const KNOWLEDGE = new Set(['serve', 'build', 'check', 'snapshot']);
|
|
17
17
|
const MAINTAIN = new Set(['scan', 'receipt', 'finish']);
|
|
18
|
+
const COVERAGE = 'coverage'; // its own helper, which takes no command word
|
|
18
19
|
// Skills earlier versions shipped; project-knowledge and spec-change took over what they did.
|
|
19
20
|
const RETIRED = ['spec-audit', 'spec-investigation', 'spec-plan'];
|
|
20
21
|
// Folders inside a project where a copy of the skills may live.
|
|
@@ -25,10 +26,12 @@ const USAGE = `project-spec ${PACKAGE.version}
|
|
|
25
26
|
Read the spec
|
|
26
27
|
project-spec serve [--repo DIR] [--vault DIR] [--port N] [--no-open]
|
|
27
28
|
Open the viewer for the repository (default: the current folder).
|
|
28
|
-
project-spec build --out FILE [--source-url URL] [--with-sources] [--
|
|
29
|
+
project-spec build --out FILE [--source-url URL] [--with-sources] [--allow-missing-sources]
|
|
29
30
|
Write the viewer as one HTML file, to show the spec on a server.
|
|
30
31
|
project-spec check | snapshot [options]
|
|
31
32
|
The knowledge helper; --repo defaults to the current folder.
|
|
33
|
+
project-spec coverage [--path FOLDER]... [--all]
|
|
34
|
+
What the code has that no spec page mentions yet: folders, messages and limits.
|
|
32
35
|
project-spec scan | receipt | finish [options]
|
|
33
36
|
The sync helper; --repo defaults to the current folder.
|
|
34
37
|
|
|
@@ -253,7 +256,7 @@ function python(script, command, args) {
|
|
|
253
256
|
if (!flag(args, '--no-open') && !args.includes('--open')) args.push('--open');
|
|
254
257
|
}
|
|
255
258
|
const exe = process.env.PROJECT_SPEC_PYTHON || (process.platform === 'win32' ? 'python' : 'python3');
|
|
256
|
-
const full = [path.join(SKILLS, 'project-knowledge', 'scripts', script), command, ...args];
|
|
259
|
+
const full = [path.join(SKILLS, 'project-knowledge', 'scripts', script), ...(command === COVERAGE ? [] : [command]), ...args];
|
|
257
260
|
if (process.env.PROJECT_SPEC_DRY_RUN) return console.log([exe, ...full].join(' ')), 0;
|
|
258
261
|
const run = spawnSync(exe, full, { stdio: 'inherit' });
|
|
259
262
|
if (run.error) throw new Problem(`could not start ${exe}: ${run.error.message}. The helpers need Python 3.10 or newer; set PROJECT_SPEC_PYTHON to its path.`);
|
|
@@ -269,6 +272,7 @@ function main(argv) {
|
|
|
269
272
|
if (command === 'update') return update(args);
|
|
270
273
|
if (KNOWLEDGE.has(command)) return python('knowledge.py', command, args);
|
|
271
274
|
if (MAINTAIN.has(command)) return python('maintain.py', command, args);
|
|
275
|
+
if (command === COVERAGE) return python('coverage.py', command, args);
|
|
272
276
|
throw new Problem(`unknown command: ${command}\n\n${USAGE}`);
|
|
273
277
|
}
|
|
274
278
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "project-spec",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.9.0",
|
|
4
4
|
"description": "Agent skills that read a repository and keep a spec people can read: one page per feature, every business rule numbered. Local viewer and one command to install and update the skills.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"agent-skills",
|
|
@@ -38,11 +38,11 @@ The full property list, section layout and example are in [gaps.md](references/g
|
|
|
38
38
|
|
|
39
39
|
Natural-language requests are fine; these are modes, not a required CLI.
|
|
40
40
|
|
|
41
|
-
- **map** (a new vault):
|
|
41
|
+
- **map** (a new vault): take the inventory of what the product has, then go through its features one at a time and finish each before starting the next ([Research before writing](#research-before-writing)). A first map is measured by how many features are finished, not by how many pages exist: thirty finished features and a list of forty not started is a better first map than seventy thin pages. A pass that runs out of time says on the run page which features are finished, which are partial and which were not started, and never calls itself full.
|
|
42
42
|
- **update**: revisit claims affected since the previous run, preserve manually written knowledge, add discoveries, and fill in what the pages you touch are missing (see [Leave each page better](#leave-each-page-better)).
|
|
43
43
|
- **sync**: reconcile changes from all contributors since the last reviewed Git checkpoint. Read [the sync workflow](references/sync.md); a sync with work you cannot accept yet (for example another contributor's uncommitted files) uses `deferred` and ends as a partial finish with exit status 3 and the checkpoint unmoved.
|
|
44
44
|
- **merge**: reconcile two contributors' vault updates from their common Git base. Read [the merge workflow](references/merge.md).
|
|
45
|
-
- **explore <area or question>**: deepen a bounded area and link the result into the vault. Deepening an area means its feature notes first: what each one is for and every rule a person would look up, with its values. Then the lists the code keeps in one place, such as its background jobs. A request to "fill in", "complete"
|
|
45
|
+
- **explore <area or question>**: deepen a bounded area and link the result into the vault. Deepening an area means its feature notes first: what each one is for and every rule a person would look up, with its values, until [the feature is finished](#when-a-feature-is-finished). Then the lists the code keeps in one place, such as its background jobs. A request to "fill in", "complete", "backfill" or "go deeper" means this mode, and so does a reader saying a page is thin.
|
|
46
46
|
- **gaps**: work the queue. List gaps by state; re-verify open gaps against the current code; every kind of gap is rechecked, proposals and questions about unexplained behavior included, because the recheck asks only whether the page's `Current behavior` still describes the code (`confirmed (still holds)` or `not reproduced (no longer holds)`) and never whether a proposal is worth doing, which is the person's decision. For a proposal that usually means confirming the capability is absent. Record each result in the gap's `Recheck` section with `recheck`, `rechecked_by` (`agent` for checks you ran), `rechecked_on`; set `recheck: "outdated"` on a gap whose cited source changed since its recorded recheck (keeping the other two fields); set `done` where the code now closes the question, recording that reading as a `not-reproduced` recheck by `agent` ([closing a gap as done](references/gaps.md#closing-a-gap-as-done)); mark duplicates. Then hand the `needs-decision` gaps to people with the options you found in `Discussion`. Never decide one yourself. When a person decides in the conversation, record it with their name; if you do not know their name, ask rather than writing `agent`.
|
|
47
47
|
- **audit <document or feature>**: compare written requirements with the code, keeping what the code satisfies apart from what needs action. Read [the audit guide](references/audit.md). In a vault each finding that needs action becomes its own gap; without a vault the audit is a report.
|
|
48
48
|
- **investigate <question>** or **recheck <gap>**: answer one question about behavior from repository and runtime evidence, or re-check a single gap and write the result into it. Read [the investigation guide](references/investigate.md). It works without a vault too.
|
|
@@ -58,9 +58,23 @@ Write vault content in English unless the user asks for another language, whatev
|
|
|
58
58
|
|
|
59
59
|
1. Read the repository's instructions and any task coordination they require. Preserve existing edits. Record the exact revision and whether you read uncommitted files. Application source is read-only: this skill writes only inside the vault.
|
|
60
60
|
2. Inspect manifests, entrypoints, routes, schema and migrations, services and tests. Skip dependencies, build output, credential files, raw logs and customer data. Never use the vault itself as evidence about the application.
|
|
61
|
-
3.
|
|
62
|
-
4.
|
|
63
|
-
5.
|
|
61
|
+
3. **Take the inventory before writing a page.** List what the product has from the places that register it, every item and not a sample: the pages or screens of every client (web, mobile, admin), the API procedures or endpoints, the jobs, the emails. Count them. Run `coverage` (below) for the size of each folder. Group the items into features and write that list on the run page with each feature's size: its entry points, its folders, its lines. This list is the plan and the ledger of the whole map. A directory name establishes neither a feature nor a behavior, so check a group against what its entry points do.
|
|
62
|
+
4. **Split a big feature before writing it.** One page holds what a person reads in one sitting: about twenty rules. A feature with more than about eight entry points, more than about 2,000 lines of its own code, or several kinds of user or flow (direct, group and support chats) will not fit, and squeezing it into one page is how fifty rules become thirteen. Give it a feature note and one note per part ("A feature with parts" under [Write the spec](#write-the-spec)).
|
|
63
|
+
5. **Go deep before going wide.** Take one feature and finish it before starting the next. Read every one of its entry points on every client and follow each to the file that decides the behavior (the schema, the service, the policy, the component). Read its rules off the code: a `max(60)` in a schema, a `disabled={items.length === 0}` on a button, a `CHECK` constraint, a default value, a permission check, every `throw` and every refusal. Read the words a person sees too, the labels, the error messages and what an empty list says, and quote them. For the things the code registers in one place (the queue and cron registrations, the route table, the email templates, the roles, the flags), enumerate from that place, every item.
|
|
64
|
+
6. **Read each client.** When the product has more than one client, read the feature in each of them. Two clients calling the same server procedure does not make them the same: one may lack a screen, a check or a confirmation. Write what differs on the feature's page ([Write the spec](#write-the-spec)); a difference that looks unintended is also a gap.
|
|
65
|
+
7. Track what was read, merely discovered, excluded or changing in each area. Discovery is not review, and a note count is not completeness. When the session will not cover everything, stop at the end of a feature, not in the middle of all of them.
|
|
66
|
+
|
|
67
|
+
### When a feature is finished
|
|
68
|
+
|
|
69
|
+
A feature is finished when nothing is left on its list, not when its page looks complete. Run the report for its folders:
|
|
70
|
+
|
|
71
|
+
```sh
|
|
72
|
+
python3 "$SKILL_DIR/scripts/coverage.py" --repo "$REPO" --vault "$VAULT" --path src/features/chat --path mobile/src/chat
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
It prints the files there that no page cites, and each sentence a person can see and each named number (`CHAT_MAX_MEDIA = 10`) that no page mentions. Go through that list. Each item is one of three things: a rule that belongs on the page (add it, quoting the message or naming the number), something the spec leaves out on purpose (a control that only opens or closes, a label, a log line), or code you have not read (read it). Run it again until what is left is only the second kind. Without `--path` the report ranks every folder of the repository by how much it has that no page mentions, which is how you find the features nobody wrote down at all.
|
|
76
|
+
|
|
77
|
+
The report is a list of places to read, not a score, and it cannot see a rule that has no message and no named number, so it does not replace reading the entry points. A page whose feature is not finished has `status: "partial"` and ends with a line saying what is left ("Not read yet: the support chat screens, the mobile app"); `current` means the whole feature was read. Put the report's first lines for each finished feature on the run page.
|
|
64
78
|
|
|
65
79
|
Delegate only when the user or repository instructions allow it; give each helper a disjoint area and integrate their evidence before accepting a claim.
|
|
66
80
|
|
|
@@ -76,13 +90,16 @@ Write for the person who asks "what does this do, and what are its rules", not f
|
|
|
76
90
|
3. `## Fields`, when the feature has a form: a table with one row per field (`Field`, `Required`, `Rule`, `Message when refused`).
|
|
77
91
|
4. `## Business rules`: a table with one numbered rule per row (`ID`, `Rule`): who may do it, when an action is available, disabled or hidden, limits and defaults, what is kept, sent or deleted, and when. One or two sentences per rule, with the value. A field's rule is in `Fields` and is not repeated here.
|
|
78
92
|
|
|
93
|
+
5. `## Where the apps differ`, when the product has several clients: a table with one row per difference (`What`, then one column per client, such as `Web` and `Mobile`). Say what each client does, in a few words; "same" is not a row. When the clients were compared and nothing differs, write that one sentence instead of the table, and say which clients were compared.
|
|
94
|
+
|
|
79
95
|
Use these headings as written and in this order, and leave out a section the feature does not have. Links to the notes it depends on go in the sentence or rule where they matter. Everything about the feature is on this page: do not move its fields, its actions or who may use them to another page, because a reader of Profile should not need four pages to learn what Profile does. A note is as long as its rules need; split a long note by feature or flow, never by kind of fact.
|
|
96
|
+
- **A feature with parts.** The vault has three levels and no more: an area, its features, and the parts of a feature that is too big for one page. A part is a feature note with `parent: "<id of the feature note>"`, in the same area, about one flow or one kind of use: "Group chats", "Support chats", "Sending a message". The feature's own note stays short: what the feature is for, how its parts fit together, the rules that hold in every part, and `## Parts`, a list with a link to each part and one line saying what it covers. Each rule is written once, in the part it belongs to, with the part's own rule word (`BR-GROUPCHAT-01`). The area page lists the feature, not its parts. Split when the page would pass about twenty rules or the feature has several kinds of user or flow; do not split a feature that fits one page, and never make a part of a part. `check` rejects a part whose feature does not link to it.
|
|
80
97
|
- **A rule's number is its name.** Write it `BR-<FEATURE>-<NN>`, for example `BR-PROFILE-01`: a short upper-case word for the feature, the same in all rules of the note and used by no other note, then a number counting up from 01. People quote these numbers in tickets, tests and conversations, so a rule keeps its number when its text changes, a new rule gets the next free number, and the number of a rule that no longer exists is never used again. Never renumber to close a hole or to reorder. `check` rejects a number that appears twice.
|
|
81
98
|
- **Write down the business rules, with their values.** The concrete rule is the content and a description of the mechanism is not enough: "a name is required and at most 60 characters, and longer input is refused with 'Name is too long'", not "the form validates the name". For every form, action and job say what is required, the limits (length, range, count, size, format, uniqueness), defaults, rounding and time zones, when an action is available, disabled or hidden ("Send is disabled while the list is empty"), what an empty list shows, what needs a confirmation, which error the person sees, and who may do it. Leave out controls that only open, close, go somewhere or try again: an action is worth a line when it changes something or has a condition. When the browser and the server check different things, write both; when they disagree, open a `conflict` gap. A note that says "validates input" or "uses a worker" without the rules it applies is not finished.
|
|
82
99
|
- **Say it in the product's words.** "Saving is refused", "the list reloads", "only owners see it", not "the mutation rejects", "refetches", "the RPC requires a nonempty team ID". Name a constant, a job or a route in code formatting when someone would search for it; keep the rest of the code's vocabulary out.
|
|
83
100
|
- **The summary says what the feature is for**, in one or two plain sentences, as a person who never opens the code would say it: no file names, symbols, status codes or words like "verified", and not a rule picked from the page ("The form accepts 1-120 characters" is a rule, not a summary). The viewer shows it under the title and in every list. It is still a claim about the code: it says no more than the page proves. Do not infer design rationale; cite a comment or ADR in the source, or say it is unknown.
|
|
84
101
|
- **A reference note** is a list of things the code registers in one place and that belong to no single feature: the background jobs, the emails and notifications, the routes, the roles and what each may do, the plan limits, the flags, the states of a record. It is one table, one row per item, every item, with the count and where the list comes from ("all 7 jobs registered in `worker/index.ts`"), so a later sync can tell when one is added; put that place in `sources`. A cell holds a value in a few words, two short sentences at most; an item that needs more has a feature note, and the row links to it. Do not write a list note for form fields or for the actions on a screen (they are in the feature's note), for a kind the area has only one or two of, or for a kind it does not have. Each rule is written once: in the feature's note when it belongs to a feature, in the list when it does not.
|
|
85
|
-
- **Cite the few files that decide the behavior.** `sources` holds the files where the rule is written (the schema, the service, the component), usually one to five. Every source is a tripwire: when it changes, the page is marked for re-reading. So leave out shared plumbing (the router, a layout, the base client), files that merely sit on the path, and tests unless a test is what the claim rests on. In the text, name the symbol a rule comes from (`INVITE_TTL_DAYS`) and link a file where it helps, without line numbers: a line number points at the wrong line after the next edit, a name still finds the place. When files are renamed or split, `check` reports the missing path; point `sources` at where the rule lives now and re-read the page.
|
|
102
|
+
- **Cite the few files that decide the behavior.** `sources` holds the files where the rule is written (the schema, the service, the component), usually one to five. A page that needs many more to back its rules is usually a feature that should have parts. Every source is a tripwire: when it changes, the page is marked for re-reading. So leave out shared plumbing (the router, a layout, the base client), files that merely sit on the path, and tests unless a test is what the claim rests on. In the text, name the symbol a rule comes from (`INVITE_TTL_DAYS`) and link a file where it helps, without line numbers: a line number points at the wrong line after the next edit, a name still finds the place. When files are renamed or split, `check` reports the missing path; point `sources` at where the rule lives now and re-read the page.
|
|
86
103
|
- **What you ran goes on the run page**, not in the note: commands, PASS, FAIL or NOT RUN, the revision. A note has no "Verification" section and no line naming its area (the frontmatter has it and the viewer shows it). Reading a test does not mean it passes today, so do not write that it does.
|
|
87
104
|
- Link related spec pages with `[[notes/example|label]]` in the sentence that needs them, and say how they relate (depends on, constrains, produces, supersedes). A note belongs to one area; give it a second only when both areas' readers would look for it there. Every substantive page needs an area and a path from the index. Add cross-area links for a reason, not for graph density.
|
|
88
105
|
- A requirements document or ticket is documented intent. It can be quoted in a gap's `Evidence`; it is not a source for what the code does, so never list it in a spec page's `sources` or describe its labels in spec text.
|
|
@@ -122,7 +139,7 @@ python3 "$SKILL_DIR/scripts/knowledge.py" check --repo "$REPO" --vault "$VAULT"
|
|
|
122
139
|
python3 "$SKILL_DIR/scripts/knowledge.py" snapshot --repo "$REPO" --vault "$VAULT" --notes note-id area-id
|
|
123
140
|
```
|
|
124
141
|
|
|
125
|
-
`check` validates structure and freshness; it cannot judge prose or coverage. `--strict` turns freshness warnings into a failing exit status, for CI. In a working tree shared with someone else's uncommitted work, `--strict --ignore-dirty` still prints warnings for sources with uncommitted changes but does not fail on them when the accepted fingerprint equals the file at `HEAD` (the only drift is uncommitted); committed drift still fails, and it is not for CI ([details](references/format.md)). `serve` shows the vault in a browser and writes nothing. `snapshot` accepts source fingerprints as reviewed; run it only for pages you actually re-read against their sources, never to clear a warning. An initial reviewed pass may use `--all`.
|
|
142
|
+
`check` validates structure and freshness; it cannot judge prose or coverage. `coverage.py` lists what the code has that no page mentions ([When a feature is finished](#when-a-feature-is-finished)). `--strict` turns freshness warnings into a failing exit status, for CI. In a working tree shared with someone else's uncommitted work, `--strict --ignore-dirty` still prints warnings for sources with uncommitted changes but does not fail on them when the accepted fingerprint equals the file at `HEAD` (the only drift is uncommitted); committed drift still fails, and it is not for CI ([details](references/format.md)). `serve` shows the vault in a browser and writes nothing. `snapshot` accepts source fingerprints as reviewed; run it only for pages you actually re-read against their sources, never to clear a warning. An initial reviewed pass may use `--all`.
|
|
126
143
|
|
|
127
144
|
To read the result, run `python3 <skill>/scripts/knowledge.py serve --repo <repository> --open` (add `--vault` when the repository has several vaults under `knowledge/`), or open the folder as an Obsidian vault. The command is long, so when you hand over offer to give it a short name in the project: a `spec` script in `package.json` (`project-spec serve` when the project has the `project-spec-skills` package, otherwise the full command), a `spec` target in a `Makefile`, or a shell alias. That file is outside the vault, so add it only when the person agrees. The viewer is one page on `127.0.0.1`; it reads the vault and the files the pages cite from disk on every page load, so it needs no build and shows the current state after a reload. It shows a note as written, under its title and summary, with the source files folded away at the end; an area page shows its notes with their summaries. So the first paragraph and the summary are what most readers see. Its **Copy for AI** turns a page and the reader's own words into a request for an agent that names the page file. It only copies text, so an agent that receives one still follows the skill it names, and a recheck or decision the reader states there is recorded under the reader's name, never as the agent's. If the vault has errors the page lists them instead. When the person wants the spec on a server for people without the repository, `knowledge.py build --repo <repository> --out <file outside the vault>` writes that same viewer as one file ([options](references/format.md)); run it only when asked, never commit the file into the vault, and leave `--with-sources` out unless the person says everyone who can open the page may read the code.
|
|
128
145
|
|
|
@@ -131,9 +148,11 @@ Before handing over, read what you wrote as the person who will use it, not as i
|
|
|
131
148
|
- Does the note start by saying what the feature is for and who uses it, before any rule?
|
|
132
149
|
- Is each rule one numbered row a person could quote, with its value, and did every rule that was already there keep its number?
|
|
133
150
|
- Could someone answer "how long may a name be", "when is this button disabled" or "which jobs run at night" from the page without opening the code? If the answer is in the code you read but not on the page, add it.
|
|
151
|
+
- Did `coverage --path` for the feature's folders end with only things the spec leaves out on purpose? Is every page whose feature is not finished marked `partial`, with what is left?
|
|
152
|
+
- Does a feature with more than about twenty rules have parts, and does a product with several clients say on each feature page where they differ?
|
|
134
153
|
- Is everything about one feature on its own page, or does the reader have to collect it from several?
|
|
135
154
|
- Would a person who never opens the code understand every sentence? Would the page survive a file being renamed?
|
|
136
155
|
- Does the area page only map, with its missing lists named?
|
|
137
156
|
- Is every doubt a gap about the product, and nothing in the queue a to-do of the vault itself?
|
|
138
157
|
|
|
139
|
-
Then validate and, when tools allow, open the viewer. Report real counts, coverage
|
|
158
|
+
Then validate and, when tools allow, open the viewer. Report real counts: how many features of the inventory are finished, partial and not started, the first lines of the `coverage` report for the whole repository, the checks you ran and how to open the result. Do not describe a map as full or complete while the inventory has features that are not finished. Keep structural validation, application tests and GUI checks separate. Committing and pushing are left to the user.
|
|
@@ -228,6 +228,10 @@
|
|
|
228
228
|
padding-left: 40px;
|
|
229
229
|
color: var(--muted);
|
|
230
230
|
}
|
|
231
|
+
.tree-row.part {
|
|
232
|
+
padding-left: 58px;
|
|
233
|
+
font-size: 0.94em;
|
|
234
|
+
}
|
|
231
235
|
|
|
232
236
|
/* Top bar: where you are, and what you can do with the page. */
|
|
233
237
|
.topbar {
|
|
@@ -1675,6 +1679,20 @@
|
|
|
1675
1679
|
const i = STATES.indexOf(stateOf(n));
|
|
1676
1680
|
return i < 0 ? STATES.length : i;
|
|
1677
1681
|
};
|
|
1682
|
+
// Priority is free text. The usual words and P0-P4 sort from most to least urgent, any other
|
|
1683
|
+
// value after them, and a gap without a priority last.
|
|
1684
|
+
const PRIORITY_RANKS = { critical: 0, urgent: 0, highest: 0, high: 1, medium: 2, normal: 2, low: 3, lowest: 4 };
|
|
1685
|
+
const priorityOf = (n) => str(n.priority).trim().toLowerCase();
|
|
1686
|
+
const priorityRank = (n) => {
|
|
1687
|
+
const p = priorityOf(n),
|
|
1688
|
+
digit = /^p?([0-4])$/.exec(p);
|
|
1689
|
+
if (!p) return 6;
|
|
1690
|
+
if (digit) return +digit[1];
|
|
1691
|
+
return Object.prototype.hasOwnProperty.call(PRIORITY_RANKS, p) ? PRIORITY_RANKS[p] : 5;
|
|
1692
|
+
};
|
|
1693
|
+
// The order of every list of gaps: by state, then the most urgent first, then by title.
|
|
1694
|
+
const byGapOrder = (a, b) =>
|
|
1695
|
+
stateRank(a) - stateRank(b) || priorityRank(a) - priorityRank(b) || byTitle(a, b);
|
|
1678
1696
|
const stateClass = (n) =>
|
|
1679
1697
|
isGap(n)
|
|
1680
1698
|
? "st-" + stateOf(n)
|
|
@@ -2181,7 +2199,7 @@
|
|
|
2181
2199
|
if (g && isGap(g)) out.set(g.id, g);
|
|
2182
2200
|
});
|
|
2183
2201
|
return [...out.values()].sort(
|
|
2184
|
-
|
|
2202
|
+
byGapOrder,
|
|
2185
2203
|
);
|
|
2186
2204
|
}
|
|
2187
2205
|
const planOf = (g) => nodes.find((p) => isPlan(p) && p.gap === g.id);
|
|
@@ -2256,11 +2274,18 @@
|
|
|
2256
2274
|
}
|
|
2257
2275
|
const primaryArea = (n) => (n.areas || [])[0] || "";
|
|
2258
2276
|
const isFeature = (n) => n.type === "note" && n.status !== "superseded";
|
|
2277
|
+
// A part of a feature: a note whose `parent` names another note. One level only, so the tree is
|
|
2278
|
+
// area, feature, part; a parent that is missing or is itself a part is ignored.
|
|
2279
|
+
const parentOf = (n) => {
|
|
2280
|
+
const p = n.type === "note" && str(n.parent) ? byId.get(n.parent) : null;
|
|
2281
|
+
return p && p !== n && isFeature(p) && !str(p.parent) ? p : null;
|
|
2282
|
+
};
|
|
2283
|
+
const partsOf = (n) => nodes.filter((p) => isFeature(p) && parentOf(p) === n).sort(byTitle);
|
|
2259
2284
|
const featuresOf = (area) =>
|
|
2260
|
-
nodes.filter((n) => isFeature(n) && primaryArea(n) === area.id).sort(byTitle);
|
|
2285
|
+
nodes.filter((n) => isFeature(n) && primaryArea(n) === area.id && !parentOf(n)).sort(byTitle);
|
|
2261
2286
|
const looseFeatures = () =>
|
|
2262
2287
|
nodes
|
|
2263
|
-
.filter((n) => isFeature(n) && byId.get(primaryArea(n))?.type !== "area")
|
|
2288
|
+
.filter((n) => isFeature(n) && byId.get(primaryArea(n))?.type !== "area" && !parentOf(n))
|
|
2264
2289
|
.sort(byTitle);
|
|
2265
2290
|
const waiting = (n) =>
|
|
2266
2291
|
gapsAbout(n).filter((g) =>
|
|
@@ -2374,15 +2399,23 @@
|
|
|
2374
2399
|
});
|
|
2375
2400
|
const tree = $("#tree");
|
|
2376
2401
|
tree.replaceChildren();
|
|
2377
|
-
const noteRow = (f) => {
|
|
2378
|
-
const b = button(null, "tree-row note", () => openNode(f)),
|
|
2379
|
-
open = waiting(f).length
|
|
2402
|
+
const noteRow = (f, cls, folded) => {
|
|
2403
|
+
const b = button(null, "tree-row note" + (cls ? " " + cls : ""), () => openNode(f)),
|
|
2404
|
+
open = waiting(f).length,
|
|
2405
|
+
meta = [open && open + " open", folded && plural(folded, "part")].filter(Boolean).join(" · ");
|
|
2380
2406
|
b.append(el("span", f.title, "label"));
|
|
2381
2407
|
b.title = f.title; // the row shows one line; the full name is a hover away
|
|
2382
|
-
if (
|
|
2408
|
+
if (meta) b.append(el("span", meta, "meta"));
|
|
2383
2409
|
b.setAttribute("aria-current", selected === f ? "page" : "false");
|
|
2384
2410
|
return b;
|
|
2385
2411
|
};
|
|
2412
|
+
// A feature's parts are listed under it while the reader is on the feature or on one of them.
|
|
2413
|
+
const featureRows = (f) => {
|
|
2414
|
+
const parts = partsOf(f),
|
|
2415
|
+
unfolded = parts.length && (selected === f || parts.includes(selected));
|
|
2416
|
+
tree.append(noteRow(f, "", unfolded ? 0 : parts.length));
|
|
2417
|
+
if (unfolded) parts.forEach((p) => tree.append(noteRow(p, "part")));
|
|
2418
|
+
};
|
|
2386
2419
|
areas.forEach((a) => {
|
|
2387
2420
|
const open = openAreas.has(a.id),
|
|
2388
2421
|
line = el("div", null, "tree-area"),
|
|
@@ -2403,12 +2436,12 @@
|
|
|
2403
2436
|
name.setAttribute("aria-current", selected === a ? "page" : "false");
|
|
2404
2437
|
line.append(twist, name);
|
|
2405
2438
|
tree.append(line);
|
|
2406
|
-
if (open) featuresOf(a).forEach(
|
|
2439
|
+
if (open) featuresOf(a).forEach(featureRows);
|
|
2407
2440
|
});
|
|
2408
2441
|
const loose = looseFeatures();
|
|
2409
2442
|
if (loose.length) {
|
|
2410
2443
|
tree.append(el("div", "Other", "eyebrow"));
|
|
2411
|
-
loose.forEach(
|
|
2444
|
+
loose.forEach(featureRows);
|
|
2412
2445
|
}
|
|
2413
2446
|
if (!areas.length && !loose.length) tree.append(el("p", "No pages yet.", "empty"));
|
|
2414
2447
|
}
|
|
@@ -2424,6 +2457,7 @@
|
|
|
2424
2457
|
if (n) {
|
|
2425
2458
|
const parent = n.type === "note" ? byId.get(primaryArea(n)) : null;
|
|
2426
2459
|
if (parent && parent.type === "area") parts.push([parent.title, { name: "page", id: parent.id }]);
|
|
2460
|
+
if (parentOf(n)) parts.push([parentOf(n).title, { name: "page", id: parentOf(n).id }]);
|
|
2427
2461
|
if (isGap(n) || isPlan(n)) parts.push(["Gaps", { name: "gaps", id: "" }]);
|
|
2428
2462
|
if (isPlan(n) && byId.get(n.gap)) parts.push([byId.get(n.gap).title, { name: "page", id: n.gap }]);
|
|
2429
2463
|
parts.push([n.title, null]);
|
|
@@ -2478,6 +2512,16 @@
|
|
|
2478
2512
|
open = waiting(f).length;
|
|
2479
2513
|
item.append(button(f.title, "rowlink", () => openNode(f)));
|
|
2480
2514
|
if (str(f.summary).trim()) item.append(el("p", f.summary, "muted"));
|
|
2515
|
+
const parts = partsOf(f);
|
|
2516
|
+
if (parts.length) {
|
|
2517
|
+
const line = el("p", null, "muted small parts");
|
|
2518
|
+
line.append(el("span", plural(parts.length, "part") + ": "));
|
|
2519
|
+
parts.forEach((part, i) => {
|
|
2520
|
+
if (i) line.append(document.createTextNode(", "));
|
|
2521
|
+
line.append(button(part.title, "wiki", () => openNode(part)));
|
|
2522
|
+
});
|
|
2523
|
+
item.append(line);
|
|
2524
|
+
}
|
|
2481
2525
|
if (open) item.append(el("span", plural(open, "open question"), "muted small"));
|
|
2482
2526
|
box.append(item);
|
|
2483
2527
|
});
|
|
@@ -2541,6 +2585,8 @@
|
|
|
2541
2585
|
path.append(button(projectTitle, null, () => go({ name: "home", id: "" })));
|
|
2542
2586
|
const parent = n.type === "note" ? byId.get(primaryArea(n)) : null;
|
|
2543
2587
|
if (parent && parent.type === "area") path.append(button(parent.title, null, () => openNode(parent)));
|
|
2588
|
+
const feature = parentOf(n);
|
|
2589
|
+
if (feature) path.append(button(feature.title, null, () => openNode(feature)));
|
|
2544
2590
|
if (isGap(n) || isPlan(n)) path.append(button("Gaps", null, () => go({ name: "gaps", id: "" })));
|
|
2545
2591
|
path.append(el("span", n.title, "now"));
|
|
2546
2592
|
card.append(path);
|
|
@@ -2551,7 +2597,7 @@
|
|
|
2551
2597
|
// Pages this one links to or is linked from: a short list beside the text, not a block inside it.
|
|
2552
2598
|
if (layerOf(n) === "spec") {
|
|
2553
2599
|
const related = neighbours(n, "spec").filter(
|
|
2554
|
-
(r) => r.id !== parent?.id && !(n.type === "area" && primaryArea(r) === n.id),
|
|
2600
|
+
(r) => r.id !== parent?.id && r !== feature && !(n.type === "area" && primaryArea(r) === n.id),
|
|
2555
2601
|
);
|
|
2556
2602
|
if (related.length) {
|
|
2557
2603
|
const list = el("div", null, "recent"),
|
|
@@ -2686,7 +2732,8 @@
|
|
|
2686
2732
|
|
|
2687
2733
|
let gapState = "all",
|
|
2688
2734
|
gapArea = "",
|
|
2689
|
-
gapKind = ""
|
|
2735
|
+
gapKind = "",
|
|
2736
|
+
gapPriority = ""; // "" for all, "none" for gaps without one, otherwise the value
|
|
2690
2737
|
const selectOf = (label, options, value, pick) => {
|
|
2691
2738
|
const s = el("select");
|
|
2692
2739
|
s.setAttribute("aria-label", label);
|
|
@@ -2708,7 +2755,11 @@
|
|
|
2708
2755
|
head.title = "Questions about what is missing, unverified or could be added. Each one is rechecked and then decided by a person.";
|
|
2709
2756
|
box.append(head);
|
|
2710
2757
|
const pool = nodes.filter(
|
|
2711
|
-
(n) =>
|
|
2758
|
+
(n) =>
|
|
2759
|
+
isGap(n) &&
|
|
2760
|
+
inArea(n, gapArea) &&
|
|
2761
|
+
(!gapKind || n.kind === gapKind) &&
|
|
2762
|
+
(!gapPriority || (priorityOf(n) || "none") === gapPriority),
|
|
2712
2763
|
),
|
|
2713
2764
|
again = () => show();
|
|
2714
2765
|
head.append(
|
|
@@ -2743,9 +2794,29 @@
|
|
|
2743
2794
|
again();
|
|
2744
2795
|
}),
|
|
2745
2796
|
);
|
|
2797
|
+
// Only the priorities the queue has, most urgent first; nothing to pick when no gap has one.
|
|
2798
|
+
const priorities = [...new Set(nodes.filter(isGap).map(priorityOf).filter(Boolean))].sort(
|
|
2799
|
+
(a, b) => priorityRank({ priority: a }) - priorityRank({ priority: b }) || a.localeCompare(b),
|
|
2800
|
+
);
|
|
2801
|
+
if (priorities.length || gapPriority)
|
|
2802
|
+
head.append(
|
|
2803
|
+
selectOf(
|
|
2804
|
+
"Priority",
|
|
2805
|
+
[
|
|
2806
|
+
["", "All priorities"],
|
|
2807
|
+
...priorities.map((p) => [p, p[0].toUpperCase() + p.slice(1) + " priority"]),
|
|
2808
|
+
["none", "No priority"],
|
|
2809
|
+
],
|
|
2810
|
+
gapPriority,
|
|
2811
|
+
(v) => {
|
|
2812
|
+
gapPriority = v;
|
|
2813
|
+
again();
|
|
2814
|
+
},
|
|
2815
|
+
),
|
|
2816
|
+
);
|
|
2746
2817
|
const shown = pool
|
|
2747
2818
|
.filter((g) => gapState === "all" || stateOf(g) === gapState)
|
|
2748
|
-
.sort(
|
|
2819
|
+
.sort(byGapOrder);
|
|
2749
2820
|
if (shown.length) {
|
|
2750
2821
|
const card = el("section", null, "card");
|
|
2751
2822
|
card.id = "gap-list";
|
|
@@ -2777,7 +2848,7 @@
|
|
|
2777
2848
|
}),
|
|
2778
2849
|
);
|
|
2779
2850
|
const shown = gaps
|
|
2780
|
-
? nodes.filter((n) => isGap(n) && inArea(n, mapArea)).sort(
|
|
2851
|
+
? nodes.filter((n) => isGap(n) && inArea(n, mapArea)).sort(byGapOrder)
|
|
2781
2852
|
: nodes.filter((n) => layerOf(n) === "spec" && inArea(n, mapArea));
|
|
2782
2853
|
draw(graphModel(shown), false);
|
|
2783
2854
|
}
|
|
@@ -3411,7 +3482,7 @@
|
|
|
3411
3482
|
? model.context.has(x.id)
|
|
3412
3483
|
? 1
|
|
3413
3484
|
: -1
|
|
3414
|
-
:
|
|
3485
|
+
: byGapOrder(x, y),
|
|
3415
3486
|
);
|
|
3416
3487
|
const cols = Math.max(
|
|
3417
3488
|
1,
|
|
@@ -54,8 +54,8 @@ Required on every page: `id`, `title`, `type`, `areas`, `sources`, `updated`. `s
|
|
|
54
54
|
|
|
55
55
|
- `id`: unique stable ASCII slug (`[a-z0-9-]+`).
|
|
56
56
|
- `status` on `area` and `note`: `current`, `partial`, `stale`, `superseded`. On `index`, `run` and `plan` additionally `planned`.
|
|
57
|
-
- `current`: checked against the current sources. This is not a runtime test or a deployment claim; say in prose which version and environment you checked.
|
|
58
|
-
- `partial`:
|
|
57
|
+
- `current`: the whole feature was read and the page checked against the current sources. This is not a runtime test or a deployment claim; say in prose which version and environment you checked.
|
|
58
|
+
- `partial`: some of the feature or area was not read. The page ends with a line saying what is left, and stays `partial` until the feature is finished.
|
|
59
59
|
- `stale`: its sources changed or disappeared and it has not been re-read.
|
|
60
60
|
- `superseded`: replaced by another page, kept as a pointer.
|
|
61
61
|
- `planned` (index, run, plan): describes intent not yet carried out.
|
|
@@ -63,6 +63,7 @@ Required on every page: `id`, `title`, `type`, `areas`, `sources`, `updated`. `s
|
|
|
63
63
|
- `areas`: existing **area IDs**. Each `note`, `gap` and `plan` needs at least one. Area pages may list themselves or leave empty; index and run pages may be empty.
|
|
64
64
|
- `sources`: existing files read as evidence, repository-relative; no directories, credentials, absolute paths or paths escaping the repo. A `current` note needs evidence; gaps, plans and maps may have none.
|
|
65
65
|
- `updated`: last substantive review date (`YYYY-MM-DD`).
|
|
66
|
+
- Optional `parent`, on a `note` only: the `id` of the feature note this note is a part of. The vault has three levels: area, feature, part. The parent is a note of the same first area that has no `parent` itself, and its body links to each of its parts; `check` rejects anything else. The viewer lists a part under its feature.
|
|
66
67
|
- Optional `tags`, `aliases`, `task_ids`: string arrays. Optional `summary` and `verification`: strings. An area's `summary` introduces it in the viewer; a note's `summary` is its plain-language statement of what the product does, shown first in the viewer's Read view (write one for every note; `check` does not require it, so older vaults still open).
|
|
67
68
|
|
|
68
69
|
Gap-only properties are errors on any other page type, `gap` is an error on a page that is not a plan, and `priority` and `confidence` are allowed only on gap and plan pages. String properties (and strings inside arrays) may not contain control characters, including line separators. Gap-only properties (`kind`, `recheck`, `rechecked_by`, `rechecked_on`, `decision`, `decided_by`, `decided_on`, `duplicate_of`, optional `priority`, `confidence`) and the plan-only `gap` property are defined in [gaps.md](gaps.md), with the rules for who may set each. Gap state (`needs-recheck`, `needs-decision`, or the decision value) is derived from them and never written into Markdown. A recheck has one meaning for every gap kind: does the `Current behavior` section still describe the code? `confirmed` means it does (confirmed, still holds; for a proposal, the capability is still absent) and `not-reproduced` means it no longer does; neither judges whether a proposal is worth doing, which is a person's decision (see [what a recheck means](gaps.md#what-a-recheck-means)). `recheck: "outdated"` means the cited sources changed after the last recorded recheck; `rechecked_by` and `rechecked_on` stay, the `Recheck` section must have text, and an open gap with it is `needs-recheck`.
|
|
@@ -81,8 +82,10 @@ A **feature note** is the one page for one screen, feature, flow or rule, and al
|
|
|
81
82
|
| Flow | `## How it works` | What a person does and what happens, in order. Left out when there is no flow. |
|
|
82
83
|
| Fields | `## Fields` | One row per field of the feature's form: `Field`, `Required`, `Rule`, `Message when refused`. |
|
|
83
84
|
| Rules | `## Business rules` | One numbered rule per row: `ID`, `Rule`. Who may do it, when an action is available, limits, defaults, what is kept, sent or deleted, and when. |
|
|
85
|
+
| Clients | `## Where the apps differ` | Only when the product has several clients: one row per difference, `What` and a column per client (`Web`, `Mobile`). One sentence instead when they were compared and do not differ. |
|
|
86
|
+
| Parts | `## Parts` | Only on a feature that has parts: a link to each part and one line saying what it covers. |
|
|
84
87
|
|
|
85
|
-
The headings are written exactly so and come in this order; a section the feature does not have is left out. A field's rule is in `Fields` and is not repeated in `Business rules`. Usually 150-400 words
|
|
88
|
+
The headings are written exactly so and come in this order; a section the feature does not have is left out. A field's rule is in `Fields` and is not repeated in `Business rules`. Usually 150-400 words and up to about twenty rules. A feature that needs more gets parts: notes with `parent` set to the feature note, each about one flow or one kind of use, never one kind of fact, and each written in this same shape. The feature note then holds what the feature is for, how the parts fit, the rules that hold in every part, and `## Parts`.
|
|
86
89
|
|
|
87
90
|
A rule is numbered `BR-<FEATURE>-<NN>` (`BR-INVITE-03`): `BR`, a short upper-case word for the feature that is the same in all rules of the note and used by no other note, and a number from 01. The number is the rule's name, quoted in tickets, tests and gaps. A rule keeps it when its text changes; a new rule takes the next free number; the number of a removed rule is not used again; rules are never renumbered. One rule per row, one or two sentences, with the value.
|
|
88
91
|
|
|
@@ -164,7 +167,9 @@ The validator enforces these body rules:
|
|
|
164
167
|
|
|
165
168
|
## Helper
|
|
166
169
|
|
|
167
|
-
|
|
170
|
+
Four scripts in `$SKILL_DIR/scripts/`; see [SKILL.md](../SKILL.md) for the variables.
|
|
171
|
+
|
|
172
|
+
`coverage.py --repo PATH [--vault PATH] [--path FOLDER]... [--all]` reads the repository's code files and the spec pages (`area` and `note`) and prints what the code has that no page mentions. It writes nothing and always exits 0. Code files are the tracked and the untracked-but-not-ignored files with a source-code extension, outside the vault, without tests, fixtures, stories, mocks, migrations, dependencies and build output. For each folder it prints the number of files and lines, how many of the files a page cites in `sources`, how many **sentences a person can see** the folder has and how many are on no page, and the same for **named numbers**. A sentence is a string or text between tags of 12 to 200 characters that starts with a capital letter and has at least three words, outside log calls; it is on a page when a page contains it, ignoring case, curly quotes and spacing. A named number is an upper-case constant set to a number (`MAX_FILES = 10`); it is on a page when a page contains its name. A file with 40 or more sentences is an article or a template and counts as a file only. A folder in which no page cites any file is one line however deep it goes; folders are ordered by how much they have that no page mentions. With `--path` only those folders are read and every unmentioned sentence and number is listed with its file, then the largest files no page cites. The first 40 folders and 60 items are printed unless `--all` is given. The report finds places to read; it cannot find a rule that has neither a message nor a named number.
|
|
168
173
|
|
|
169
174
|
`knowledge.py {check,snapshot,serve,build} --repo PATH [--vault PATH]`
|
|
170
175
|
|
|
@@ -204,7 +209,7 @@ Canonical Markdown must not contain raw merge conflicts. Validation rejects comp
|
|
|
204
209
|
|
|
205
210
|
`serve` shows the vault in a browser: `knowledge.py serve --repo REPO [--vault VAULT] [--port N] [--open]`. Without `--vault`, this and every other command use the single vault under `knowledge/` (`knowledge/00-index.md` or `knowledge/*/00-index.md`) and stop with a message when there is none or several. It listens on `127.0.0.1` only (port 8757, or a free one when that is taken), answers only `/` and only when addressed as this machine, never writes, and builds the page again from disk on every request, so a reload shows the current files. When the vault has errors the page lists them. The page data is the graph above plus `live: true`, `vaultPath` (the vault's path relative to the repository), `history`, `pending` and `files`. `history` is the last 30 commits that touched vault pages (not `.knowledge/`), newest first, each `{"sha", "date", "subject", "pages", "more"}`; a page entry is `{"path", "status"}` with `status` `added`, `changed` or `removed`, plus `was` and `now` when the commit changed the page's `summary` (or, failing that, its `title`); at most 80 pages are listed per commit and `more` counts the rest. `pending` lists vault pages with uncommitted changes in the same `{"path", "status"}` shape. Both are empty outside a Git work tree. `files` is the text of the repository files that pages cite in `sources` or link to, keyed by repository-relative path, each `{"text", "truncated"}`. A file is included only when it is inside the repository and outside the vault, is not a credential path, and is UTF-8 text; one file is cut at 300,000 characters and all files together at 4,000,000, after which the rest stay ordinary links. The viewer opens a link to an included file in place: Markdown rendered, anything else as numbered lines, with the lines of a `#L10` or `#L10-L20` fragment marked.
|
|
206
211
|
|
|
207
|
-
`build` writes the same viewer as one file, for a server that shows the spec to people who do not have the repository: `knowledge.py build --repo REPO [--vault VAULT] --out FILE [--source-url URL] [--with-sources]`. `--out` must be a file outside the vault. The command stops with the `check` errors when the vault has any, and writes nothing then; freshness warnings do not stop it. The page data is the graph plus `built: true`, `commit` (the full `HEAD`, empty outside Git), `sourceUrl`, `vaultPath`, `history`, `pending` and `files`. `files` is empty unless `--with-sources` is given, because the text of cited files is source code. `--source-url` must be an `http(s)` address; `{commit}` in it is replaced by `HEAD`, and it is an error to name `{commit}` outside a Git work tree. On a built page a cited file whose text is not included is a link to `sourceUrl/<repository path>` (with `#L10-L20` when lines are named) in a new tab, or plain text when there is no `sourceUrl`; it is never a relative link, which would lead into the site that hosts the page. In a shallow clone the commit whose parents were not fetched is left out of `history` (by `serve` too), since Git would show every page as added there, and `build` says that the history is short.
|
|
212
|
+
`build` writes the same viewer as one file, for a server that shows the spec to people who do not have the repository: `knowledge.py build --repo REPO [--vault VAULT] --out FILE [--source-url URL] [--with-sources]`. `--out` must be a file outside the vault. The command stops with the `check` errors when the vault has any, and writes nothing then; freshness warnings do not stop it. The page data is the graph plus `built: true`, `commit` (the full `HEAD`, empty outside Git), `sourceUrl`, `vaultPath`, `history`, `pending` and `files`. `files` is empty unless `--with-sources` is given, because the text of cited files is source code. `--source-url` must be an `http(s)` address; `{commit}` in it is replaced by `HEAD`, and it is an error to name `{commit}` outside a Git work tree. `--allow-missing-sources` is for a build that sees only part of the repository (a Docker build context): a cited path where nothing exists is not an error and does not mark its page as changed, and the command lists those files; `check` and `snapshot` never accept it. On a built page a cited file whose text is not included is a link to `sourceUrl/<repository path>` (with `#L10-L20` when lines are named) in a new tab, or plain text when there is no `sourceUrl`; it is never a relative link, which would lead into the site that hosts the page. In a shallow clone the commit whose parents were not fetched is left out of `history` (by `serve` too), since Git would show every page as added there, and `build` says that the history is short.
|
|
208
213
|
|
|
209
214
|
`assets/viewer.html` has exactly one `__KNOWLEDGE_DATA__` placeholder for script-safe JSON. Treat contents as untrusted text: no raw Markdown HTML execution, unsafe URL schemes, external scripts, fonts, analytics or fetches. Render readable basic Markdown, pipe tables and wikilinks; leave unsupported syntax as text. The viewer has a tree of areas with their notes (a note is listed under its first area; superseded notes are not listed), a search over pages, gaps and the cited files, and five places: the overview, a page, the gap queue, the map and recent changes. A note shows its title, `summary`, the page text and the gaps about it that are not closed, then its source files folded; an area shows its notes with their summaries and folds its own text. The pages linked to or from the page are a short list beside it. Navigation keeps a trail for Back and Forward and writes the place into the address (`#/page/notes/example`, `#/gaps`), so a reload or a copied link returns to it. The map draws one frame per primary area (the area page, or the first entry of `areas`) with its pages inside; an edge between a page and its primary area is not drawn because the frame says it, links inside one area are solid and links across areas dashed; `nav` pages are not drawn. Recent changes reads `history`, `pending` and the "changed source" lines of `warnings`. "Copy for AI" only composes text from the page, a choice of what the agent should do and the reader's own words, and copies it to the clipboard. The reader's name, the recently viewed pages and the last commit seen are kept in the browser's `localStorage`, and the viewer works without it.
|
|
210
215
|
|
|
@@ -27,7 +27,7 @@ A gap must not have `status`; its progress is carried by `recheck` and `decision
|
|
|
27
27
|
| `decided_by` | when `decision` is `accepted`, `rejected` or `deferred` | a person's name; the value `agent` is an error |
|
|
28
28
|
| `decided_on` | when `decision` is `accepted`, `rejected` or `deferred` | `YYYY-MM-DD` |
|
|
29
29
|
| `duplicate_of` | when `decision` is `duplicate` | id of another existing gap (not itself) |
|
|
30
|
-
| `priority`, `confidence` | optional, gap and plan pages only | short strings, e.g. `"high"` |
|
|
30
|
+
| `priority`, `confidence` | optional, gap and plan pages only | short strings, e.g. `"high"`. Use `high`, `medium` or `low` for `priority` (or `P0` to `P4`): the viewer lists the most urgent gap of each state first and can show one priority only. |
|
|
31
31
|
|
|
32
32
|
`decided_by` and `decided_on` are also allowed on `done` and `duplicate` but not required. With `outdated`, `rechecked_by` and `rechecked_on` stay in place as the record of the last recheck actually done. A fresh recheck replaces all three fields and adds a new dated entry to the `Recheck` section; keep the older entries.
|
|
33
33
|
|
|
@@ -0,0 +1,200 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
"""List what the code has that no spec page mentions yet, using only the standard library.
|
|
3
|
+
|
|
4
|
+
It reads the application's code files and the spec pages and compares three things a spec is supposed to hold:
|
|
5
|
+
the files pages cite, the sentences a person can see (messages, labels, empty states) and the numeric limits.
|
|
6
|
+
What it prints is a list of places to read, not a score: a sentence that is not on a page may be a button that
|
|
7
|
+
only opens something, which the spec leaves out on purpose.
|
|
8
|
+
"""
|
|
9
|
+
import argparse
|
|
10
|
+
import os
|
|
11
|
+
import posixpath
|
|
12
|
+
import re
|
|
13
|
+
import sys
|
|
14
|
+
from pathlib import Path
|
|
15
|
+
|
|
16
|
+
import knowledge
|
|
17
|
+
|
|
18
|
+
CODE = {'.ts', '.tsx', '.js', '.jsx', '.mjs', '.cjs', '.vue', '.svelte', '.py', '.rb', '.go', '.rs', '.java', '.kt', '.kts',
|
|
19
|
+
'.swift', '.cs', '.php', '.dart', '.ex', '.exs', '.scala', '.sql', '.c', '.cc', '.cpp', '.h', '.m', '.mm'}
|
|
20
|
+
SKIP_DIRS = {'node_modules', '.git', 'dist', 'build', 'out', 'vendor', 'target', '.next', '.nuxt', '.expo', 'coverage', '__pycache__',
|
|
21
|
+
'.venv', 'venv', 'Pods', '.gradle', 'bin', 'obj', '.turbo', '.cache', 'generated', '__generated__',
|
|
22
|
+
# A migration is the history of the schema, not what the product does today.
|
|
23
|
+
'migrations', 'Migrations', 'migration'}
|
|
24
|
+
# A file with this many sentences is text someone wrote for readers (an article, a template, legal terms): the spec
|
|
25
|
+
# lists such things, it does not quote them, so they count as files and lines only.
|
|
26
|
+
CONTENT_SENTENCES = 40
|
|
27
|
+
# Tests, fixtures and stories say what the code does in someone else's words; they are not what a person sees.
|
|
28
|
+
TEST_DIR = re.compile(r'(^|/)(tests?|__tests__|__mocks__|__snapshots__|fixtures?|testdata|e2e|stories|specs?|mocks?|seeds?)(/|$)', re.I)
|
|
29
|
+
TEST_FILE = re.compile(r'(\.(test|spec|stories|e2e|mock|fixture)\.|_test\.|(^|/)test_[^/]*$|\.d\.ts$|\.min\.|\.generated\.|\.g\.dart$)', re.I)
|
|
30
|
+
FILE_LIMIT = 1_000_000
|
|
31
|
+
QUOTED = re.compile(r'''(["'`])((?:\\.|(?!\1)[^\\\n])*)\1''')
|
|
32
|
+
JSX_TEXT = re.compile(r'>\s*([A-Z][^<>{}\n]{10,200}?)\s*<')
|
|
33
|
+
LOG_LINE = re.compile(r'\b(console|logger|log|logging|Sentry|debug)\s*\.\s*\w+\s*\(|^\s*(import|from|describe|it|test|#include|using|package)\b|^\s*(//|#|\*|/\*)')
|
|
34
|
+
NOT_PROSE = re.compile(r'[{}<>;=\\]|\$\(|^\s*(SELECT|INSERT|UPDATE|DELETE|CREATE|ALTER|DROP|WITH)\b|https?://|\w+\.\w+\(|\w/\w+/\w')
|
|
35
|
+
# A named number: MAX_ATTACHMENTS = 10, TTL_MS: number = 60 * 60 * 1000
|
|
36
|
+
LIMIT = re.compile(r'\b([A-Z][A-Z0-9]*(?:_[A-Z0-9]+)+)\s*(?::[^=\n]{1,40})?=\s*([-+]?\d[\d_.]*(?:\s*[*/+-]\s*\d[\d_.]*)*)\s*[;,]?\s*(?:(?://|#).*)?$', re.M)
|
|
37
|
+
PUNCTUATION = str.maketrans({'‘': "'", '’': "'", '“': '"', '”': '"', '–': '-', '—': '-', ' ': ' '})
|
|
38
|
+
|
|
39
|
+
|
|
40
|
+
def plain(text):
|
|
41
|
+
"""Text as it is compared: case, curly quotes, runs of spaces and Markdown's escaped pipe do not count."""
|
|
42
|
+
return re.sub(r'\s+', ' ', text.translate(PUNCTUATION).replace('\\|', '|')).casefold().strip()
|
|
43
|
+
|
|
44
|
+
|
|
45
|
+
def is_sentence(text):
|
|
46
|
+
"""Does this string read like something shown to a person, rather than a key, a class list or a query?"""
|
|
47
|
+
text = text.strip()
|
|
48
|
+
if not 12 <= len(text) <= 200 or not text[0].isupper() or len(text.split()) < 3 or NOT_PROSE.search(text):
|
|
49
|
+
return False
|
|
50
|
+
letters = sum(1 for c in text if c.isalpha() or c == ' ')
|
|
51
|
+
return letters / len(text) >= 0.8
|
|
52
|
+
|
|
53
|
+
|
|
54
|
+
def sentences_in(text):
|
|
55
|
+
found = []
|
|
56
|
+
for line in text.split('\n'):
|
|
57
|
+
if LOG_LINE.search(line):
|
|
58
|
+
continue
|
|
59
|
+
for _, inner in QUOTED.findall(line):
|
|
60
|
+
inner = re.sub(r"\\(['\"`])", r'\1', inner)
|
|
61
|
+
if is_sentence(inner):
|
|
62
|
+
found.append(inner.strip())
|
|
63
|
+
found += [m.strip() for m in JSX_TEXT.findall(line) if is_sentence(m)]
|
|
64
|
+
return list(dict.fromkeys(found))
|
|
65
|
+
|
|
66
|
+
|
|
67
|
+
def code_files(repo, vault):
|
|
68
|
+
"""Repository-relative paths of the application's own code: tracked or untracked but not ignored, no tests."""
|
|
69
|
+
listed = knowledge.run_git(repo, ['ls-files', '-z', '--cached', '--others', '--exclude-standard'])
|
|
70
|
+
if listed.returncode == 0:
|
|
71
|
+
names = [os.fsdecode(n) for n in listed.stdout.split(b'\0') if n]
|
|
72
|
+
else:
|
|
73
|
+
names = []
|
|
74
|
+
for folder, dirs, files in os.walk(repo):
|
|
75
|
+
dirs[:] = sorted(d for d in dirs if d not in SKIP_DIRS and not d.startswith('.'))
|
|
76
|
+
names += [Path(os.path.relpath(Path(folder) / f, repo)).as_posix() for f in sorted(files)]
|
|
77
|
+
inside = Path(os.path.relpath(vault, repo)).as_posix() + '/'
|
|
78
|
+
return sorted(n for n in names
|
|
79
|
+
if posixpath.splitext(n)[1].lower() in CODE and not n.startswith(inside)
|
|
80
|
+
and not any(part in SKIP_DIRS for part in n.split('/')[:-1])
|
|
81
|
+
and not TEST_DIR.search(posixpath.dirname(n)) and not TEST_FILE.search(n))
|
|
82
|
+
|
|
83
|
+
|
|
84
|
+
def measure(repo, vault, notes, prefixes=()):
|
|
85
|
+
"""Per code file: lines, whether a page cites it, and the sentences and limits no page mentions."""
|
|
86
|
+
spec = [n for n in notes if n['meta']['type'] in {'area', 'note'}]
|
|
87
|
+
pages = plain('\n'.join(f"{n['meta']['title']}\n{n['meta'].get('summary', '')}\n{n['body']}" for n in spec))
|
|
88
|
+
words = set(re.findall(r'[A-Z][A-Z0-9_]+', '\n'.join(n['body'] for n in spec)))
|
|
89
|
+
cited = {posixpath.normpath(s) for n in spec for s in n['meta']['sources']}
|
|
90
|
+
files = {}
|
|
91
|
+
for name in code_files(repo, vault):
|
|
92
|
+
if prefixes and not any(name == p or name.startswith(p.rstrip('/') + '/') for p in prefixes):
|
|
93
|
+
continue
|
|
94
|
+
path = repo / name
|
|
95
|
+
try:
|
|
96
|
+
if path.is_symlink() or path.stat().st_size > FILE_LIMIT:
|
|
97
|
+
continue
|
|
98
|
+
text = path.read_bytes().decode('utf-8')
|
|
99
|
+
except (OSError, UnicodeDecodeError):
|
|
100
|
+
continue
|
|
101
|
+
sentences = sentences_in(text)
|
|
102
|
+
content = len(sentences) >= CONTENT_SENTENCES
|
|
103
|
+
if content:
|
|
104
|
+
sentences = []
|
|
105
|
+
limits = list(dict.fromkeys(LIMIT.findall(text)))
|
|
106
|
+
files[name] = {
|
|
107
|
+
'lines': text.count('\n') + 1, 'cited': name in cited, 'content': content, 'sentences': len(sentences), 'limits': len(limits),
|
|
108
|
+
'unquoted': [s for s in sentences if plain(s).rstrip('.!?:…') not in pages],
|
|
109
|
+
'unnamed': [(n, v.strip()) for n, v in limits if n not in words]}
|
|
110
|
+
return files
|
|
111
|
+
|
|
112
|
+
|
|
113
|
+
def folders(files, prefixes=()):
|
|
114
|
+
"""Group files into the places to read: a folder no page cites anything in is one line, however deep it goes.
|
|
115
|
+
|
|
116
|
+
With prefixes (the folders the caller asked about) a line never names a folder above the one asked for."""
|
|
117
|
+
cited_under = {}
|
|
118
|
+
for name, info in files.items():
|
|
119
|
+
if info['cited']:
|
|
120
|
+
parts = name.split('/')[:-1]
|
|
121
|
+
for i in range(len(parts) + 1):
|
|
122
|
+
cited_under['/'.join(parts[:i])] = True
|
|
123
|
+
groups = {}
|
|
124
|
+
for name, info in files.items():
|
|
125
|
+
parts = name.split('/')[:-1]
|
|
126
|
+
folder = '/'.join(parts)
|
|
127
|
+
floor = max((len(p.split('/')) for p in prefixes if name.startswith(p.rstrip('/') + '/')), default=1)
|
|
128
|
+
# Climb to the highest folder that still has no cited file under it.
|
|
129
|
+
for i in range(min(floor, len(parts)) or 1, len(parts) + 1):
|
|
130
|
+
if not cited_under.get('/'.join(parts[:i])):
|
|
131
|
+
folder = '/'.join(parts[:i])
|
|
132
|
+
break
|
|
133
|
+
g = groups.setdefault(folder or '.', {'files': 0, 'lines': 0, 'cited': 0, 'sentences': 0, 'unquoted': 0, 'limits': 0, 'unnamed': 0})
|
|
134
|
+
g['files'] += 1
|
|
135
|
+
g['lines'] += info['lines']
|
|
136
|
+
g['cited'] += info['cited']
|
|
137
|
+
g['sentences'] += info['sentences']
|
|
138
|
+
g['unquoted'] += len(info['unquoted'])
|
|
139
|
+
g['limits'] += info['limits']
|
|
140
|
+
g['unnamed'] += len(info['unnamed'])
|
|
141
|
+
return groups
|
|
142
|
+
|
|
143
|
+
|
|
144
|
+
def main(argv=None):
|
|
145
|
+
parser = argparse.ArgumentParser(description=__doc__.split('\n')[0])
|
|
146
|
+
parser.add_argument('--repo', required=True, type=Path)
|
|
147
|
+
parser.add_argument('--vault', type=Path, help='default: the single vault under knowledge/ in the repository')
|
|
148
|
+
parser.add_argument('--path', action='append', default=[], metavar='FOLDER',
|
|
149
|
+
help='only this folder or file of the repository, and list each sentence and limit no page mentions; repeatable')
|
|
150
|
+
parser.add_argument('--all', action='store_true', help='print every line instead of the first 40 folders or 60 items')
|
|
151
|
+
args = parser.parse_args(argv)
|
|
152
|
+
repo = args.repo.resolve()
|
|
153
|
+
if not repo.is_dir():
|
|
154
|
+
parser.error('repo must be an existing directory')
|
|
155
|
+
try:
|
|
156
|
+
vault = args.vault.resolve() if args.vault else knowledge.find_vault(repo)
|
|
157
|
+
except ValueError as exc:
|
|
158
|
+
parser.error(str(exc))
|
|
159
|
+
if not vault.is_dir():
|
|
160
|
+
parser.error('repo and vault must be existing directories')
|
|
161
|
+
notes, _, errors, _, _ = knowledge.load_vault(repo, vault, freshness=False, absent=[])
|
|
162
|
+
if errors:
|
|
163
|
+
print(f'coverage: the vault has {len(errors)} errors; pages with errors are not counted (run check)', file=sys.stderr)
|
|
164
|
+
prefixes = [posixpath.normpath(p.replace(os.sep, '/')) for p in args.path]
|
|
165
|
+
files = measure(repo, vault, notes, prefixes)
|
|
166
|
+
if not files:
|
|
167
|
+
print('coverage: no code files found' + (' under ' + ', '.join(prefixes) if prefixes else ''))
|
|
168
|
+
return 0
|
|
169
|
+
total = lambda key: sum(len(f[key]) if isinstance(f[key], list) else f[key] for f in files.values())
|
|
170
|
+
print(f"coverage: {len(files)} code files, {total('lines')} lines; pages cite {total('cited')} of them")
|
|
171
|
+
print(f"coverage: {total('sentences')} sentences a person can see, {total('unquoted')} on no page; "
|
|
172
|
+
f"{total('limits')} named numbers, {total('unnamed')} on no page")
|
|
173
|
+
texts = sum(1 for f in files.values() if f['content'])
|
|
174
|
+
print('coverage: this is a list of places to read, not a score; tests, fixtures, migrations and generated code are left out'
|
|
175
|
+
+ (f'; {texts} files that are mostly text (articles, templates) count as files only' if texts else ''))
|
|
176
|
+
groups = sorted(folders(files, prefixes).items(), key=lambda kv: (-(kv[1]['unquoted'] + kv[1]['unnamed']), -kv[1]['lines'], kv[0]))
|
|
177
|
+
shown = groups if args.all else groups[:40]
|
|
178
|
+
width = max(len(name) for name, _ in shown)
|
|
179
|
+
print(f"\n{'folder'.ljust(width)} files lines cited sentences not on a page numbers not on a page")
|
|
180
|
+
for name, g in shown:
|
|
181
|
+
print(f"{name.ljust(width)} {g['files']:5} {g['lines']:6} {g['cited']:5} {g['unquoted']:9} of {g['sentences']:<9} {g['unnamed']:7} of {g['limits']}")
|
|
182
|
+
if len(groups) > len(shown):
|
|
183
|
+
print(f'... and {len(groups) - len(shown)} more folders (--all prints them)')
|
|
184
|
+
if prefixes:
|
|
185
|
+
items = [(name, 'says', s) for name, f in files.items() for s in f['unquoted']]
|
|
186
|
+
items += [(name, 'sets', f'{n} = {v}') for name, f in files.items() for n, v in f['unnamed']]
|
|
187
|
+
print(f'\n{len(items)} things under {", ".join(prefixes)} that no page mentions:')
|
|
188
|
+
for name, verb, what in (items if args.all else items[:60]):
|
|
189
|
+
print(f' {name} {verb}: {what}')
|
|
190
|
+
if not args.all and len(items) > 60:
|
|
191
|
+
print(f' ... and {len(items) - 60} more (--all prints them)')
|
|
192
|
+
uncited = sorted((f['lines'], name) for name, f in files.items() if not f['cited'])
|
|
193
|
+
print(f'\n{len(uncited)} of {len(files)} files there are cited by no page; the largest:')
|
|
194
|
+
for lines, name in reversed(uncited[-(len(uncited) if args.all else 15):]):
|
|
195
|
+
print(f' {name} ({lines} lines)')
|
|
196
|
+
return 0
|
|
197
|
+
|
|
198
|
+
|
|
199
|
+
if __name__ == '__main__':
|
|
200
|
+
sys.exit(main())
|
|
@@ -37,7 +37,7 @@ RULE_ROW = re.compile(r'^ {0,3}\|[ \t]*(BR-[A-Z0-9]+(?:-[A-Z0-9]+)*-\d{2,4})[ \t
|
|
|
37
37
|
MD_REF = re.compile(r'^ {0,3}\[[^\]\n]+\]:\s*(<[^>]*>|\S+)', re.M)
|
|
38
38
|
GAP_PAGE_ONLY = ('priority', 'confidence')
|
|
39
39
|
ARRAYS = {'areas', 'sources', 'tags', 'aliases', 'task_ids'}
|
|
40
|
-
STRINGS = {'id', 'title', 'type', 'status', 'updated', 'summary', 'verification', 'priority', 'confidence', 'gap',
|
|
40
|
+
STRINGS = {'id', 'title', 'type', 'status', 'updated', 'summary', 'verification', 'priority', 'confidence', 'gap', 'parent',
|
|
41
41
|
*GAP_ONLY}
|
|
42
42
|
REQUIRED = {'id', 'title', 'type', 'areas', 'sources', 'updated'}
|
|
43
43
|
LINK = re.compile(r'(!?)\[\[([^\[\]\n]+)\]\]') # no brackets inside, so a run of '[[' cannot backtrack
|
|
@@ -215,6 +215,8 @@ def validate_meta(meta):
|
|
|
215
215
|
raise ValueError(f"status must be one of {', '.join(sorted(allowed))}{hint}")
|
|
216
216
|
if kind == 'note' and meta['status'] == 'current' and not meta['sources']:
|
|
217
217
|
raise ValueError('current note requires sources')
|
|
218
|
+
if 'parent' in meta and (kind != 'note' or not meta['parent'].strip()):
|
|
219
|
+
raise ValueError('parent is only valid on note pages and names the feature note this one is a part of')
|
|
218
220
|
if kind == 'plan':
|
|
219
221
|
if not meta.get('gap', '').strip():
|
|
220
222
|
raise ValueError('plan requires gap: the id of the accepted gap it implements')
|
|
@@ -313,20 +315,29 @@ def line_ending_variant(data, old):
|
|
|
313
315
|
return old in {hashlib.sha256(lf).hexdigest(), hashlib.sha256(lf.replace(b'\n', b'\r\n')).hexdigest()}
|
|
314
316
|
|
|
315
317
|
|
|
318
|
+
def run_git(repo, args, **options):
|
|
319
|
+
"""Git's answer for the repository. Where Git is not installed (a build container, say) it is the answer
|
|
320
|
+
of a folder that is not a work tree, so the caller goes on without history instead of stopping."""
|
|
321
|
+
try:
|
|
322
|
+
return subprocess.run(['git', '-C', str(repo), *args], capture_output=True, **options)
|
|
323
|
+
except FileNotFoundError:
|
|
324
|
+
return subprocess.CompletedProcess(args, 127, b'', b'')
|
|
325
|
+
|
|
326
|
+
|
|
316
327
|
def head_blob(repo, source):
|
|
317
328
|
"""Bytes of the source as committed at HEAD, or None when HEAD has no such file (untracked, new or no commits)."""
|
|
318
|
-
result =
|
|
329
|
+
result = run_git(repo, ['cat-file', 'blob', f'HEAD:./{posixpath.normpath(source)}'])
|
|
319
330
|
return None if result.returncode else result.stdout
|
|
320
331
|
|
|
321
332
|
|
|
322
333
|
def uncommitted_paths(repo):
|
|
323
334
|
"""Repo-relative paths Git reports as modified, staged or untracked (one status call), or None outside a work tree."""
|
|
324
|
-
probe =
|
|
335
|
+
probe = run_git(repo, ['rev-parse', '--is-inside-work-tree', '--show-prefix'])
|
|
325
336
|
lines = probe.stdout.decode(errors='replace').split('\n')
|
|
326
337
|
if probe.returncode or lines[0] != 'true':
|
|
327
338
|
return None
|
|
328
339
|
prefix = lines[1] if len(lines) > 1 else ''
|
|
329
|
-
status =
|
|
340
|
+
status = run_git(repo, ['status', '--porcelain=v1', '-z', '--untracked-files=all'])
|
|
330
341
|
if status.returncode:
|
|
331
342
|
return None
|
|
332
343
|
raw, found, i = status.stdout.split(b'\0'), set(), 0
|
|
@@ -347,10 +358,13 @@ def error_page(message):
|
|
|
347
358
|
return head if head.endswith('.md') else None
|
|
348
359
|
|
|
349
360
|
|
|
350
|
-
def load_vault(repo, vault, freshness=True, dirty=None, report=None):
|
|
361
|
+
def load_vault(repo, vault, freshness=True, dirty=None, report=None, absent=None):
|
|
351
362
|
"""dirty: repo-relative paths with uncommitted changes. Their freshness warning is marked, not counted, only when
|
|
352
363
|
the accepted fingerprint equals the file's HEAD blob (up to LF/CRLF), so the sole drift is uncommitted.
|
|
353
364
|
|
|
365
|
+
absent, when given, collects cited files that are not there instead of reporting them: for a build that sees
|
|
366
|
+
only part of the repository, such as a Docker build context. Their pages are not marked as changed.
|
|
367
|
+
|
|
354
368
|
report, when given, receives 'failed': {page path: parsed id or None} for pages that failed validation.
|
|
355
369
|
"""
|
|
356
370
|
errors, warnings, notes, failed, failed_kind, cascade = [], [], [], {}, {}, []
|
|
@@ -415,10 +429,26 @@ def load_vault(repo, vault, freshness=True, dirty=None, report=None):
|
|
|
415
429
|
paths[stem] = note
|
|
416
430
|
basenames.setdefault(PurePosixPath(stem).name, []).append(note)
|
|
417
431
|
area_ids = {n['meta']['id'] for n in notes if n['meta']['type'] == 'area'}
|
|
418
|
-
edges, attachments, rule_pages = set(), {}, {}
|
|
432
|
+
edges, attachments, rule_pages, linked, parts = set(), {}, {}, set(), []
|
|
419
433
|
for note in notes:
|
|
420
434
|
meta, rel = note['meta'], note['path']
|
|
421
435
|
note['stale'] = False
|
|
436
|
+
if meta.get('parent'):
|
|
437
|
+
# Three levels and no more: an area, its features, and the parts of a feature that is too big for one page.
|
|
438
|
+
other, name = ids.get(meta['parent']), meta['parent']
|
|
439
|
+
if other is None and name in failed_ids and failed_id_kinds[name] & {None, 'note'}:
|
|
440
|
+
cascade.append(f'{rel}: parent {name} names a page that failed validation')
|
|
441
|
+
elif other is None or other['meta']['type'] != 'note':
|
|
442
|
+
errors.append(f'{rel}: parent {name} is not the id of an existing note')
|
|
443
|
+
elif other is note:
|
|
444
|
+
errors.append(f'{rel}: parent must name a different note')
|
|
445
|
+
elif other['meta'].get('parent'):
|
|
446
|
+
errors.append(f"{rel}: parent {name} is itself a part of {other['meta']['parent']}; a part belongs to a feature, not to another part")
|
|
447
|
+
elif other['meta']['areas'][:1] != meta['areas'][:1]:
|
|
448
|
+
errors.append(f"{rel}: parent {name} is in another area; a part has the same first area as its feature")
|
|
449
|
+
else:
|
|
450
|
+
edges.add((other['meta']['id'], meta['id']))
|
|
451
|
+
parts.append((other, note))
|
|
422
452
|
if meta['type'] == 'gap':
|
|
423
453
|
note['state'] = gap_state(meta)
|
|
424
454
|
if meta['type'] == 'gap' and meta.get('duplicate_of'):
|
|
@@ -495,6 +525,7 @@ def load_vault(repo, vault, freshness=True, dirty=None, report=None):
|
|
|
495
525
|
"if that part needs its own link, make it its own note and list it here")
|
|
496
526
|
else:
|
|
497
527
|
edges.add((meta['id'], choices[0]['meta']['id']))
|
|
528
|
+
linked.add((meta['id'], choices[0]['meta']['id']))
|
|
498
529
|
if meta['type'] == 'area':
|
|
499
530
|
# An area page is a map of its notes. A table on it is behavior written in the wrong place.
|
|
500
531
|
tables = sum(1 for line in text.split('\n') if TABLE_RULE.match(line))
|
|
@@ -541,6 +572,9 @@ def load_vault(repo, vault, freshness=True, dirty=None, report=None):
|
|
|
541
572
|
if credential_path(path.resolve().relative_to(repo).as_posix()):
|
|
542
573
|
raise ValueError('credential source is forbidden')
|
|
543
574
|
if not path.is_file():
|
|
575
|
+
if absent is not None and not path.exists():
|
|
576
|
+
absent.append(source)
|
|
577
|
+
continue
|
|
544
578
|
if closed:
|
|
545
579
|
warnings.append(f'{rel}: missing source {source}')
|
|
546
580
|
note['stale'] = True
|
|
@@ -564,6 +598,10 @@ def load_vault(repo, vault, freshness=True, dirty=None, report=None):
|
|
|
564
598
|
note['stale'] = True
|
|
565
599
|
except (OSError, ValueError) as exc:
|
|
566
600
|
errors.append(f'{rel}: {source}: {exc}')
|
|
601
|
+
for feature, part in parts:
|
|
602
|
+
# The viewer nests a part under its feature from `parent`; a plain Markdown reader has only the link.
|
|
603
|
+
if (feature['meta']['id'], part['meta']['id']) not in linked:
|
|
604
|
+
errors.append(f"{part['path']}: its feature {feature['path']} does not link to it; list every part on the feature's page")
|
|
567
605
|
if report is not None:
|
|
568
606
|
report['failed'] = failed
|
|
569
607
|
report['cascade'] = cascade
|
|
@@ -634,7 +672,7 @@ def node_of(note):
|
|
|
634
672
|
'kind': m.get('kind', ''), 'recheck': m.get('recheck', ''), 'recheckedBy': m.get('rechecked_by', ''),
|
|
635
673
|
'recheckedOn': m.get('rechecked_on', ''), 'decision': m.get('decision', ''), 'decidedBy': m.get('decided_by', ''),
|
|
636
674
|
'decidedOn': m.get('decided_on', ''), 'priority': m.get('priority', ''), 'confidence': m.get('confidence', ''),
|
|
637
|
-
'duplicateOf': m.get('duplicate_of', ''), 'recheckOutdated': gap and (m['recheck'] == 'outdated' or recheck_marker(note)), 'state': note.get('state', ''), 'gap': m.get('gap', ''),
|
|
675
|
+
'parent': m.get('parent', ''), 'duplicateOf': m.get('duplicate_of', ''), 'recheckOutdated': gap and (m['recheck'] == 'outdated' or recheck_marker(note)), 'state': note.get('state', ''), 'gap': m.get('gap', ''),
|
|
638
676
|
'layer': LAYERS[m['type']]}
|
|
639
677
|
|
|
640
678
|
|
|
@@ -691,7 +729,7 @@ def front_value(line):
|
|
|
691
729
|
|
|
692
730
|
def shallow_commits(repo):
|
|
693
731
|
"""Commits whose parents a shallow clone left out. Empty in a full clone and outside Git."""
|
|
694
|
-
found =
|
|
732
|
+
found = run_git(repo, ['rev-parse', '--git-path', 'shallow'])
|
|
695
733
|
if found.returncode:
|
|
696
734
|
return set()
|
|
697
735
|
try:
|
|
@@ -709,21 +747,21 @@ def vault_history(repo, vault):
|
|
|
709
747
|
spec, prefix = page_pathspec(repo, vault)
|
|
710
748
|
if spec is None:
|
|
711
749
|
return []
|
|
712
|
-
head =
|
|
750
|
+
head = run_git(repo, ['rev-parse', 'HEAD'])
|
|
713
751
|
if head.returncode:
|
|
714
752
|
return []
|
|
715
753
|
key = (str(repo), str(vault), head.stdout.strip())
|
|
716
754
|
if key in _history_cache:
|
|
717
755
|
return _history_cache[key]
|
|
718
756
|
# Git prints paths from the top of the work tree, which may be above the repository folder given here.
|
|
719
|
-
top =
|
|
757
|
+
top = run_git(repo, ['rev-parse', '--show-prefix']).stdout.decode(errors='replace').strip()
|
|
720
758
|
prefix = top + prefix
|
|
721
759
|
# A marker nobody could have written into a commit message or a page beforehand separates the commits.
|
|
722
760
|
marker = '\x01' + os.urandom(12).hex() + '\x01'
|
|
723
761
|
try:
|
|
724
|
-
log =
|
|
762
|
+
log = run_git(repo, ['-c', 'core.quotepath=off', 'log', f'-n{HISTORY_COMMITS}', '--date=short', '--no-color',
|
|
725
763
|
'--no-renames', '-p', '-U0', f'--format={marker}%H%x02%ad%x02%s', '--', *spec],
|
|
726
|
-
|
|
764
|
+
timeout=20)
|
|
727
765
|
except subprocess.TimeoutExpired:
|
|
728
766
|
return []
|
|
729
767
|
if log.returncode:
|
|
@@ -768,11 +806,10 @@ def vault_pending(repo, vault):
|
|
|
768
806
|
spec, prefix = page_pathspec(repo, vault)
|
|
769
807
|
if spec is None:
|
|
770
808
|
return []
|
|
771
|
-
status =
|
|
772
|
-
capture_output=True)
|
|
809
|
+
status = run_git(repo, ['-c', 'core.quotepath=off', 'status', '--porcelain=v1', '-z', '--untracked-files=all', '--no-renames', '--', *spec])
|
|
773
810
|
if status.returncode:
|
|
774
811
|
return []
|
|
775
|
-
top =
|
|
812
|
+
top = run_git(repo, ['rev-parse', '--show-prefix']).stdout.decode(errors='replace').strip()
|
|
776
813
|
pending = []
|
|
777
814
|
for item in status.stdout.split(b'\0'):
|
|
778
815
|
text = os.fsdecode(item)
|
|
@@ -823,7 +860,7 @@ def built_page(repo, vault, notes, edges, warnings, with_sources=False, source_u
|
|
|
823
860
|
The text of cited repository files is left out unless asked for, because the file may be shown to people
|
|
824
861
|
who cannot read the repository. Returns (html, payload).
|
|
825
862
|
"""
|
|
826
|
-
head =
|
|
863
|
+
head = run_git(repo, ['rev-parse', 'HEAD'])
|
|
827
864
|
commit = '' if head.returncode else head.stdout.decode('ascii', errors='replace').strip()
|
|
828
865
|
payload = page_payload(repo, vault, notes, edges, warnings, with_sources) | {
|
|
829
866
|
'built': True, 'commit': commit, 'sourceUrl': source_base(source_url, commit) if source_url else ''}
|
|
@@ -868,6 +905,8 @@ def main(argv=None):
|
|
|
868
905
|
parser.add_argument('--out', type=Path, help='build: the HTML file to write, outside the vault')
|
|
869
906
|
parser.add_argument('--with-sources', action='store_true', help='build: include the text of the repository files the pages cite')
|
|
870
907
|
parser.add_argument('--source-url', default='', help='build: where a cited file can be read, e.g. https://github.com/OWNER/REPO/blob/{commit}')
|
|
908
|
+
parser.add_argument('--allow-missing-sources', action='store_true',
|
|
909
|
+
help='build: do not stop when a cited file is not there, for a build that sees only part of the repository')
|
|
871
910
|
parser.add_argument('--ignore-dirty', action='store_true',
|
|
872
911
|
help='check: do not count freshness warnings for sources with uncommitted changes (needs a Git work tree)')
|
|
873
912
|
group = parser.add_mutually_exclusive_group()
|
|
@@ -882,8 +921,8 @@ def main(argv=None):
|
|
|
882
921
|
parser.error('--open is only for serve')
|
|
883
922
|
if args.command == 'build' and not args.out:
|
|
884
923
|
parser.error('build requires --out FILE')
|
|
885
|
-
if args.command != 'build' and (args.out or args.with_sources or args.source_url):
|
|
886
|
-
parser.error('--out, --with-sources
|
|
924
|
+
if args.command != 'build' and (args.out or args.with_sources or args.source_url or args.allow_missing_sources):
|
|
925
|
+
parser.error('--out, --with-sources, --source-url and --allow-missing-sources are only for build')
|
|
887
926
|
if args.command == 'snapshot' and not (args.all or args.notes):
|
|
888
927
|
parser.error('snapshot requires --all or --notes')
|
|
889
928
|
if args.command != 'snapshot' and (args.all or args.notes):
|
|
@@ -917,7 +956,8 @@ def main(argv=None):
|
|
|
917
956
|
if dirty is None:
|
|
918
957
|
parser.error('--ignore-dirty needs --repo to be a Git work tree')
|
|
919
958
|
report = {}
|
|
920
|
-
|
|
959
|
+
absent = [] if args.allow_missing_sources else None
|
|
960
|
+
notes, edges, errors, warnings, baseline = load_vault(repo, vault, args.command != 'snapshot', dirty, report, absent)
|
|
921
961
|
ignored = []
|
|
922
962
|
if args.command == 'snapshot' and args.notes:
|
|
923
963
|
# Only the selected pages must be valid; errors elsewhere are counted, not blocking.
|
|
@@ -964,6 +1004,9 @@ def main(argv=None):
|
|
|
964
1004
|
+ (f', commit {payload["commit"][:7]}' if payload['commit'] else ''))
|
|
965
1005
|
print('build: ' + (f'the text of {files} cited source files is inside the file; show it only to people who may read the repository'
|
|
966
1006
|
if args.with_sources else 'cited source files are named but their text is not included (--with-sources adds it)'))
|
|
1007
|
+
if absent:
|
|
1008
|
+
names = sorted(set(absent))
|
|
1009
|
+
print(f'build: {len(names)} cited files are not in this folder, so nobody compared them with the pages: ' + ', '.join(names[:8]) + (' ...' if len(names) > 8 else ''))
|
|
967
1010
|
if shallow_commits(repo):
|
|
968
1011
|
print('build: this is a shallow clone, so the Changes page starts at the first commit that was fetched in full; fetch the whole history to show more')
|
|
969
1012
|
return 0
|
|
@@ -0,0 +1,128 @@
|
|
|
1
|
+
import json
|
|
2
|
+
import subprocess
|
|
3
|
+
import sys
|
|
4
|
+
import tempfile
|
|
5
|
+
import unittest
|
|
6
|
+
from contextlib import redirect_stderr, redirect_stdout
|
|
7
|
+
from io import StringIO
|
|
8
|
+
from pathlib import Path
|
|
9
|
+
|
|
10
|
+
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
|
11
|
+
import coverage as c # noqa: E402
|
|
12
|
+
|
|
13
|
+
|
|
14
|
+
class CoverageTests(unittest.TestCase):
|
|
15
|
+
def setUp(self):
|
|
16
|
+
self.temp = tempfile.TemporaryDirectory()
|
|
17
|
+
self.repo = Path(self.temp.name).resolve() / 'repo'
|
|
18
|
+
self.vault = self.repo / 'knowledge/demo'
|
|
19
|
+
(self.vault / 'notes').mkdir(parents=True)
|
|
20
|
+
(self.vault / 'areas').mkdir()
|
|
21
|
+
self.page('00-index.md', 'index', 'index', areas=[], body='[[areas/chat]]')
|
|
22
|
+
self.page('areas/chat.md', 'chat', 'area', areas=[], body='[[notes/chats]]')
|
|
23
|
+
self.page('notes/chats.md', 'chats', sources=['src/chat/send.ts'],
|
|
24
|
+
body='A message is refused with "Message can’t be empty". At most 10 photos (`CHAT_MAX_MEDIA`).\n\n'
|
|
25
|
+
'| ID | Rule |\n| --- | --- |\n| BR-CHAT-01 | Only group chats can be renamed, by a member \\| staff. |')
|
|
26
|
+
self.code('src/chat/send.ts', '''
|
|
27
|
+
export const CHAT_MAX_MEDIA = 10;
|
|
28
|
+
export const COLD_OUTREACH_PER_HOUR: number = 5; // new people
|
|
29
|
+
const RETRY_MS = 60 * 1000
|
|
30
|
+
const queryKey = "chats.list";
|
|
31
|
+
throw new Error("Message can't be empty");
|
|
32
|
+
throw new Error('Only group chats can be renamed');
|
|
33
|
+
throw new Error("You cannot remove yourself");
|
|
34
|
+
console.error("Failed to load the chat list");
|
|
35
|
+
const cls = "flex items-center gap-2 Text here";
|
|
36
|
+
const sql = "SELECT id FROM chats WHERE owner = 1";
|
|
37
|
+
''')
|
|
38
|
+
self.code('src/chat/window/Header.tsx', '<Button>Back to chat list</Button>\n<p>{count} unread</p>\nconst t = `Hello ${name}, welcome back`;')
|
|
39
|
+
self.code('src/chat/window/deep/Picker.tsx', 'const label = "Smileys and people";')
|
|
40
|
+
self.code('src/chat/send.test.ts', 'it("says this in a test only", () => {})')
|
|
41
|
+
self.code('src/chat/__tests__/more.ts', 'const s = "Only in a test folder";')
|
|
42
|
+
self.code('src/orders/pay.py', 'MAX_TOTAL = 5000\nmessage = "The order was already paid"\n')
|
|
43
|
+
self.code('database/migrations/001.sql', "INSERT INTO t VALUES ('A sentence in a migration');")
|
|
44
|
+
self.code('node_modules/x/index.js', 'const s = "A sentence in a dependency";')
|
|
45
|
+
self.code('src/help/article.ts', '\n'.join(f'const p{i} = "This is paragraph number {i} of an article";' for i in range(45)))
|
|
46
|
+
(self.repo / 'README.md').write_text('Not code. "A sentence in the readme file"', encoding='utf-8')
|
|
47
|
+
|
|
48
|
+
def tearDown(self):
|
|
49
|
+
self.temp.cleanup()
|
|
50
|
+
|
|
51
|
+
def page(self, path, ident, ptype='note', areas=None, sources=None, body=''):
|
|
52
|
+
meta = {'id': ident, 'title': 'Page ' + ident, 'type': ptype, 'status': 'current', 'areas': ['chat'] if areas is None else areas,
|
|
53
|
+
'sources': sources or [], 'updated': '2026-11-18'}
|
|
54
|
+
(self.vault / path).write_text('---\n' + ''.join(f'{key}: {json.dumps(value)}\n' for key, value in meta.items()) + '---\n' + body, encoding='utf-8')
|
|
55
|
+
|
|
56
|
+
def code(self, path, text):
|
|
57
|
+
file = self.repo / path
|
|
58
|
+
file.parent.mkdir(parents=True, exist_ok=True)
|
|
59
|
+
file.write_text(text, encoding='utf-8')
|
|
60
|
+
|
|
61
|
+
def run_cli(self, *args):
|
|
62
|
+
out, err = StringIO(), StringIO()
|
|
63
|
+
with redirect_stdout(out), redirect_stderr(err):
|
|
64
|
+
status = c.main(['--repo', str(self.repo), *args])
|
|
65
|
+
return status, out.getvalue(), err.getvalue()
|
|
66
|
+
|
|
67
|
+
def measured(self, *prefixes):
|
|
68
|
+
import knowledge
|
|
69
|
+
notes = knowledge.load_vault(self.repo, self.vault, freshness=False, absent=[])[0]
|
|
70
|
+
return c.measure(self.repo, self.vault, notes, prefixes)
|
|
71
|
+
|
|
72
|
+
def test_reads_only_the_application_code(self):
|
|
73
|
+
self.assertEqual(sorted(self.measured()), ['src/chat/send.ts', 'src/chat/window/Header.tsx', 'src/chat/window/deep/Picker.tsx',
|
|
74
|
+
'src/help/article.ts', 'src/orders/pay.py'])
|
|
75
|
+
# with Git, an ignored folder is left out as well, and an untracked file is still read
|
|
76
|
+
subprocess.run(['git', '-C', str(self.repo), 'init', '-q'], check=True)
|
|
77
|
+
(self.repo / '.gitignore').write_text('src/orders/\n', encoding='utf-8')
|
|
78
|
+
self.assertNotIn('src/orders/pay.py', self.measured())
|
|
79
|
+
self.assertIn('src/chat/send.ts', self.measured())
|
|
80
|
+
|
|
81
|
+
def test_finds_the_sentences_and_numbers_no_page_mentions(self):
|
|
82
|
+
files = self.measured()
|
|
83
|
+
send = files['src/chat/send.ts']
|
|
84
|
+
self.assertEqual((send['cited'], send['sentences'], send['limits']), (True, 3, 3))
|
|
85
|
+
self.assertEqual(send['unquoted'], ['You cannot remove yourself'], 'a curly apostrophe and a table row still match; logs, class lists and queries are not sentences')
|
|
86
|
+
self.assertEqual(send['unnamed'], [('COLD_OUTREACH_PER_HOUR', '5'), ('RETRY_MS', '60 * 1000')])
|
|
87
|
+
self.assertEqual(files['src/chat/window/Header.tsx']['unquoted'], ['Back to chat list'], 'text between tags counts; text with a placeholder does not')
|
|
88
|
+
self.assertEqual(files['src/orders/pay.py']['unnamed'], [('MAX_TOTAL', '5000')])
|
|
89
|
+
article = files['src/help/article.ts']
|
|
90
|
+
self.assertEqual((article['content'], article['sentences'], article['unquoted']), (True, 0, []), 'an article counts as a file only')
|
|
91
|
+
|
|
92
|
+
def test_groups_by_folder_and_keeps_an_uncited_folder_as_one_line(self):
|
|
93
|
+
groups = c.folders(self.measured())
|
|
94
|
+
self.assertEqual(sorted(groups), ['src/chat', 'src/chat/window', 'src/help', 'src/orders'])
|
|
95
|
+
self.assertEqual(groups['src/chat/window'], {'files': 2, 'lines': 4, 'cited': 0, 'sentences': 2, 'unquoted': 2, 'limits': 0, 'unnamed': 0})
|
|
96
|
+
self.assertEqual((groups['src/chat']['files'], groups['src/chat']['cited']), (1, 1))
|
|
97
|
+
# nothing cited anywhere: one line per top folder, and never above the folder that was asked for
|
|
98
|
+
self.page('notes/chats.md', 'chats', sources=['README.md'])
|
|
99
|
+
self.assertEqual(sorted(c.folders(self.measured())), ['src'])
|
|
100
|
+
self.assertEqual(sorted(c.folders(self.measured('src/chat'), ['src/chat'])), ['src/chat'])
|
|
101
|
+
|
|
102
|
+
def test_report_lists_what_to_read(self):
|
|
103
|
+
status, out, err = self.run_cli()
|
|
104
|
+
self.assertEqual((status, err), (0, ''))
|
|
105
|
+
self.assertIn('coverage: 5 code files, ', out)
|
|
106
|
+
self.assertIn('pages cite 1 of them', out)
|
|
107
|
+
self.assertIn('6 sentences a person can see, 4 on no page; 4 named numbers, 3 on no page', out)
|
|
108
|
+
self.assertIn('1 files that are mostly text (articles, templates) count as files only', out)
|
|
109
|
+
self.assertLess(out.index('src/chat/window'), out.index('src/help'), 'the folder with the most to read comes first')
|
|
110
|
+
self.assertNotIn('things under', out)
|
|
111
|
+
status, out, _ = self.run_cli('--path', 'src/chat/')
|
|
112
|
+
self.assertIn('coverage: 3 code files', out)
|
|
113
|
+
self.assertIn('5 things under src/chat that no page mentions:', out)
|
|
114
|
+
self.assertIn(' src/chat/send.ts says: You cannot remove yourself', out)
|
|
115
|
+
self.assertIn(' src/chat/send.ts sets: RETRY_MS = 60 * 1000', out)
|
|
116
|
+
self.assertIn('2 of 3 files there are cited by no page', out)
|
|
117
|
+
self.assertEqual(self.run_cli('--path', 'nowhere')[1].strip(), 'coverage: no code files found under nowhere')
|
|
118
|
+
|
|
119
|
+
def test_a_vault_with_errors_is_reported_and_still_measured(self):
|
|
120
|
+
self.page('notes/bad.md', 'bad', sources=['src/chat/send.ts'], body='[[missing-page]]')
|
|
121
|
+
status, out, err = self.run_cli()
|
|
122
|
+
self.assertEqual(status, 0)
|
|
123
|
+
self.assertIn('the vault has 1 errors', err)
|
|
124
|
+
self.assertIn('coverage: 5 code files', out)
|
|
125
|
+
|
|
126
|
+
|
|
127
|
+
if __name__ == '__main__':
|
|
128
|
+
unittest.main()
|
|
@@ -612,6 +612,48 @@ class GapLayerTests(Fixture):
|
|
|
612
612
|
self.assertEqual((status, data['files']['source.txt']['text']), (0, 'evidence'))
|
|
613
613
|
self.assertIn('the text of 1 cited source files is inside the file', printed)
|
|
614
614
|
|
|
615
|
+
def test_build_can_run_where_part_of_the_repository_is_missing(self):
|
|
616
|
+
self.note('notes/m.md', 'm', sources=['source.txt', 'Dockerfile'])
|
|
617
|
+
status, _, err, data = self.built()
|
|
618
|
+
self.assertEqual((status, data), (1, None))
|
|
619
|
+
self.assertIn('source is missing or not a file', err)
|
|
620
|
+
status, printed, err, data = self.built('--allow-missing-sources')
|
|
621
|
+
self.assertEqual(status, 0)
|
|
622
|
+
self.assertIn('build: 1 cited files are not in this folder, so nobody compared them with the pages: Dockerfile', printed)
|
|
623
|
+
self.assertNotIn('Dockerfile', err)
|
|
624
|
+
self.note('notes/o.md', 'o', sources=['Dockerfile'])
|
|
625
|
+
data = self.built('--allow-missing-sources')[3]
|
|
626
|
+
pages = {n['id']: (n['sources'], n['stale']) for n in data['nodes']}
|
|
627
|
+
self.assertEqual(pages['o'], (['Dockerfile'], False), 'a file nobody could read does not mark its page as changed')
|
|
628
|
+
self.assertEqual(pages['m'], (['source.txt', 'Dockerfile'], True), 'source.txt has no accepted fingerprint')
|
|
629
|
+
(self.vault / 'notes/o.md').unlink()
|
|
630
|
+
# a folder where a file should be is still an error, and so is the same vault for check
|
|
631
|
+
(self.repo / 'Dockerfile').mkdir()
|
|
632
|
+
self.assertEqual(self.built('--allow-missing-sources')[0], 1)
|
|
633
|
+
(self.repo / 'Dockerfile').rmdir()
|
|
634
|
+
self.assertEqual(self.run_cli('check')[0], 1)
|
|
635
|
+
self.assertIn('only for build', self.refused('check', '--allow-missing-sources'))
|
|
636
|
+
|
|
637
|
+
def test_build_and_serve_work_where_git_is_not_installed(self):
|
|
638
|
+
import os
|
|
639
|
+
from unittest import mock
|
|
640
|
+
k._history_cache.clear()
|
|
641
|
+
self.vault.rename(self.repo / 'vault')
|
|
642
|
+
self.vault = self.repo / 'vault'
|
|
643
|
+
self.git('init', '-q')
|
|
644
|
+
self.git('add', '-A')
|
|
645
|
+
self.git('commit', '-q', '-m', 'First')
|
|
646
|
+
empty = self.root / 'no-tools'
|
|
647
|
+
empty.mkdir()
|
|
648
|
+
with mock.patch.dict(os.environ, {'PATH': str(empty)}):
|
|
649
|
+
status, printed, err, data = self.built()
|
|
650
|
+
self.assertEqual((status, data['commit'], data['history'], data['pending']), (0, '', [], []), err)
|
|
651
|
+
self.assertNotIn('commit ', printed)
|
|
652
|
+
self.assertIn('knowledge-data', k.live_page(self.repo.resolve(), self.vault.resolve()))
|
|
653
|
+
self.assertIn('--ignore-dirty needs --repo to be a Git work tree', self.refused('check', '--ignore-dirty'))
|
|
654
|
+
k._history_cache.clear()
|
|
655
|
+
self.assertEqual([c['subject'] for c in self.built()[3]['history']], ['First'], 'with Git back, the history is too')
|
|
656
|
+
|
|
615
657
|
def test_build_links_cited_files_to_the_commit_it_was_built_from(self):
|
|
616
658
|
k._history_cache.clear()
|
|
617
659
|
self.vault.rename(self.repo / 'vault')
|
|
@@ -683,6 +725,39 @@ class GapLayerTests(Fixture):
|
|
|
683
725
|
self.assertEqual(k.main(['check', '--repo', str(self.repo)]), 0)
|
|
684
726
|
self.assertIn('check: 1 notes', out.getvalue())
|
|
685
727
|
|
|
728
|
+
def test_a_feature_too_big_for_one_page_has_parts(self):
|
|
729
|
+
self.note('notes/chats.md', 'chats', sources=['source.txt'], body='Parts: [[notes/group-chats|group chats]].')
|
|
730
|
+
self.note('notes/group-chats.md', 'group-chats', sources=['source.txt'], parent='chats')
|
|
731
|
+
status, out, err = self.run_cli('check')
|
|
732
|
+
self.assertEqual(status, 0, err)
|
|
733
|
+
notes, edges, errors, _, _ = k.load_vault(self.repo.resolve(), self.vault.resolve())
|
|
734
|
+
self.assertIn(('chats', 'group-chats'), edges)
|
|
735
|
+
self.assertEqual(k.node_of(next(n for n in notes if n['meta']['id'] == 'group-chats'))['parent'], 'chats')
|
|
736
|
+
cases = (
|
|
737
|
+
(dict(parent='chats'), 'Parts: none listed.', 'notes/group-chats.md: its feature notes/chats.md does not link to it'),
|
|
738
|
+
(dict(parent='nobody'), None, 'parent nobody is not the id of an existing note'),
|
|
739
|
+
(dict(parent='a'), None, 'parent a is not the id of an existing note'),
|
|
740
|
+
(dict(parent='group-chats'), None, 'parent must name a different note'),
|
|
741
|
+
(dict(parent=''), None, 'parent is only valid on note pages and names the feature note'),
|
|
742
|
+
)
|
|
743
|
+
for extra, parent_body, message in cases:
|
|
744
|
+
with self.subTest(message=message):
|
|
745
|
+
self.note('notes/chats.md', 'chats', sources=['source.txt'], body=parent_body or 'Parts: [[notes/group-chats|group chats]].')
|
|
746
|
+
self.note('notes/group-chats.md', 'group-chats', sources=['source.txt'], **extra)
|
|
747
|
+
status, _, err = self.run_cli('check')
|
|
748
|
+
self.assertEqual(status, 1)
|
|
749
|
+
self.assertIn(message, err)
|
|
750
|
+
# three levels and no more, and a part stays in its feature's area
|
|
751
|
+
self.note('notes/chats.md', 'chats', sources=['source.txt'], body='[[notes/group-chats|group chats]] [[notes/renaming|renaming]]')
|
|
752
|
+
self.note('notes/group-chats.md', 'group-chats', sources=['source.txt'], parent='chats', body='[[notes/renaming|renaming]]')
|
|
753
|
+
self.note('notes/renaming.md', 'renaming', sources=['source.txt'], parent='group-chats')
|
|
754
|
+
self.assertIn('parent group-chats is itself a part of chats; a part belongs to a feature, not to another part', self.run_cli('check')[2])
|
|
755
|
+
self.note('areas/b.md', 'b', 'area', areas=[])
|
|
756
|
+
self.note('notes/renaming.md', 'renaming', sources=['source.txt'], parent='chats', areas=['b'])
|
|
757
|
+
self.assertIn('parent chats is in another area', self.run_cli('check')[2])
|
|
758
|
+
self.note('areas/a.md', 'a', 'area', areas=[], parent='chats')
|
|
759
|
+
self.assertIn('parent is only valid on note pages', self.run_cli('check')[2])
|
|
760
|
+
|
|
686
761
|
def test_area_pages_are_maps_without_tables_or_links_into_notes(self):
|
|
687
762
|
table = '| Screen | What it does |\n| --- | --- |\n| Profile | Edits the name. [[notes/n|Profile]] |\n'
|
|
688
763
|
for body in (table, '| a | b |\n|:-|-:|\n| 1 | 2 |', 'a | b\n--- | ---\n1 | 2', '| one column |\n| --- |\n| x |'):
|
|
@@ -27,7 +27,7 @@ A gap must not have `status`; its progress is carried by `recheck` and `decision
|
|
|
27
27
|
| `decided_by` | when `decision` is `accepted`, `rejected` or `deferred` | a person's name; the value `agent` is an error |
|
|
28
28
|
| `decided_on` | when `decision` is `accepted`, `rejected` or `deferred` | `YYYY-MM-DD` |
|
|
29
29
|
| `duplicate_of` | when `decision` is `duplicate` | id of another existing gap (not itself) |
|
|
30
|
-
| `priority`, `confidence` | optional, gap and plan pages only | short strings, e.g. `"high"` |
|
|
30
|
+
| `priority`, `confidence` | optional, gap and plan pages only | short strings, e.g. `"high"`. Use `high`, `medium` or `low` for `priority` (or `P0` to `P4`): the viewer lists the most urgent gap of each state first and can show one priority only. |
|
|
31
31
|
|
|
32
32
|
`decided_by` and `decided_on` are also allowed on `done` and `duplicate` but not required. With `outdated`, `rechecked_by` and `rechecked_on` stay in place as the record of the last recheck actually done. A fresh recheck replaces all three fields and adds a new dated entry to the `Recheck` section; keep the older entries.
|
|
33
33
|
|