project-spec 0.7.0 → 0.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +74 -1
- package/bin/project-spec.mjs +8 -2
- package/package.json +1 -1
- package/skills/project-knowledge/SKILL.md +28 -9
- package/skills/project-knowledge/assets/viewer.html +139 -26
- package/skills/project-knowledge/references/format.md +12 -5
- package/skills/project-knowledge/references/gaps.md +1 -1
- package/skills/project-knowledge/scripts/coverage.py +200 -0
- package/skills/project-knowledge/scripts/knowledge.py +127 -21
- package/skills/project-knowledge/scripts/test_coverage.py +128 -0
- package/skills/project-knowledge/scripts/test_knowledge.py +155 -0
- package/skills/spec-change/references/gaps.md +1 -1
|
@@ -54,8 +54,8 @@ Required on every page: `id`, `title`, `type`, `areas`, `sources`, `updated`. `s
|
|
|
54
54
|
|
|
55
55
|
- `id`: unique stable ASCII slug (`[a-z0-9-]+`).
|
|
56
56
|
- `status` on `area` and `note`: `current`, `partial`, `stale`, `superseded`. On `index`, `run` and `plan` additionally `planned`.
|
|
57
|
-
- `current`: checked against the current sources. This is not a runtime test or a deployment claim; say in prose which version and environment you checked.
|
|
58
|
-
- `partial`:
|
|
57
|
+
- `current`: the whole feature was read and the page checked against the current sources. This is not a runtime test or a deployment claim; say in prose which version and environment you checked.
|
|
58
|
+
- `partial`: some of the feature or area was not read. The page ends with a line saying what is left, and stays `partial` until the feature is finished.
|
|
59
59
|
- `stale`: its sources changed or disappeared and it has not been re-read.
|
|
60
60
|
- `superseded`: replaced by another page, kept as a pointer.
|
|
61
61
|
- `planned` (index, run, plan): describes intent not yet carried out.
|
|
@@ -63,6 +63,7 @@ Required on every page: `id`, `title`, `type`, `areas`, `sources`, `updated`. `s
|
|
|
63
63
|
- `areas`: existing **area IDs**. Each `note`, `gap` and `plan` needs at least one. Area pages may list themselves or leave empty; index and run pages may be empty.
|
|
64
64
|
- `sources`: existing files read as evidence, repository-relative; no directories, credentials, absolute paths or paths escaping the repo. A `current` note needs evidence; gaps, plans and maps may have none.
|
|
65
65
|
- `updated`: last substantive review date (`YYYY-MM-DD`).
|
|
66
|
+
- Optional `parent`, on a `note` only: the `id` of the feature note this note is a part of. The vault has three levels: area, feature, part. The parent is a note of the same first area that has no `parent` itself, and its body links to each of its parts; `check` rejects anything else. The viewer lists a part under its feature.
|
|
66
67
|
- Optional `tags`, `aliases`, `task_ids`: string arrays. Optional `summary` and `verification`: strings. An area's `summary` introduces it in the viewer; a note's `summary` is its plain-language statement of what the product does, shown first in the viewer's Read view (write one for every note; `check` does not require it, so older vaults still open).
|
|
67
68
|
|
|
68
69
|
Gap-only properties are errors on any other page type, `gap` is an error on a page that is not a plan, and `priority` and `confidence` are allowed only on gap and plan pages. String properties (and strings inside arrays) may not contain control characters, including line separators. Gap-only properties (`kind`, `recheck`, `rechecked_by`, `rechecked_on`, `decision`, `decided_by`, `decided_on`, `duplicate_of`, optional `priority`, `confidence`) and the plan-only `gap` property are defined in [gaps.md](gaps.md), with the rules for who may set each. Gap state (`needs-recheck`, `needs-decision`, or the decision value) is derived from them and never written into Markdown. A recheck has one meaning for every gap kind: does the `Current behavior` section still describe the code? `confirmed` means it does (confirmed, still holds; for a proposal, the capability is still absent) and `not-reproduced` means it no longer does; neither judges whether a proposal is worth doing, which is a person's decision (see [what a recheck means](gaps.md#what-a-recheck-means)). `recheck: "outdated"` means the cited sources changed after the last recorded recheck; `rechecked_by` and `rechecked_on` stay, the `Recheck` section must have text, and an open gap with it is `needs-recheck`.
|
|
@@ -81,8 +82,10 @@ A **feature note** is the one page for one screen, feature, flow or rule, and al
|
|
|
81
82
|
| Flow | `## How it works` | What a person does and what happens, in order. Left out when there is no flow. |
|
|
82
83
|
| Fields | `## Fields` | One row per field of the feature's form: `Field`, `Required`, `Rule`, `Message when refused`. |
|
|
83
84
|
| Rules | `## Business rules` | One numbered rule per row: `ID`, `Rule`. Who may do it, when an action is available, limits, defaults, what is kept, sent or deleted, and when. |
|
|
85
|
+
| Clients | `## Where the apps differ` | Only when the product has several clients: one row per difference, `What` and a column per client (`Web`, `Mobile`). One sentence instead when they were compared and do not differ. |
|
|
86
|
+
| Parts | `## Parts` | Only on a feature that has parts: a link to each part and one line saying what it covers. |
|
|
84
87
|
|
|
85
|
-
The headings are written exactly so and come in this order; a section the feature does not have is left out. A field's rule is in `Fields` and is not repeated in `Business rules`. Usually 150-400 words
|
|
88
|
+
The headings are written exactly so and come in this order; a section the feature does not have is left out. A field's rule is in `Fields` and is not repeated in `Business rules`. Usually 150-400 words and up to about twenty rules. A feature that needs more gets parts: notes with `parent` set to the feature note, each about one flow or one kind of use, never one kind of fact, and each written in this same shape. The feature note then holds what the feature is for, how the parts fit, the rules that hold in every part, and `## Parts`.
|
|
86
89
|
|
|
87
90
|
A rule is numbered `BR-<FEATURE>-<NN>` (`BR-INVITE-03`): `BR`, a short upper-case word for the feature that is the same in all rules of the note and used by no other note, and a number from 01. The number is the rule's name, quoted in tickets, tests and gaps. A rule keeps it when its text changes; a new rule takes the next free number; the number of a removed rule is not used again; rules are never renumbered. One rule per row, one or two sentences, with the value.
|
|
88
91
|
|
|
@@ -164,9 +167,11 @@ The validator enforces these body rules:
|
|
|
164
167
|
|
|
165
168
|
## Helper
|
|
166
169
|
|
|
167
|
-
|
|
170
|
+
Four scripts in `$SKILL_DIR/scripts/`; see [SKILL.md](../SKILL.md) for the variables.
|
|
168
171
|
|
|
169
|
-
`
|
|
172
|
+
`coverage.py --repo PATH [--vault PATH] [--path FOLDER]... [--all]` reads the repository's code files and the spec pages (`area` and `note`) and prints what the code has that no page mentions. It writes nothing and always exits 0. Code files are the tracked and the untracked-but-not-ignored files with a source-code extension, outside the vault, without tests, fixtures, stories, mocks, migrations, dependencies and build output. For each folder it prints the number of files and lines, how many of the files a page cites in `sources`, how many **sentences a person can see** the folder has and how many are on no page, and the same for **named numbers**. A sentence is a string or text between tags of 12 to 200 characters that starts with a capital letter and has at least three words, outside log calls; it is on a page when a page contains it, ignoring case, curly quotes and spacing. A named number is an upper-case constant set to a number (`MAX_FILES = 10`); it is on a page when a page contains its name. A file with 40 or more sentences is an article or a template and counts as a file only. A folder in which no page cites any file is one line however deep it goes; folders are ordered by how much they have that no page mentions. With `--path` only those folders are read and every unmentioned sentence and number is listed with its file, then the largest files no page cites. The first 40 folders and 60 items are printed unless `--all` is given. The report finds places to read; it cannot find a rule that has neither a message nor a named number.
|
|
173
|
+
|
|
174
|
+
`knowledge.py {check,snapshot,serve,build} --repo PATH [--vault PATH]`
|
|
170
175
|
|
|
171
176
|
- `check`: read-only, exit 1 for structural errors; freshness warnings normally leave exit status 0. Add `--strict` to also fail on warnings. Page-level errors are printed first, and a failing `check` or `snapshot` ends with a line giving the number of errors and the number of pages they are on. A page whose frontmatter fails validation still counts as a target for links, embeds, area membership, a plan's `gap` and `duplicate_of`, so one broken page yields its own errors only; fix it and nothing else needs touching. Only the "missing" error is hidden this way: an ambiguous link (two or more candidates), a link to a page of the wrong type (a spec page linking a gap, or an `areas` entry naming a page known to be another type) and a run page's link are still reported. `snapshot --notes X` refuses X while X's area references, links, `gap` or `duplicate_of` point at a page that failed validation. Missing source files are errors except on explicitly `stale` or `superseded` notes (warnings). Changed or unbaselined evidence warns. A source that differs from its baseline only by line endings (typically a Windows checkout) warns "line endings differ in source X"; do not snapshot on that checkout, because it would record the converted bytes. Nothing is rewritten.
|
|
172
177
|
|
|
@@ -204,6 +209,8 @@ Canonical Markdown must not contain raw merge conflicts. Validation rejects comp
|
|
|
204
209
|
|
|
205
210
|
`serve` shows the vault in a browser: `knowledge.py serve --repo REPO [--vault VAULT] [--port N] [--open]`. Without `--vault`, this and every other command use the single vault under `knowledge/` (`knowledge/00-index.md` or `knowledge/*/00-index.md`) and stop with a message when there is none or several. It listens on `127.0.0.1` only (port 8757, or a free one when that is taken), answers only `/` and only when addressed as this machine, never writes, and builds the page again from disk on every request, so a reload shows the current files. When the vault has errors the page lists them. The page data is the graph above plus `live: true`, `vaultPath` (the vault's path relative to the repository), `history`, `pending` and `files`. `history` is the last 30 commits that touched vault pages (not `.knowledge/`), newest first, each `{"sha", "date", "subject", "pages", "more"}`; a page entry is `{"path", "status"}` with `status` `added`, `changed` or `removed`, plus `was` and `now` when the commit changed the page's `summary` (or, failing that, its `title`); at most 80 pages are listed per commit and `more` counts the rest. `pending` lists vault pages with uncommitted changes in the same `{"path", "status"}` shape. Both are empty outside a Git work tree. `files` is the text of the repository files that pages cite in `sources` or link to, keyed by repository-relative path, each `{"text", "truncated"}`. A file is included only when it is inside the repository and outside the vault, is not a credential path, and is UTF-8 text; one file is cut at 300,000 characters and all files together at 4,000,000, after which the rest stay ordinary links. The viewer opens a link to an included file in place: Markdown rendered, anything else as numbered lines, with the lines of a `#L10` or `#L10-L20` fragment marked.
|
|
206
211
|
|
|
212
|
+
`build` writes the same viewer as one file, for a server that shows the spec to people who do not have the repository: `knowledge.py build --repo REPO [--vault VAULT] --out FILE [--source-url URL] [--with-sources]`. `--out` must be a file outside the vault. The command stops with the `check` errors when the vault has any, and writes nothing then; freshness warnings do not stop it. The page data is the graph plus `built: true`, `commit` (the full `HEAD`, empty outside Git), `sourceUrl`, `vaultPath`, `history`, `pending` and `files`. `files` is empty unless `--with-sources` is given, because the text of cited files is source code. `--source-url` must be an `http(s)` address; `{commit}` in it is replaced by `HEAD`, and it is an error to name `{commit}` outside a Git work tree. `--allow-missing-sources` is for a build that sees only part of the repository (a Docker build context): a cited path where nothing exists is not an error and does not mark its page as changed, and the command lists those files; `check` and `snapshot` never accept it. On a built page a cited file whose text is not included is a link to `sourceUrl/<repository path>` (with `#L10-L20` when lines are named) in a new tab, or plain text when there is no `sourceUrl`; it is never a relative link, which would lead into the site that hosts the page. In a shallow clone the commit whose parents were not fetched is left out of `history` (by `serve` too), since Git would show every page as added there, and `build` says that the history is short.
|
|
213
|
+
|
|
207
214
|
`assets/viewer.html` has exactly one `__KNOWLEDGE_DATA__` placeholder for script-safe JSON. Treat contents as untrusted text: no raw Markdown HTML execution, unsafe URL schemes, external scripts, fonts, analytics or fetches. Render readable basic Markdown, pipe tables and wikilinks; leave unsupported syntax as text. The viewer has a tree of areas with their notes (a note is listed under its first area; superseded notes are not listed), a search over pages, gaps and the cited files, and five places: the overview, a page, the gap queue, the map and recent changes. A note shows its title, `summary`, the page text and the gaps about it that are not closed, then its source files folded; an area shows its notes with their summaries and folds its own text. The pages linked to or from the page are a short list beside it. Navigation keeps a trail for Back and Forward and writes the place into the address (`#/page/notes/example`, `#/gaps`), so a reload or a copied link returns to it. The map draws one frame per primary area (the area page, or the first entry of `areas`) with its pages inside; an edge between a page and its primary area is not drawn because the frame says it, links inside one area are solid and links across areas dashed; `nav` pages are not drawn. Recent changes reads `history`, `pending` and the "changed source" lines of `warnings`. "Copy for AI" only composes text from the page, a choice of what the agent should do and the reader's own words, and copies it to the clipboard. The reader's name, the recently viewed pages and the last commit seen are kept in the browser's `localStorage`, and the viewer works without it.
|
|
208
215
|
|
|
209
216
|
## Format references
|
|
@@ -27,7 +27,7 @@ A gap must not have `status`; its progress is carried by `recheck` and `decision
|
|
|
27
27
|
| `decided_by` | when `decision` is `accepted`, `rejected` or `deferred` | a person's name; the value `agent` is an error |
|
|
28
28
|
| `decided_on` | when `decision` is `accepted`, `rejected` or `deferred` | `YYYY-MM-DD` |
|
|
29
29
|
| `duplicate_of` | when `decision` is `duplicate` | id of another existing gap (not itself) |
|
|
30
|
-
| `priority`, `confidence` | optional, gap and plan pages only | short strings, e.g. `"high"` |
|
|
30
|
+
| `priority`, `confidence` | optional, gap and plan pages only | short strings, e.g. `"high"`. Use `high`, `medium` or `low` for `priority` (or `P0` to `P4`): the viewer lists the most urgent gap of each state first and can show one priority only. |
|
|
31
31
|
|
|
32
32
|
`decided_by` and `decided_on` are also allowed on `done` and `duplicate` but not required. With `outdated`, `rechecked_by` and `rechecked_on` stay in place as the record of the last recheck actually done. A fresh recheck replaces all three fields and adds a new dated entry to the `Recheck` section; keep the older entries.
|
|
33
33
|
|
|
@@ -0,0 +1,200 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
"""List what the code has that no spec page mentions yet, using only the standard library.
|
|
3
|
+
|
|
4
|
+
It reads the application's code files and the spec pages and compares three things a spec is supposed to hold:
|
|
5
|
+
the files pages cite, the sentences a person can see (messages, labels, empty states) and the numeric limits.
|
|
6
|
+
What it prints is a list of places to read, not a score: a sentence that is not on a page may be a button that
|
|
7
|
+
only opens something, which the spec leaves out on purpose.
|
|
8
|
+
"""
|
|
9
|
+
import argparse
|
|
10
|
+
import os
|
|
11
|
+
import posixpath
|
|
12
|
+
import re
|
|
13
|
+
import sys
|
|
14
|
+
from pathlib import Path
|
|
15
|
+
|
|
16
|
+
import knowledge
|
|
17
|
+
|
|
18
|
+
CODE = {'.ts', '.tsx', '.js', '.jsx', '.mjs', '.cjs', '.vue', '.svelte', '.py', '.rb', '.go', '.rs', '.java', '.kt', '.kts',
|
|
19
|
+
'.swift', '.cs', '.php', '.dart', '.ex', '.exs', '.scala', '.sql', '.c', '.cc', '.cpp', '.h', '.m', '.mm'}
|
|
20
|
+
SKIP_DIRS = {'node_modules', '.git', 'dist', 'build', 'out', 'vendor', 'target', '.next', '.nuxt', '.expo', 'coverage', '__pycache__',
|
|
21
|
+
'.venv', 'venv', 'Pods', '.gradle', 'bin', 'obj', '.turbo', '.cache', 'generated', '__generated__',
|
|
22
|
+
# A migration is the history of the schema, not what the product does today.
|
|
23
|
+
'migrations', 'Migrations', 'migration'}
|
|
24
|
+
# A file with this many sentences is text someone wrote for readers (an article, a template, legal terms): the spec
|
|
25
|
+
# lists such things, it does not quote them, so they count as files and lines only.
|
|
26
|
+
CONTENT_SENTENCES = 40
|
|
27
|
+
# Tests, fixtures and stories say what the code does in someone else's words; they are not what a person sees.
|
|
28
|
+
TEST_DIR = re.compile(r'(^|/)(tests?|__tests__|__mocks__|__snapshots__|fixtures?|testdata|e2e|stories|specs?|mocks?|seeds?)(/|$)', re.I)
|
|
29
|
+
TEST_FILE = re.compile(r'(\.(test|spec|stories|e2e|mock|fixture)\.|_test\.|(^|/)test_[^/]*$|\.d\.ts$|\.min\.|\.generated\.|\.g\.dart$)', re.I)
|
|
30
|
+
FILE_LIMIT = 1_000_000
|
|
31
|
+
QUOTED = re.compile(r'''(["'`])((?:\\.|(?!\1)[^\\\n])*)\1''')
|
|
32
|
+
JSX_TEXT = re.compile(r'>\s*([A-Z][^<>{}\n]{10,200}?)\s*<')
|
|
33
|
+
LOG_LINE = re.compile(r'\b(console|logger|log|logging|Sentry|debug)\s*\.\s*\w+\s*\(|^\s*(import|from|describe|it|test|#include|using|package)\b|^\s*(//|#|\*|/\*)')
|
|
34
|
+
NOT_PROSE = re.compile(r'[{}<>;=\\]|\$\(|^\s*(SELECT|INSERT|UPDATE|DELETE|CREATE|ALTER|DROP|WITH)\b|https?://|\w+\.\w+\(|\w/\w+/\w')
|
|
35
|
+
# A named number: MAX_ATTACHMENTS = 10, TTL_MS: number = 60 * 60 * 1000
|
|
36
|
+
LIMIT = re.compile(r'\b([A-Z][A-Z0-9]*(?:_[A-Z0-9]+)+)\s*(?::[^=\n]{1,40})?=\s*([-+]?\d[\d_.]*(?:\s*[*/+-]\s*\d[\d_.]*)*)\s*[;,]?\s*(?:(?://|#).*)?$', re.M)
|
|
37
|
+
PUNCTUATION = str.maketrans({'‘': "'", '’': "'", '“': '"', '”': '"', '–': '-', '—': '-', ' ': ' '})
|
|
38
|
+
|
|
39
|
+
|
|
40
|
+
def plain(text):
|
|
41
|
+
"""Text as it is compared: case, curly quotes, runs of spaces and Markdown's escaped pipe do not count."""
|
|
42
|
+
return re.sub(r'\s+', ' ', text.translate(PUNCTUATION).replace('\\|', '|')).casefold().strip()
|
|
43
|
+
|
|
44
|
+
|
|
45
|
+
def is_sentence(text):
|
|
46
|
+
"""Does this string read like something shown to a person, rather than a key, a class list or a query?"""
|
|
47
|
+
text = text.strip()
|
|
48
|
+
if not 12 <= len(text) <= 200 or not text[0].isupper() or len(text.split()) < 3 or NOT_PROSE.search(text):
|
|
49
|
+
return False
|
|
50
|
+
letters = sum(1 for c in text if c.isalpha() or c == ' ')
|
|
51
|
+
return letters / len(text) >= 0.8
|
|
52
|
+
|
|
53
|
+
|
|
54
|
+
def sentences_in(text):
|
|
55
|
+
found = []
|
|
56
|
+
for line in text.split('\n'):
|
|
57
|
+
if LOG_LINE.search(line):
|
|
58
|
+
continue
|
|
59
|
+
for _, inner in QUOTED.findall(line):
|
|
60
|
+
inner = re.sub(r"\\(['\"`])", r'\1', inner)
|
|
61
|
+
if is_sentence(inner):
|
|
62
|
+
found.append(inner.strip())
|
|
63
|
+
found += [m.strip() for m in JSX_TEXT.findall(line) if is_sentence(m)]
|
|
64
|
+
return list(dict.fromkeys(found))
|
|
65
|
+
|
|
66
|
+
|
|
67
|
+
def code_files(repo, vault):
|
|
68
|
+
"""Repository-relative paths of the application's own code: tracked or untracked but not ignored, no tests."""
|
|
69
|
+
listed = knowledge.run_git(repo, ['ls-files', '-z', '--cached', '--others', '--exclude-standard'])
|
|
70
|
+
if listed.returncode == 0:
|
|
71
|
+
names = [os.fsdecode(n) for n in listed.stdout.split(b'\0') if n]
|
|
72
|
+
else:
|
|
73
|
+
names = []
|
|
74
|
+
for folder, dirs, files in os.walk(repo):
|
|
75
|
+
dirs[:] = sorted(d for d in dirs if d not in SKIP_DIRS and not d.startswith('.'))
|
|
76
|
+
names += [Path(os.path.relpath(Path(folder) / f, repo)).as_posix() for f in sorted(files)]
|
|
77
|
+
inside = Path(os.path.relpath(vault, repo)).as_posix() + '/'
|
|
78
|
+
return sorted(n for n in names
|
|
79
|
+
if posixpath.splitext(n)[1].lower() in CODE and not n.startswith(inside)
|
|
80
|
+
and not any(part in SKIP_DIRS for part in n.split('/')[:-1])
|
|
81
|
+
and not TEST_DIR.search(posixpath.dirname(n)) and not TEST_FILE.search(n))
|
|
82
|
+
|
|
83
|
+
|
|
84
|
+
def measure(repo, vault, notes, prefixes=()):
|
|
85
|
+
"""Per code file: lines, whether a page cites it, and the sentences and limits no page mentions."""
|
|
86
|
+
spec = [n for n in notes if n['meta']['type'] in {'area', 'note'}]
|
|
87
|
+
pages = plain('\n'.join(f"{n['meta']['title']}\n{n['meta'].get('summary', '')}\n{n['body']}" for n in spec))
|
|
88
|
+
words = set(re.findall(r'[A-Z][A-Z0-9_]+', '\n'.join(n['body'] for n in spec)))
|
|
89
|
+
cited = {posixpath.normpath(s) for n in spec for s in n['meta']['sources']}
|
|
90
|
+
files = {}
|
|
91
|
+
for name in code_files(repo, vault):
|
|
92
|
+
if prefixes and not any(name == p or name.startswith(p.rstrip('/') + '/') for p in prefixes):
|
|
93
|
+
continue
|
|
94
|
+
path = repo / name
|
|
95
|
+
try:
|
|
96
|
+
if path.is_symlink() or path.stat().st_size > FILE_LIMIT:
|
|
97
|
+
continue
|
|
98
|
+
text = path.read_bytes().decode('utf-8')
|
|
99
|
+
except (OSError, UnicodeDecodeError):
|
|
100
|
+
continue
|
|
101
|
+
sentences = sentences_in(text)
|
|
102
|
+
content = len(sentences) >= CONTENT_SENTENCES
|
|
103
|
+
if content:
|
|
104
|
+
sentences = []
|
|
105
|
+
limits = list(dict.fromkeys(LIMIT.findall(text)))
|
|
106
|
+
files[name] = {
|
|
107
|
+
'lines': text.count('\n') + 1, 'cited': name in cited, 'content': content, 'sentences': len(sentences), 'limits': len(limits),
|
|
108
|
+
'unquoted': [s for s in sentences if plain(s).rstrip('.!?:…') not in pages],
|
|
109
|
+
'unnamed': [(n, v.strip()) for n, v in limits if n not in words]}
|
|
110
|
+
return files
|
|
111
|
+
|
|
112
|
+
|
|
113
|
+
def folders(files, prefixes=()):
|
|
114
|
+
"""Group files into the places to read: a folder no page cites anything in is one line, however deep it goes.
|
|
115
|
+
|
|
116
|
+
With prefixes (the folders the caller asked about) a line never names a folder above the one asked for."""
|
|
117
|
+
cited_under = {}
|
|
118
|
+
for name, info in files.items():
|
|
119
|
+
if info['cited']:
|
|
120
|
+
parts = name.split('/')[:-1]
|
|
121
|
+
for i in range(len(parts) + 1):
|
|
122
|
+
cited_under['/'.join(parts[:i])] = True
|
|
123
|
+
groups = {}
|
|
124
|
+
for name, info in files.items():
|
|
125
|
+
parts = name.split('/')[:-1]
|
|
126
|
+
folder = '/'.join(parts)
|
|
127
|
+
floor = max((len(p.split('/')) for p in prefixes if name.startswith(p.rstrip('/') + '/')), default=1)
|
|
128
|
+
# Climb to the highest folder that still has no cited file under it.
|
|
129
|
+
for i in range(min(floor, len(parts)) or 1, len(parts) + 1):
|
|
130
|
+
if not cited_under.get('/'.join(parts[:i])):
|
|
131
|
+
folder = '/'.join(parts[:i])
|
|
132
|
+
break
|
|
133
|
+
g = groups.setdefault(folder or '.', {'files': 0, 'lines': 0, 'cited': 0, 'sentences': 0, 'unquoted': 0, 'limits': 0, 'unnamed': 0})
|
|
134
|
+
g['files'] += 1
|
|
135
|
+
g['lines'] += info['lines']
|
|
136
|
+
g['cited'] += info['cited']
|
|
137
|
+
g['sentences'] += info['sentences']
|
|
138
|
+
g['unquoted'] += len(info['unquoted'])
|
|
139
|
+
g['limits'] += info['limits']
|
|
140
|
+
g['unnamed'] += len(info['unnamed'])
|
|
141
|
+
return groups
|
|
142
|
+
|
|
143
|
+
|
|
144
|
+
def main(argv=None):
|
|
145
|
+
parser = argparse.ArgumentParser(description=__doc__.split('\n')[0])
|
|
146
|
+
parser.add_argument('--repo', required=True, type=Path)
|
|
147
|
+
parser.add_argument('--vault', type=Path, help='default: the single vault under knowledge/ in the repository')
|
|
148
|
+
parser.add_argument('--path', action='append', default=[], metavar='FOLDER',
|
|
149
|
+
help='only this folder or file of the repository, and list each sentence and limit no page mentions; repeatable')
|
|
150
|
+
parser.add_argument('--all', action='store_true', help='print every line instead of the first 40 folders or 60 items')
|
|
151
|
+
args = parser.parse_args(argv)
|
|
152
|
+
repo = args.repo.resolve()
|
|
153
|
+
if not repo.is_dir():
|
|
154
|
+
parser.error('repo must be an existing directory')
|
|
155
|
+
try:
|
|
156
|
+
vault = args.vault.resolve() if args.vault else knowledge.find_vault(repo)
|
|
157
|
+
except ValueError as exc:
|
|
158
|
+
parser.error(str(exc))
|
|
159
|
+
if not vault.is_dir():
|
|
160
|
+
parser.error('repo and vault must be existing directories')
|
|
161
|
+
notes, _, errors, _, _ = knowledge.load_vault(repo, vault, freshness=False, absent=[])
|
|
162
|
+
if errors:
|
|
163
|
+
print(f'coverage: the vault has {len(errors)} errors; pages with errors are not counted (run check)', file=sys.stderr)
|
|
164
|
+
prefixes = [posixpath.normpath(p.replace(os.sep, '/')) for p in args.path]
|
|
165
|
+
files = measure(repo, vault, notes, prefixes)
|
|
166
|
+
if not files:
|
|
167
|
+
print('coverage: no code files found' + (' under ' + ', '.join(prefixes) if prefixes else ''))
|
|
168
|
+
return 0
|
|
169
|
+
total = lambda key: sum(len(f[key]) if isinstance(f[key], list) else f[key] for f in files.values())
|
|
170
|
+
print(f"coverage: {len(files)} code files, {total('lines')} lines; pages cite {total('cited')} of them")
|
|
171
|
+
print(f"coverage: {total('sentences')} sentences a person can see, {total('unquoted')} on no page; "
|
|
172
|
+
f"{total('limits')} named numbers, {total('unnamed')} on no page")
|
|
173
|
+
texts = sum(1 for f in files.values() if f['content'])
|
|
174
|
+
print('coverage: this is a list of places to read, not a score; tests, fixtures, migrations and generated code are left out'
|
|
175
|
+
+ (f'; {texts} files that are mostly text (articles, templates) count as files only' if texts else ''))
|
|
176
|
+
groups = sorted(folders(files, prefixes).items(), key=lambda kv: (-(kv[1]['unquoted'] + kv[1]['unnamed']), -kv[1]['lines'], kv[0]))
|
|
177
|
+
shown = groups if args.all else groups[:40]
|
|
178
|
+
width = max(len(name) for name, _ in shown)
|
|
179
|
+
print(f"\n{'folder'.ljust(width)} files lines cited sentences not on a page numbers not on a page")
|
|
180
|
+
for name, g in shown:
|
|
181
|
+
print(f"{name.ljust(width)} {g['files']:5} {g['lines']:6} {g['cited']:5} {g['unquoted']:9} of {g['sentences']:<9} {g['unnamed']:7} of {g['limits']}")
|
|
182
|
+
if len(groups) > len(shown):
|
|
183
|
+
print(f'... and {len(groups) - len(shown)} more folders (--all prints them)')
|
|
184
|
+
if prefixes:
|
|
185
|
+
items = [(name, 'says', s) for name, f in files.items() for s in f['unquoted']]
|
|
186
|
+
items += [(name, 'sets', f'{n} = {v}') for name, f in files.items() for n, v in f['unnamed']]
|
|
187
|
+
print(f'\n{len(items)} things under {", ".join(prefixes)} that no page mentions:')
|
|
188
|
+
for name, verb, what in (items if args.all else items[:60]):
|
|
189
|
+
print(f' {name} {verb}: {what}')
|
|
190
|
+
if not args.all and len(items) > 60:
|
|
191
|
+
print(f' ... and {len(items) - 60} more (--all prints them)')
|
|
192
|
+
uncited = sorted((f['lines'], name) for name, f in files.items() if not f['cited'])
|
|
193
|
+
print(f'\n{len(uncited)} of {len(files)} files there are cited by no page; the largest:')
|
|
194
|
+
for lines, name in reversed(uncited[-(len(uncited) if args.all else 15):]):
|
|
195
|
+
print(f' {name} ({lines} lines)')
|
|
196
|
+
return 0
|
|
197
|
+
|
|
198
|
+
|
|
199
|
+
if __name__ == '__main__':
|
|
200
|
+
sys.exit(main())
|
|
@@ -37,7 +37,7 @@ RULE_ROW = re.compile(r'^ {0,3}\|[ \t]*(BR-[A-Z0-9]+(?:-[A-Z0-9]+)*-\d{2,4})[ \t
|
|
|
37
37
|
MD_REF = re.compile(r'^ {0,3}\[[^\]\n]+\]:\s*(<[^>]*>|\S+)', re.M)
|
|
38
38
|
GAP_PAGE_ONLY = ('priority', 'confidence')
|
|
39
39
|
ARRAYS = {'areas', 'sources', 'tags', 'aliases', 'task_ids'}
|
|
40
|
-
STRINGS = {'id', 'title', 'type', 'status', 'updated', 'summary', 'verification', 'priority', 'confidence', 'gap',
|
|
40
|
+
STRINGS = {'id', 'title', 'type', 'status', 'updated', 'summary', 'verification', 'priority', 'confidence', 'gap', 'parent',
|
|
41
41
|
*GAP_ONLY}
|
|
42
42
|
REQUIRED = {'id', 'title', 'type', 'areas', 'sources', 'updated'}
|
|
43
43
|
LINK = re.compile(r'(!?)\[\[([^\[\]\n]+)\]\]') # no brackets inside, so a run of '[[' cannot backtrack
|
|
@@ -215,6 +215,8 @@ def validate_meta(meta):
|
|
|
215
215
|
raise ValueError(f"status must be one of {', '.join(sorted(allowed))}{hint}")
|
|
216
216
|
if kind == 'note' and meta['status'] == 'current' and not meta['sources']:
|
|
217
217
|
raise ValueError('current note requires sources')
|
|
218
|
+
if 'parent' in meta and (kind != 'note' or not meta['parent'].strip()):
|
|
219
|
+
raise ValueError('parent is only valid on note pages and names the feature note this one is a part of')
|
|
218
220
|
if kind == 'plan':
|
|
219
221
|
if not meta.get('gap', '').strip():
|
|
220
222
|
raise ValueError('plan requires gap: the id of the accepted gap it implements')
|
|
@@ -313,20 +315,29 @@ def line_ending_variant(data, old):
|
|
|
313
315
|
return old in {hashlib.sha256(lf).hexdigest(), hashlib.sha256(lf.replace(b'\n', b'\r\n')).hexdigest()}
|
|
314
316
|
|
|
315
317
|
|
|
318
|
+
def run_git(repo, args, **options):
|
|
319
|
+
"""Git's answer for the repository. Where Git is not installed (a build container, say) it is the answer
|
|
320
|
+
of a folder that is not a work tree, so the caller goes on without history instead of stopping."""
|
|
321
|
+
try:
|
|
322
|
+
return subprocess.run(['git', '-C', str(repo), *args], capture_output=True, **options)
|
|
323
|
+
except FileNotFoundError:
|
|
324
|
+
return subprocess.CompletedProcess(args, 127, b'', b'')
|
|
325
|
+
|
|
326
|
+
|
|
316
327
|
def head_blob(repo, source):
|
|
317
328
|
"""Bytes of the source as committed at HEAD, or None when HEAD has no such file (untracked, new or no commits)."""
|
|
318
|
-
result =
|
|
329
|
+
result = run_git(repo, ['cat-file', 'blob', f'HEAD:./{posixpath.normpath(source)}'])
|
|
319
330
|
return None if result.returncode else result.stdout
|
|
320
331
|
|
|
321
332
|
|
|
322
333
|
def uncommitted_paths(repo):
|
|
323
334
|
"""Repo-relative paths Git reports as modified, staged or untracked (one status call), or None outside a work tree."""
|
|
324
|
-
probe =
|
|
335
|
+
probe = run_git(repo, ['rev-parse', '--is-inside-work-tree', '--show-prefix'])
|
|
325
336
|
lines = probe.stdout.decode(errors='replace').split('\n')
|
|
326
337
|
if probe.returncode or lines[0] != 'true':
|
|
327
338
|
return None
|
|
328
339
|
prefix = lines[1] if len(lines) > 1 else ''
|
|
329
|
-
status =
|
|
340
|
+
status = run_git(repo, ['status', '--porcelain=v1', '-z', '--untracked-files=all'])
|
|
330
341
|
if status.returncode:
|
|
331
342
|
return None
|
|
332
343
|
raw, found, i = status.stdout.split(b'\0'), set(), 0
|
|
@@ -347,10 +358,13 @@ def error_page(message):
|
|
|
347
358
|
return head if head.endswith('.md') else None
|
|
348
359
|
|
|
349
360
|
|
|
350
|
-
def load_vault(repo, vault, freshness=True, dirty=None, report=None):
|
|
361
|
+
def load_vault(repo, vault, freshness=True, dirty=None, report=None, absent=None):
|
|
351
362
|
"""dirty: repo-relative paths with uncommitted changes. Their freshness warning is marked, not counted, only when
|
|
352
363
|
the accepted fingerprint equals the file's HEAD blob (up to LF/CRLF), so the sole drift is uncommitted.
|
|
353
364
|
|
|
365
|
+
absent, when given, collects cited files that are not there instead of reporting them: for a build that sees
|
|
366
|
+
only part of the repository, such as a Docker build context. Their pages are not marked as changed.
|
|
367
|
+
|
|
354
368
|
report, when given, receives 'failed': {page path: parsed id or None} for pages that failed validation.
|
|
355
369
|
"""
|
|
356
370
|
errors, warnings, notes, failed, failed_kind, cascade = [], [], [], {}, {}, []
|
|
@@ -415,10 +429,26 @@ def load_vault(repo, vault, freshness=True, dirty=None, report=None):
|
|
|
415
429
|
paths[stem] = note
|
|
416
430
|
basenames.setdefault(PurePosixPath(stem).name, []).append(note)
|
|
417
431
|
area_ids = {n['meta']['id'] for n in notes if n['meta']['type'] == 'area'}
|
|
418
|
-
edges, attachments, rule_pages = set(), {}, {}
|
|
432
|
+
edges, attachments, rule_pages, linked, parts = set(), {}, {}, set(), []
|
|
419
433
|
for note in notes:
|
|
420
434
|
meta, rel = note['meta'], note['path']
|
|
421
435
|
note['stale'] = False
|
|
436
|
+
if meta.get('parent'):
|
|
437
|
+
# Three levels and no more: an area, its features, and the parts of a feature that is too big for one page.
|
|
438
|
+
other, name = ids.get(meta['parent']), meta['parent']
|
|
439
|
+
if other is None and name in failed_ids and failed_id_kinds[name] & {None, 'note'}:
|
|
440
|
+
cascade.append(f'{rel}: parent {name} names a page that failed validation')
|
|
441
|
+
elif other is None or other['meta']['type'] != 'note':
|
|
442
|
+
errors.append(f'{rel}: parent {name} is not the id of an existing note')
|
|
443
|
+
elif other is note:
|
|
444
|
+
errors.append(f'{rel}: parent must name a different note')
|
|
445
|
+
elif other['meta'].get('parent'):
|
|
446
|
+
errors.append(f"{rel}: parent {name} is itself a part of {other['meta']['parent']}; a part belongs to a feature, not to another part")
|
|
447
|
+
elif other['meta']['areas'][:1] != meta['areas'][:1]:
|
|
448
|
+
errors.append(f"{rel}: parent {name} is in another area; a part has the same first area as its feature")
|
|
449
|
+
else:
|
|
450
|
+
edges.add((other['meta']['id'], meta['id']))
|
|
451
|
+
parts.append((other, note))
|
|
422
452
|
if meta['type'] == 'gap':
|
|
423
453
|
note['state'] = gap_state(meta)
|
|
424
454
|
if meta['type'] == 'gap' and meta.get('duplicate_of'):
|
|
@@ -495,6 +525,7 @@ def load_vault(repo, vault, freshness=True, dirty=None, report=None):
|
|
|
495
525
|
"if that part needs its own link, make it its own note and list it here")
|
|
496
526
|
else:
|
|
497
527
|
edges.add((meta['id'], choices[0]['meta']['id']))
|
|
528
|
+
linked.add((meta['id'], choices[0]['meta']['id']))
|
|
498
529
|
if meta['type'] == 'area':
|
|
499
530
|
# An area page is a map of its notes. A table on it is behavior written in the wrong place.
|
|
500
531
|
tables = sum(1 for line in text.split('\n') if TABLE_RULE.match(line))
|
|
@@ -541,6 +572,9 @@ def load_vault(repo, vault, freshness=True, dirty=None, report=None):
|
|
|
541
572
|
if credential_path(path.resolve().relative_to(repo).as_posix()):
|
|
542
573
|
raise ValueError('credential source is forbidden')
|
|
543
574
|
if not path.is_file():
|
|
575
|
+
if absent is not None and not path.exists():
|
|
576
|
+
absent.append(source)
|
|
577
|
+
continue
|
|
544
578
|
if closed:
|
|
545
579
|
warnings.append(f'{rel}: missing source {source}')
|
|
546
580
|
note['stale'] = True
|
|
@@ -564,6 +598,10 @@ def load_vault(repo, vault, freshness=True, dirty=None, report=None):
|
|
|
564
598
|
note['stale'] = True
|
|
565
599
|
except (OSError, ValueError) as exc:
|
|
566
600
|
errors.append(f'{rel}: {source}: {exc}')
|
|
601
|
+
for feature, part in parts:
|
|
602
|
+
# The viewer nests a part under its feature from `parent`; a plain Markdown reader has only the link.
|
|
603
|
+
if (feature['meta']['id'], part['meta']['id']) not in linked:
|
|
604
|
+
errors.append(f"{part['path']}: its feature {feature['path']} does not link to it; list every part on the feature's page")
|
|
567
605
|
if report is not None:
|
|
568
606
|
report['failed'] = failed
|
|
569
607
|
report['cascade'] = cascade
|
|
@@ -634,7 +672,7 @@ def node_of(note):
|
|
|
634
672
|
'kind': m.get('kind', ''), 'recheck': m.get('recheck', ''), 'recheckedBy': m.get('rechecked_by', ''),
|
|
635
673
|
'recheckedOn': m.get('rechecked_on', ''), 'decision': m.get('decision', ''), 'decidedBy': m.get('decided_by', ''),
|
|
636
674
|
'decidedOn': m.get('decided_on', ''), 'priority': m.get('priority', ''), 'confidence': m.get('confidence', ''),
|
|
637
|
-
'duplicateOf': m.get('duplicate_of', ''), 'recheckOutdated': gap and (m['recheck'] == 'outdated' or recheck_marker(note)), 'state': note.get('state', ''), 'gap': m.get('gap', ''),
|
|
675
|
+
'parent': m.get('parent', ''), 'duplicateOf': m.get('duplicate_of', ''), 'recheckOutdated': gap and (m['recheck'] == 'outdated' or recheck_marker(note)), 'state': note.get('state', ''), 'gap': m.get('gap', ''),
|
|
638
676
|
'layer': LAYERS[m['type']]}
|
|
639
677
|
|
|
640
678
|
|
|
@@ -689,6 +727,17 @@ def front_value(line):
|
|
|
689
727
|
return value if isinstance(value, str) else ''
|
|
690
728
|
|
|
691
729
|
|
|
730
|
+
def shallow_commits(repo):
|
|
731
|
+
"""Commits whose parents a shallow clone left out. Empty in a full clone and outside Git."""
|
|
732
|
+
found = run_git(repo, ['rev-parse', '--git-path', 'shallow'])
|
|
733
|
+
if found.returncode:
|
|
734
|
+
return set()
|
|
735
|
+
try:
|
|
736
|
+
return set((repo / os.fsdecode(found.stdout).strip()).read_text(encoding='ascii').split())
|
|
737
|
+
except (OSError, UnicodeDecodeError):
|
|
738
|
+
return set()
|
|
739
|
+
|
|
740
|
+
|
|
692
741
|
def vault_history(repo, vault):
|
|
693
742
|
"""Recent commits that touched vault pages, newest first: sha, date, subject and what happened to each page.
|
|
694
743
|
|
|
@@ -698,29 +747,31 @@ def vault_history(repo, vault):
|
|
|
698
747
|
spec, prefix = page_pathspec(repo, vault)
|
|
699
748
|
if spec is None:
|
|
700
749
|
return []
|
|
701
|
-
head =
|
|
750
|
+
head = run_git(repo, ['rev-parse', 'HEAD'])
|
|
702
751
|
if head.returncode:
|
|
703
752
|
return []
|
|
704
753
|
key = (str(repo), str(vault), head.stdout.strip())
|
|
705
754
|
if key in _history_cache:
|
|
706
755
|
return _history_cache[key]
|
|
707
756
|
# Git prints paths from the top of the work tree, which may be above the repository folder given here.
|
|
708
|
-
top =
|
|
757
|
+
top = run_git(repo, ['rev-parse', '--show-prefix']).stdout.decode(errors='replace').strip()
|
|
709
758
|
prefix = top + prefix
|
|
710
759
|
# A marker nobody could have written into a commit message or a page beforehand separates the commits.
|
|
711
760
|
marker = '\x01' + os.urandom(12).hex() + '\x01'
|
|
712
761
|
try:
|
|
713
|
-
log =
|
|
762
|
+
log = run_git(repo, ['-c', 'core.quotepath=off', 'log', f'-n{HISTORY_COMMITS}', '--date=short', '--no-color',
|
|
714
763
|
'--no-renames', '-p', '-U0', f'--format={marker}%H%x02%ad%x02%s', '--', *spec],
|
|
715
|
-
|
|
764
|
+
timeout=20)
|
|
716
765
|
except subprocess.TimeoutExpired:
|
|
717
766
|
return []
|
|
718
767
|
if log.returncode:
|
|
719
768
|
return []
|
|
720
|
-
commits = []
|
|
769
|
+
commits, boundary = [], shallow_commits(repo)
|
|
721
770
|
for chunk in log.stdout.decode('utf-8', errors='replace').split(marker)[1:]:
|
|
722
771
|
header, _, body = chunk.partition('\n')
|
|
723
772
|
sha, date, subject = (header.split('\x02', 2) + ['', ''])[:3]
|
|
773
|
+
if sha in boundary: # its parents were not fetched, so Git would show every page as added
|
|
774
|
+
continue
|
|
724
775
|
subject = CONTROL.sub('', subject)
|
|
725
776
|
pages, page = [], None
|
|
726
777
|
for line in body.split('\n'):
|
|
@@ -755,11 +806,10 @@ def vault_pending(repo, vault):
|
|
|
755
806
|
spec, prefix = page_pathspec(repo, vault)
|
|
756
807
|
if spec is None:
|
|
757
808
|
return []
|
|
758
|
-
status =
|
|
759
|
-
capture_output=True)
|
|
809
|
+
status = run_git(repo, ['-c', 'core.quotepath=off', 'status', '--porcelain=v1', '-z', '--untracked-files=all', '--no-renames', '--', *spec])
|
|
760
810
|
if status.returncode:
|
|
761
811
|
return []
|
|
762
|
-
top =
|
|
812
|
+
top = run_git(repo, ['rev-parse', '--show-prefix']).stdout.decode(errors='replace').strip()
|
|
763
813
|
pending = []
|
|
764
814
|
for item in status.stdout.split(b'\0'):
|
|
765
815
|
text = os.fsdecode(item)
|
|
@@ -772,6 +822,13 @@ def vault_pending(repo, vault):
|
|
|
772
822
|
return sorted(pending, key=lambda p: p['path'])
|
|
773
823
|
|
|
774
824
|
|
|
825
|
+
def page_payload(repo, vault, notes, edges, warnings, sources=True):
|
|
826
|
+
files = cited_files(repo, vault, notes)[0] if sources else {}
|
|
827
|
+
return graph_of(notes, edges, warnings) | {
|
|
828
|
+
'vaultPath': Path(os.path.relpath(vault, repo)).as_posix(), 'files': files,
|
|
829
|
+
'history': vault_history(repo, vault), 'pending': vault_pending(repo, vault)}
|
|
830
|
+
|
|
831
|
+
|
|
775
832
|
def live_page(repo, vault):
|
|
776
833
|
"""The viewer for the vault as it is on disk right now, or a page listing what stops it from loading."""
|
|
777
834
|
try:
|
|
@@ -785,10 +842,29 @@ def live_page(repo, vault):
|
|
|
785
842
|
f'<h1>The spec has {len(errors)} error{"s" if len(errors) != 1 else ""}</h1>'
|
|
786
843
|
'<p>Fix the pages below (the <code>check</code> command prints the same list), then reload.</p>'
|
|
787
844
|
f'<ul>{items}</ul></body>')
|
|
788
|
-
|
|
789
|
-
|
|
790
|
-
|
|
791
|
-
|
|
845
|
+
return viewer_html(page_payload(repo, vault, notes, edges, warnings) | {'live': True})
|
|
846
|
+
|
|
847
|
+
|
|
848
|
+
def source_base(value, commit):
|
|
849
|
+
"""The address `--source-url` names, with `{commit}` filled in. Raises ValueError with the reason when it cannot be used."""
|
|
850
|
+
if CONTROL.search(value) or ' ' in value or not re.match(r'https?://[^/]', value, re.I):
|
|
851
|
+
raise ValueError('--source-url must be an http(s) address, e.g. https://github.com/OWNER/REPO/blob/{commit}')
|
|
852
|
+
if '{commit}' in value and not commit:
|
|
853
|
+
raise ValueError('--source-url names {commit}, but --repo is not a Git work tree with a commit')
|
|
854
|
+
return value.replace('{commit}', commit).rstrip('/')
|
|
855
|
+
|
|
856
|
+
|
|
857
|
+
def built_page(repo, vault, notes, edges, warnings, with_sources=False, source_url=''):
|
|
858
|
+
"""The viewer as one file to put on a server: the vault as it is now, with nothing read later.
|
|
859
|
+
|
|
860
|
+
The text of cited repository files is left out unless asked for, because the file may be shown to people
|
|
861
|
+
who cannot read the repository. Returns (html, payload).
|
|
862
|
+
"""
|
|
863
|
+
head = run_git(repo, ['rev-parse', 'HEAD'])
|
|
864
|
+
commit = '' if head.returncode else head.stdout.decode('ascii', errors='replace').strip()
|
|
865
|
+
payload = page_payload(repo, vault, notes, edges, warnings, with_sources) | {
|
|
866
|
+
'built': True, 'commit': commit, 'sourceUrl': source_base(source_url, commit) if source_url else ''}
|
|
867
|
+
return viewer_html(payload), payload
|
|
792
868
|
|
|
793
869
|
|
|
794
870
|
def make_server(repo, vault, port):
|
|
@@ -820,12 +896,17 @@ def make_server(repo, vault, port):
|
|
|
820
896
|
|
|
821
897
|
def main(argv=None):
|
|
822
898
|
parser = argparse.ArgumentParser()
|
|
823
|
-
parser.add_argument('command', choices=['check', 'snapshot', 'serve'])
|
|
899
|
+
parser.add_argument('command', choices=['check', 'snapshot', 'serve', 'build'])
|
|
824
900
|
parser.add_argument('--repo', required=True, type=Path)
|
|
825
901
|
parser.add_argument('--vault', type=Path, help='default: the single vault under knowledge/ in the repository')
|
|
826
902
|
parser.add_argument('--strict', action='store_true', help='fail check when freshness warnings exist')
|
|
827
903
|
parser.add_argument('--port', type=int, default=8757, help='serve: port on 127.0.0.1 (another free port is used if it is taken)')
|
|
828
904
|
parser.add_argument('--open', action='store_true', help='serve: open the page in the default browser')
|
|
905
|
+
parser.add_argument('--out', type=Path, help='build: the HTML file to write, outside the vault')
|
|
906
|
+
parser.add_argument('--with-sources', action='store_true', help='build: include the text of the repository files the pages cite')
|
|
907
|
+
parser.add_argument('--source-url', default='', help='build: where a cited file can be read, e.g. https://github.com/OWNER/REPO/blob/{commit}')
|
|
908
|
+
parser.add_argument('--allow-missing-sources', action='store_true',
|
|
909
|
+
help='build: do not stop when a cited file is not there, for a build that sees only part of the repository')
|
|
829
910
|
parser.add_argument('--ignore-dirty', action='store_true',
|
|
830
911
|
help='check: do not count freshness warnings for sources with uncommitted changes (needs a Git work tree)')
|
|
831
912
|
group = parser.add_mutually_exclusive_group()
|
|
@@ -838,6 +919,10 @@ def main(argv=None):
|
|
|
838
919
|
parser.error('--ignore-dirty is only for check')
|
|
839
920
|
if args.open and args.command != 'serve':
|
|
840
921
|
parser.error('--open is only for serve')
|
|
922
|
+
if args.command == 'build' and not args.out:
|
|
923
|
+
parser.error('build requires --out FILE')
|
|
924
|
+
if args.command != 'build' and (args.out or args.with_sources or args.source_url or args.allow_missing_sources):
|
|
925
|
+
parser.error('--out, --with-sources, --source-url and --allow-missing-sources are only for build')
|
|
841
926
|
if args.command == 'snapshot' and not (args.all or args.notes):
|
|
842
927
|
parser.error('snapshot requires --all or --notes')
|
|
843
928
|
if args.command != 'snapshot' and (args.all or args.notes):
|
|
@@ -871,7 +956,8 @@ def main(argv=None):
|
|
|
871
956
|
if dirty is None:
|
|
872
957
|
parser.error('--ignore-dirty needs --repo to be a Git work tree')
|
|
873
958
|
report = {}
|
|
874
|
-
|
|
959
|
+
absent = [] if args.allow_missing_sources else None
|
|
960
|
+
notes, edges, errors, warnings, baseline = load_vault(repo, vault, args.command != 'snapshot', dirty, report, absent)
|
|
875
961
|
ignored = []
|
|
876
962
|
if args.command == 'snapshot' and args.notes:
|
|
877
963
|
# Only the selected pages must be valid; errors elsewhere are counted, not blocking.
|
|
@@ -904,6 +990,26 @@ def main(argv=None):
|
|
|
904
990
|
return 1
|
|
905
991
|
if ignored:
|
|
906
992
|
print(f'{args.command}: {len(ignored)} errors on {len({error_page(m) for m in ignored if error_page(m)})} other pages do not block this snapshot (run check to see them)', file=sys.stderr)
|
|
993
|
+
if args.command == 'build':
|
|
994
|
+
out = Path(os.path.abspath(args.out))
|
|
995
|
+
if out.is_dir() or out.resolve().is_relative_to(vault):
|
|
996
|
+
parser.error('--out must be a file outside the vault: nothing is generated into the vault')
|
|
997
|
+
try:
|
|
998
|
+
page, payload = built_page(repo, vault, notes, edges, warnings, args.with_sources, args.source_url)
|
|
999
|
+
except ValueError as exc:
|
|
1000
|
+
parser.error(str(exc))
|
|
1001
|
+
atomic_write(out, page)
|
|
1002
|
+
files = len(payload['files'])
|
|
1003
|
+
print(f'build: wrote {out} ({len(page.encode("utf-8")) // 1024} kB), {len(notes)} notes, {len(warnings)} warnings'
|
|
1004
|
+
+ (f', commit {payload["commit"][:7]}' if payload['commit'] else ''))
|
|
1005
|
+
print('build: ' + (f'the text of {files} cited source files is inside the file; show it only to people who may read the repository'
|
|
1006
|
+
if args.with_sources else 'cited source files are named but their text is not included (--with-sources adds it)'))
|
|
1007
|
+
if absent:
|
|
1008
|
+
names = sorted(set(absent))
|
|
1009
|
+
print(f'build: {len(names)} cited files are not in this folder, so nobody compared them with the pages: ' + ', '.join(names[:8]) + (' ...' if len(names) > 8 else ''))
|
|
1010
|
+
if shallow_commits(repo):
|
|
1011
|
+
print('build: this is a shallow clone, so the Changes page starts at the first commit that was fetched in full; fetch the whole history to show more')
|
|
1012
|
+
return 0
|
|
907
1013
|
if args.command == 'snapshot':
|
|
908
1014
|
for note in notes:
|
|
909
1015
|
meta = note['meta']
|