project-spec 0.7.0 → 0.9.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -54,8 +54,8 @@ Required on every page: `id`, `title`, `type`, `areas`, `sources`, `updated`. `s
54
54
 
55
55
  - `id`: unique stable ASCII slug (`[a-z0-9-]+`).
56
56
  - `status` on `area` and `note`: `current`, `partial`, `stale`, `superseded`. On `index`, `run` and `plan` additionally `planned`.
57
- - `current`: checked against the current sources. This is not a runtime test or a deployment claim; say in prose which version and environment you checked.
58
- - `partial`: sampled; some of the area or behavior was not read.
57
+ - `current`: the whole feature was read and the page checked against the current sources. This is not a runtime test or a deployment claim; say in prose which version and environment you checked.
58
+ - `partial`: some of the feature or area was not read. The page ends with a line saying what is left, and stays `partial` until the feature is finished.
59
59
  - `stale`: its sources changed or disappeared and it has not been re-read.
60
60
  - `superseded`: replaced by another page, kept as a pointer.
61
61
  - `planned` (index, run, plan): describes intent not yet carried out.
@@ -63,6 +63,7 @@ Required on every page: `id`, `title`, `type`, `areas`, `sources`, `updated`. `s
63
63
  - `areas`: existing **area IDs**. Each `note`, `gap` and `plan` needs at least one. Area pages may list themselves or leave empty; index and run pages may be empty.
64
64
  - `sources`: existing files read as evidence, repository-relative; no directories, credentials, absolute paths or paths escaping the repo. A `current` note needs evidence; gaps, plans and maps may have none.
65
65
  - `updated`: last substantive review date (`YYYY-MM-DD`).
66
+ - Optional `parent`, on a `note` only: the `id` of the feature note this note is a part of. The vault has three levels: area, feature, part. The parent is a note of the same first area that has no `parent` itself, and its body links to each of its parts; `check` rejects anything else. The viewer lists a part under its feature.
66
67
  - Optional `tags`, `aliases`, `task_ids`: string arrays. Optional `summary` and `verification`: strings. An area's `summary` introduces it in the viewer; a note's `summary` is its plain-language statement of what the product does, shown first in the viewer's Read view (write one for every note; `check` does not require it, so older vaults still open).
67
68
 
68
69
  Gap-only properties are errors on any other page type, `gap` is an error on a page that is not a plan, and `priority` and `confidence` are allowed only on gap and plan pages. String properties (and strings inside arrays) may not contain control characters, including line separators. Gap-only properties (`kind`, `recheck`, `rechecked_by`, `rechecked_on`, `decision`, `decided_by`, `decided_on`, `duplicate_of`, optional `priority`, `confidence`) and the plan-only `gap` property are defined in [gaps.md](gaps.md), with the rules for who may set each. Gap state (`needs-recheck`, `needs-decision`, or the decision value) is derived from them and never written into Markdown. A recheck has one meaning for every gap kind: does the `Current behavior` section still describe the code? `confirmed` means it does (confirmed, still holds; for a proposal, the capability is still absent) and `not-reproduced` means it no longer does; neither judges whether a proposal is worth doing, which is a person's decision (see [what a recheck means](gaps.md#what-a-recheck-means)). `recheck: "outdated"` means the cited sources changed after the last recorded recheck; `rechecked_by` and `rechecked_on` stay, the `Recheck` section must have text, and an open gap with it is `needs-recheck`.
@@ -81,8 +82,10 @@ A **feature note** is the one page for one screen, feature, flow or rule, and al
81
82
  | Flow | `## How it works` | What a person does and what happens, in order. Left out when there is no flow. |
82
83
  | Fields | `## Fields` | One row per field of the feature's form: `Field`, `Required`, `Rule`, `Message when refused`. |
83
84
  | Rules | `## Business rules` | One numbered rule per row: `ID`, `Rule`. Who may do it, when an action is available, limits, defaults, what is kept, sent or deleted, and when. |
85
+ | Clients | `## Where the apps differ` | Only when the product has several clients: one row per difference, `What` and a column per client (`Web`, `Mobile`). One sentence instead when they were compared and do not differ. |
86
+ | Parts | `## Parts` | Only on a feature that has parts: a link to each part and one line saying what it covers. |
84
87
 
85
- The headings are written exactly so and come in this order; a section the feature does not have is left out. A field's rule is in `Fields` and is not repeated in `Business rules`. Usually 150-400 words; when its rules need more, split the note by feature or flow, never by kind of fact, so that everything about one feature stays on one page.
88
+ The headings are written exactly so and come in this order; a section the feature does not have is left out. A field's rule is in `Fields` and is not repeated in `Business rules`. Usually 150-400 words and up to about twenty rules. A feature that needs more gets parts: notes with `parent` set to the feature note, each about one flow or one kind of use, never one kind of fact, and each written in this same shape. The feature note then holds what the feature is for, how the parts fit, the rules that hold in every part, and `## Parts`.
86
89
 
87
90
  A rule is numbered `BR-<FEATURE>-<NN>` (`BR-INVITE-03`): `BR`, a short upper-case word for the feature that is the same in all rules of the note and used by no other note, and a number from 01. The number is the rule's name, quoted in tickets, tests and gaps. A rule keeps it when its text changes; a new rule takes the next free number; the number of a removed rule is not used again; rules are never renumbered. One rule per row, one or two sentences, with the value.
88
91
 
@@ -164,9 +167,11 @@ The validator enforces these body rules:
164
167
 
165
168
  ## Helper
166
169
 
167
- Three scripts in `$SKILL_DIR/scripts/`; see [SKILL.md](../SKILL.md) for the variables.
170
+ Four scripts in `$SKILL_DIR/scripts/`; see [SKILL.md](../SKILL.md) for the variables.
168
171
 
169
- `knowledge.py {check,snapshot,serve} --repo PATH [--vault PATH]`
172
+ `coverage.py --repo PATH [--vault PATH] [--path FOLDER]... [--all]` reads the repository's code files and the spec pages (`area` and `note`) and prints what the code has that no page mentions. It writes nothing and always exits 0. Code files are the tracked and the untracked-but-not-ignored files with a source-code extension, outside the vault, without tests, fixtures, stories, mocks, migrations, dependencies and build output. For each folder it prints the number of files and lines, how many of the files a page cites in `sources`, how many **sentences a person can see** the folder has and how many are on no page, and the same for **named numbers**. A sentence is a string or text between tags of 12 to 200 characters that starts with a capital letter and has at least three words, outside log calls; it is on a page when a page contains it, ignoring case, curly quotes and spacing. A named number is an upper-case constant set to a number (`MAX_FILES = 10`); it is on a page when a page contains its name. A file with 40 or more sentences is an article or a template and counts as a file only. A folder in which no page cites any file is one line however deep it goes; folders are ordered by how much they have that no page mentions. With `--path` only those folders are read and every unmentioned sentence and number is listed with its file, then the largest files no page cites. The first 40 folders and 60 items are printed unless `--all` is given. The report finds places to read; it cannot find a rule that has neither a message nor a named number.
173
+
174
+ `knowledge.py {check,snapshot,serve,build} --repo PATH [--vault PATH]`
170
175
 
171
176
  - `check`: read-only, exit 1 for structural errors; freshness warnings normally leave exit status 0. Add `--strict` to also fail on warnings. Page-level errors are printed first, and a failing `check` or `snapshot` ends with a line giving the number of errors and the number of pages they are on. A page whose frontmatter fails validation still counts as a target for links, embeds, area membership, a plan's `gap` and `duplicate_of`, so one broken page yields its own errors only; fix it and nothing else needs touching. Only the "missing" error is hidden this way: an ambiguous link (two or more candidates), a link to a page of the wrong type (a spec page linking a gap, or an `areas` entry naming a page known to be another type) and a run page's link are still reported. `snapshot --notes X` refuses X while X's area references, links, `gap` or `duplicate_of` point at a page that failed validation. Missing source files are errors except on explicitly `stale` or `superseded` notes (warnings). Changed or unbaselined evidence warns. A source that differs from its baseline only by line endings (typically a Windows checkout) warns "line endings differ in source X"; do not snapshot on that checkout, because it would record the converted bytes. Nothing is rewritten.
172
177
 
@@ -204,6 +209,8 @@ Canonical Markdown must not contain raw merge conflicts. Validation rejects comp
204
209
 
205
210
  `serve` shows the vault in a browser: `knowledge.py serve --repo REPO [--vault VAULT] [--port N] [--open]`. Without `--vault`, this and every other command use the single vault under `knowledge/` (`knowledge/00-index.md` or `knowledge/*/00-index.md`) and stop with a message when there is none or several. It listens on `127.0.0.1` only (port 8757, or a free one when that is taken), answers only `/` and only when addressed as this machine, never writes, and builds the page again from disk on every request, so a reload shows the current files. When the vault has errors the page lists them. The page data is the graph above plus `live: true`, `vaultPath` (the vault's path relative to the repository), `history`, `pending` and `files`. `history` is the last 30 commits that touched vault pages (not `.knowledge/`), newest first, each `{"sha", "date", "subject", "pages", "more"}`; a page entry is `{"path", "status"}` with `status` `added`, `changed` or `removed`, plus `was` and `now` when the commit changed the page's `summary` (or, failing that, its `title`); at most 80 pages are listed per commit and `more` counts the rest. `pending` lists vault pages with uncommitted changes in the same `{"path", "status"}` shape. Both are empty outside a Git work tree. `files` is the text of the repository files that pages cite in `sources` or link to, keyed by repository-relative path, each `{"text", "truncated"}`. A file is included only when it is inside the repository and outside the vault, is not a credential path, and is UTF-8 text; one file is cut at 300,000 characters and all files together at 4,000,000, after which the rest stay ordinary links. The viewer opens a link to an included file in place: Markdown rendered, anything else as numbered lines, with the lines of a `#L10` or `#L10-L20` fragment marked.
206
211
 
212
+ `build` writes the same viewer as one file, for a server that shows the spec to people who do not have the repository: `knowledge.py build --repo REPO [--vault VAULT] --out FILE [--source-url URL] [--with-sources]`. `--out` must be a file outside the vault. The command stops with the `check` errors when the vault has any, and writes nothing then; freshness warnings do not stop it. The page data is the graph plus `built: true`, `commit` (the full `HEAD`, empty outside Git), `sourceUrl`, `vaultPath`, `history`, `pending` and `files`. `files` is empty unless `--with-sources` is given, because the text of cited files is source code. `--source-url` must be an `http(s)` address; `{commit}` in it is replaced by `HEAD`, and it is an error to name `{commit}` outside a Git work tree. `--allow-missing-sources` is for a build that sees only part of the repository (a Docker build context): a cited path where nothing exists is not an error and does not mark its page as changed, and the command lists those files; `check` and `snapshot` never accept it. On a built page a cited file whose text is not included is a link to `sourceUrl/<repository path>` (with `#L10-L20` when lines are named) in a new tab, or plain text when there is no `sourceUrl`; it is never a relative link, which would lead into the site that hosts the page. In a shallow clone the commit whose parents were not fetched is left out of `history` (by `serve` too), since Git would show every page as added there, and `build` says that the history is short.
213
+
207
214
  `assets/viewer.html` has exactly one `__KNOWLEDGE_DATA__` placeholder for script-safe JSON. Treat contents as untrusted text: no raw Markdown HTML execution, unsafe URL schemes, external scripts, fonts, analytics or fetches. Render readable basic Markdown, pipe tables and wikilinks; leave unsupported syntax as text. The viewer has a tree of areas with their notes (a note is listed under its first area; superseded notes are not listed), a search over pages, gaps and the cited files, and five places: the overview, a page, the gap queue, the map and recent changes. A note shows its title, `summary`, the page text and the gaps about it that are not closed, then its source files folded; an area shows its notes with their summaries and folds its own text. The pages linked to or from the page are a short list beside it. Navigation keeps a trail for Back and Forward and writes the place into the address (`#/page/notes/example`, `#/gaps`), so a reload or a copied link returns to it. The map draws one frame per primary area (the area page, or the first entry of `areas`) with its pages inside; an edge between a page and its primary area is not drawn because the frame says it, links inside one area are solid and links across areas dashed; `nav` pages are not drawn. Recent changes reads `history`, `pending` and the "changed source" lines of `warnings`. "Copy for AI" only composes text from the page, a choice of what the agent should do and the reader's own words, and copies it to the clipboard. The reader's name, the recently viewed pages and the last commit seen are kept in the browser's `localStorage`, and the viewer works without it.
208
215
 
209
216
  ## Format references
@@ -27,7 +27,7 @@ A gap must not have `status`; its progress is carried by `recheck` and `decision
27
27
  | `decided_by` | when `decision` is `accepted`, `rejected` or `deferred` | a person's name; the value `agent` is an error |
28
28
  | `decided_on` | when `decision` is `accepted`, `rejected` or `deferred` | `YYYY-MM-DD` |
29
29
  | `duplicate_of` | when `decision` is `duplicate` | id of another existing gap (not itself) |
30
- | `priority`, `confidence` | optional, gap and plan pages only | short strings, e.g. `"high"` |
30
+ | `priority`, `confidence` | optional, gap and plan pages only | short strings, e.g. `"high"`. Use `high`, `medium` or `low` for `priority` (or `P0` to `P4`): the viewer lists the most urgent gap of each state first and can show one priority only. |
31
31
 
32
32
  `decided_by` and `decided_on` are also allowed on `done` and `duplicate` but not required. With `outdated`, `rechecked_by` and `rechecked_on` stay in place as the record of the last recheck actually done. A fresh recheck replaces all three fields and adds a new dated entry to the `Recheck` section; keep the older entries.
33
33
 
@@ -0,0 +1,200 @@
1
+ #!/usr/bin/env python3
2
+ """List what the code has that no spec page mentions yet, using only the standard library.
3
+
4
+ It reads the application's code files and the spec pages and compares three things a spec is supposed to hold:
5
+ the files pages cite, the sentences a person can see (messages, labels, empty states) and the numeric limits.
6
+ What it prints is a list of places to read, not a score: a sentence that is not on a page may be a button that
7
+ only opens something, which the spec leaves out on purpose.
8
+ """
9
+ import argparse
10
+ import os
11
+ import posixpath
12
+ import re
13
+ import sys
14
+ from pathlib import Path
15
+
16
+ import knowledge
17
+
18
+ CODE = {'.ts', '.tsx', '.js', '.jsx', '.mjs', '.cjs', '.vue', '.svelte', '.py', '.rb', '.go', '.rs', '.java', '.kt', '.kts',
19
+ '.swift', '.cs', '.php', '.dart', '.ex', '.exs', '.scala', '.sql', '.c', '.cc', '.cpp', '.h', '.m', '.mm'}
20
+ SKIP_DIRS = {'node_modules', '.git', 'dist', 'build', 'out', 'vendor', 'target', '.next', '.nuxt', '.expo', 'coverage', '__pycache__',
21
+ '.venv', 'venv', 'Pods', '.gradle', 'bin', 'obj', '.turbo', '.cache', 'generated', '__generated__',
22
+ # A migration is the history of the schema, not what the product does today.
23
+ 'migrations', 'Migrations', 'migration'}
24
+ # A file with this many sentences is text someone wrote for readers (an article, a template, legal terms): the spec
25
+ # lists such things, it does not quote them, so they count as files and lines only.
26
+ CONTENT_SENTENCES = 40
27
+ # Tests, fixtures and stories say what the code does in someone else's words; they are not what a person sees.
28
+ TEST_DIR = re.compile(r'(^|/)(tests?|__tests__|__mocks__|__snapshots__|fixtures?|testdata|e2e|stories|specs?|mocks?|seeds?)(/|$)', re.I)
29
+ TEST_FILE = re.compile(r'(\.(test|spec|stories|e2e|mock|fixture)\.|_test\.|(^|/)test_[^/]*$|\.d\.ts$|\.min\.|\.generated\.|\.g\.dart$)', re.I)
30
+ FILE_LIMIT = 1_000_000
31
+ QUOTED = re.compile(r'''(["'`])((?:\\.|(?!\1)[^\\\n])*)\1''')
32
+ JSX_TEXT = re.compile(r'>\s*([A-Z][^<>{}\n]{10,200}?)\s*<')
33
+ LOG_LINE = re.compile(r'\b(console|logger|log|logging|Sentry|debug)\s*\.\s*\w+\s*\(|^\s*(import|from|describe|it|test|#include|using|package)\b|^\s*(//|#|\*|/\*)')
34
+ NOT_PROSE = re.compile(r'[{}<>;=\\]|\$\(|^\s*(SELECT|INSERT|UPDATE|DELETE|CREATE|ALTER|DROP|WITH)\b|https?://|\w+\.\w+\(|\w/\w+/\w')
35
+ # A named number: MAX_ATTACHMENTS = 10, TTL_MS: number = 60 * 60 * 1000
36
+ LIMIT = re.compile(r'\b([A-Z][A-Z0-9]*(?:_[A-Z0-9]+)+)\s*(?::[^=\n]{1,40})?=\s*([-+]?\d[\d_.]*(?:\s*[*/+-]\s*\d[\d_.]*)*)\s*[;,]?\s*(?:(?://|#).*)?$', re.M)
37
+ PUNCTUATION = str.maketrans({'‘': "'", '’': "'", '“': '"', '”': '"', '–': '-', '—': '-', ' ': ' '})
38
+
39
+
40
+ def plain(text):
41
+ """Text as it is compared: case, curly quotes, runs of spaces and Markdown's escaped pipe do not count."""
42
+ return re.sub(r'\s+', ' ', text.translate(PUNCTUATION).replace('\\|', '|')).casefold().strip()
43
+
44
+
45
+ def is_sentence(text):
46
+ """Does this string read like something shown to a person, rather than a key, a class list or a query?"""
47
+ text = text.strip()
48
+ if not 12 <= len(text) <= 200 or not text[0].isupper() or len(text.split()) < 3 or NOT_PROSE.search(text):
49
+ return False
50
+ letters = sum(1 for c in text if c.isalpha() or c == ' ')
51
+ return letters / len(text) >= 0.8
52
+
53
+
54
+ def sentences_in(text):
55
+ found = []
56
+ for line in text.split('\n'):
57
+ if LOG_LINE.search(line):
58
+ continue
59
+ for _, inner in QUOTED.findall(line):
60
+ inner = re.sub(r"\\(['\"`])", r'\1', inner)
61
+ if is_sentence(inner):
62
+ found.append(inner.strip())
63
+ found += [m.strip() for m in JSX_TEXT.findall(line) if is_sentence(m)]
64
+ return list(dict.fromkeys(found))
65
+
66
+
67
+ def code_files(repo, vault):
68
+ """Repository-relative paths of the application's own code: tracked or untracked but not ignored, no tests."""
69
+ listed = knowledge.run_git(repo, ['ls-files', '-z', '--cached', '--others', '--exclude-standard'])
70
+ if listed.returncode == 0:
71
+ names = [os.fsdecode(n) for n in listed.stdout.split(b'\0') if n]
72
+ else:
73
+ names = []
74
+ for folder, dirs, files in os.walk(repo):
75
+ dirs[:] = sorted(d for d in dirs if d not in SKIP_DIRS and not d.startswith('.'))
76
+ names += [Path(os.path.relpath(Path(folder) / f, repo)).as_posix() for f in sorted(files)]
77
+ inside = Path(os.path.relpath(vault, repo)).as_posix() + '/'
78
+ return sorted(n for n in names
79
+ if posixpath.splitext(n)[1].lower() in CODE and not n.startswith(inside)
80
+ and not any(part in SKIP_DIRS for part in n.split('/')[:-1])
81
+ and not TEST_DIR.search(posixpath.dirname(n)) and not TEST_FILE.search(n))
82
+
83
+
84
+ def measure(repo, vault, notes, prefixes=()):
85
+ """Per code file: lines, whether a page cites it, and the sentences and limits no page mentions."""
86
+ spec = [n for n in notes if n['meta']['type'] in {'area', 'note'}]
87
+ pages = plain('\n'.join(f"{n['meta']['title']}\n{n['meta'].get('summary', '')}\n{n['body']}" for n in spec))
88
+ words = set(re.findall(r'[A-Z][A-Z0-9_]+', '\n'.join(n['body'] for n in spec)))
89
+ cited = {posixpath.normpath(s) for n in spec for s in n['meta']['sources']}
90
+ files = {}
91
+ for name in code_files(repo, vault):
92
+ if prefixes and not any(name == p or name.startswith(p.rstrip('/') + '/') for p in prefixes):
93
+ continue
94
+ path = repo / name
95
+ try:
96
+ if path.is_symlink() or path.stat().st_size > FILE_LIMIT:
97
+ continue
98
+ text = path.read_bytes().decode('utf-8')
99
+ except (OSError, UnicodeDecodeError):
100
+ continue
101
+ sentences = sentences_in(text)
102
+ content = len(sentences) >= CONTENT_SENTENCES
103
+ if content:
104
+ sentences = []
105
+ limits = list(dict.fromkeys(LIMIT.findall(text)))
106
+ files[name] = {
107
+ 'lines': text.count('\n') + 1, 'cited': name in cited, 'content': content, 'sentences': len(sentences), 'limits': len(limits),
108
+ 'unquoted': [s for s in sentences if plain(s).rstrip('.!?:…') not in pages],
109
+ 'unnamed': [(n, v.strip()) for n, v in limits if n not in words]}
110
+ return files
111
+
112
+
113
+ def folders(files, prefixes=()):
114
+ """Group files into the places to read: a folder no page cites anything in is one line, however deep it goes.
115
+
116
+ With prefixes (the folders the caller asked about) a line never names a folder above the one asked for."""
117
+ cited_under = {}
118
+ for name, info in files.items():
119
+ if info['cited']:
120
+ parts = name.split('/')[:-1]
121
+ for i in range(len(parts) + 1):
122
+ cited_under['/'.join(parts[:i])] = True
123
+ groups = {}
124
+ for name, info in files.items():
125
+ parts = name.split('/')[:-1]
126
+ folder = '/'.join(parts)
127
+ floor = max((len(p.split('/')) for p in prefixes if name.startswith(p.rstrip('/') + '/')), default=1)
128
+ # Climb to the highest folder that still has no cited file under it.
129
+ for i in range(min(floor, len(parts)) or 1, len(parts) + 1):
130
+ if not cited_under.get('/'.join(parts[:i])):
131
+ folder = '/'.join(parts[:i])
132
+ break
133
+ g = groups.setdefault(folder or '.', {'files': 0, 'lines': 0, 'cited': 0, 'sentences': 0, 'unquoted': 0, 'limits': 0, 'unnamed': 0})
134
+ g['files'] += 1
135
+ g['lines'] += info['lines']
136
+ g['cited'] += info['cited']
137
+ g['sentences'] += info['sentences']
138
+ g['unquoted'] += len(info['unquoted'])
139
+ g['limits'] += info['limits']
140
+ g['unnamed'] += len(info['unnamed'])
141
+ return groups
142
+
143
+
144
+ def main(argv=None):
145
+ parser = argparse.ArgumentParser(description=__doc__.split('\n')[0])
146
+ parser.add_argument('--repo', required=True, type=Path)
147
+ parser.add_argument('--vault', type=Path, help='default: the single vault under knowledge/ in the repository')
148
+ parser.add_argument('--path', action='append', default=[], metavar='FOLDER',
149
+ help='only this folder or file of the repository, and list each sentence and limit no page mentions; repeatable')
150
+ parser.add_argument('--all', action='store_true', help='print every line instead of the first 40 folders or 60 items')
151
+ args = parser.parse_args(argv)
152
+ repo = args.repo.resolve()
153
+ if not repo.is_dir():
154
+ parser.error('repo must be an existing directory')
155
+ try:
156
+ vault = args.vault.resolve() if args.vault else knowledge.find_vault(repo)
157
+ except ValueError as exc:
158
+ parser.error(str(exc))
159
+ if not vault.is_dir():
160
+ parser.error('repo and vault must be existing directories')
161
+ notes, _, errors, _, _ = knowledge.load_vault(repo, vault, freshness=False, absent=[])
162
+ if errors:
163
+ print(f'coverage: the vault has {len(errors)} errors; pages with errors are not counted (run check)', file=sys.stderr)
164
+ prefixes = [posixpath.normpath(p.replace(os.sep, '/')) for p in args.path]
165
+ files = measure(repo, vault, notes, prefixes)
166
+ if not files:
167
+ print('coverage: no code files found' + (' under ' + ', '.join(prefixes) if prefixes else ''))
168
+ return 0
169
+ total = lambda key: sum(len(f[key]) if isinstance(f[key], list) else f[key] for f in files.values())
170
+ print(f"coverage: {len(files)} code files, {total('lines')} lines; pages cite {total('cited')} of them")
171
+ print(f"coverage: {total('sentences')} sentences a person can see, {total('unquoted')} on no page; "
172
+ f"{total('limits')} named numbers, {total('unnamed')} on no page")
173
+ texts = sum(1 for f in files.values() if f['content'])
174
+ print('coverage: this is a list of places to read, not a score; tests, fixtures, migrations and generated code are left out'
175
+ + (f'; {texts} files that are mostly text (articles, templates) count as files only' if texts else ''))
176
+ groups = sorted(folders(files, prefixes).items(), key=lambda kv: (-(kv[1]['unquoted'] + kv[1]['unnamed']), -kv[1]['lines'], kv[0]))
177
+ shown = groups if args.all else groups[:40]
178
+ width = max(len(name) for name, _ in shown)
179
+ print(f"\n{'folder'.ljust(width)} files lines cited sentences not on a page numbers not on a page")
180
+ for name, g in shown:
181
+ print(f"{name.ljust(width)} {g['files']:5} {g['lines']:6} {g['cited']:5} {g['unquoted']:9} of {g['sentences']:<9} {g['unnamed']:7} of {g['limits']}")
182
+ if len(groups) > len(shown):
183
+ print(f'... and {len(groups) - len(shown)} more folders (--all prints them)')
184
+ if prefixes:
185
+ items = [(name, 'says', s) for name, f in files.items() for s in f['unquoted']]
186
+ items += [(name, 'sets', f'{n} = {v}') for name, f in files.items() for n, v in f['unnamed']]
187
+ print(f'\n{len(items)} things under {", ".join(prefixes)} that no page mentions:')
188
+ for name, verb, what in (items if args.all else items[:60]):
189
+ print(f' {name} {verb}: {what}')
190
+ if not args.all and len(items) > 60:
191
+ print(f' ... and {len(items) - 60} more (--all prints them)')
192
+ uncited = sorted((f['lines'], name) for name, f in files.items() if not f['cited'])
193
+ print(f'\n{len(uncited)} of {len(files)} files there are cited by no page; the largest:')
194
+ for lines, name in reversed(uncited[-(len(uncited) if args.all else 15):]):
195
+ print(f' {name} ({lines} lines)')
196
+ return 0
197
+
198
+
199
+ if __name__ == '__main__':
200
+ sys.exit(main())
@@ -37,7 +37,7 @@ RULE_ROW = re.compile(r'^ {0,3}\|[ \t]*(BR-[A-Z0-9]+(?:-[A-Z0-9]+)*-\d{2,4})[ \t
37
37
  MD_REF = re.compile(r'^ {0,3}\[[^\]\n]+\]:\s*(<[^>]*>|\S+)', re.M)
38
38
  GAP_PAGE_ONLY = ('priority', 'confidence')
39
39
  ARRAYS = {'areas', 'sources', 'tags', 'aliases', 'task_ids'}
40
- STRINGS = {'id', 'title', 'type', 'status', 'updated', 'summary', 'verification', 'priority', 'confidence', 'gap',
40
+ STRINGS = {'id', 'title', 'type', 'status', 'updated', 'summary', 'verification', 'priority', 'confidence', 'gap', 'parent',
41
41
  *GAP_ONLY}
42
42
  REQUIRED = {'id', 'title', 'type', 'areas', 'sources', 'updated'}
43
43
  LINK = re.compile(r'(!?)\[\[([^\[\]\n]+)\]\]') # no brackets inside, so a run of '[[' cannot backtrack
@@ -215,6 +215,8 @@ def validate_meta(meta):
215
215
  raise ValueError(f"status must be one of {', '.join(sorted(allowed))}{hint}")
216
216
  if kind == 'note' and meta['status'] == 'current' and not meta['sources']:
217
217
  raise ValueError('current note requires sources')
218
+ if 'parent' in meta and (kind != 'note' or not meta['parent'].strip()):
219
+ raise ValueError('parent is only valid on note pages and names the feature note this one is a part of')
218
220
  if kind == 'plan':
219
221
  if not meta.get('gap', '').strip():
220
222
  raise ValueError('plan requires gap: the id of the accepted gap it implements')
@@ -313,20 +315,29 @@ def line_ending_variant(data, old):
313
315
  return old in {hashlib.sha256(lf).hexdigest(), hashlib.sha256(lf.replace(b'\n', b'\r\n')).hexdigest()}
314
316
 
315
317
 
318
+ def run_git(repo, args, **options):
319
+ """Git's answer for the repository. Where Git is not installed (a build container, say) it is the answer
320
+ of a folder that is not a work tree, so the caller goes on without history instead of stopping."""
321
+ try:
322
+ return subprocess.run(['git', '-C', str(repo), *args], capture_output=True, **options)
323
+ except FileNotFoundError:
324
+ return subprocess.CompletedProcess(args, 127, b'', b'')
325
+
326
+
316
327
  def head_blob(repo, source):
317
328
  """Bytes of the source as committed at HEAD, or None when HEAD has no such file (untracked, new or no commits)."""
318
- result = subprocess.run(['git', '-C', str(repo), 'cat-file', 'blob', f'HEAD:./{posixpath.normpath(source)}'], capture_output=True)
329
+ result = run_git(repo, ['cat-file', 'blob', f'HEAD:./{posixpath.normpath(source)}'])
319
330
  return None if result.returncode else result.stdout
320
331
 
321
332
 
322
333
  def uncommitted_paths(repo):
323
334
  """Repo-relative paths Git reports as modified, staged or untracked (one status call), or None outside a work tree."""
324
- probe = subprocess.run(['git', '-C', str(repo), 'rev-parse', '--is-inside-work-tree', '--show-prefix'], capture_output=True)
335
+ probe = run_git(repo, ['rev-parse', '--is-inside-work-tree', '--show-prefix'])
325
336
  lines = probe.stdout.decode(errors='replace').split('\n')
326
337
  if probe.returncode or lines[0] != 'true':
327
338
  return None
328
339
  prefix = lines[1] if len(lines) > 1 else ''
329
- status = subprocess.run(['git', '-C', str(repo), 'status', '--porcelain=v1', '-z', '--untracked-files=all'], capture_output=True)
340
+ status = run_git(repo, ['status', '--porcelain=v1', '-z', '--untracked-files=all'])
330
341
  if status.returncode:
331
342
  return None
332
343
  raw, found, i = status.stdout.split(b'\0'), set(), 0
@@ -347,10 +358,13 @@ def error_page(message):
347
358
  return head if head.endswith('.md') else None
348
359
 
349
360
 
350
- def load_vault(repo, vault, freshness=True, dirty=None, report=None):
361
+ def load_vault(repo, vault, freshness=True, dirty=None, report=None, absent=None):
351
362
  """dirty: repo-relative paths with uncommitted changes. Their freshness warning is marked, not counted, only when
352
363
  the accepted fingerprint equals the file's HEAD blob (up to LF/CRLF), so the sole drift is uncommitted.
353
364
 
365
+ absent, when given, collects cited files that are not there instead of reporting them: for a build that sees
366
+ only part of the repository, such as a Docker build context. Their pages are not marked as changed.
367
+
354
368
  report, when given, receives 'failed': {page path: parsed id or None} for pages that failed validation.
355
369
  """
356
370
  errors, warnings, notes, failed, failed_kind, cascade = [], [], [], {}, {}, []
@@ -415,10 +429,26 @@ def load_vault(repo, vault, freshness=True, dirty=None, report=None):
415
429
  paths[stem] = note
416
430
  basenames.setdefault(PurePosixPath(stem).name, []).append(note)
417
431
  area_ids = {n['meta']['id'] for n in notes if n['meta']['type'] == 'area'}
418
- edges, attachments, rule_pages = set(), {}, {}
432
+ edges, attachments, rule_pages, linked, parts = set(), {}, {}, set(), []
419
433
  for note in notes:
420
434
  meta, rel = note['meta'], note['path']
421
435
  note['stale'] = False
436
+ if meta.get('parent'):
437
+ # Three levels and no more: an area, its features, and the parts of a feature that is too big for one page.
438
+ other, name = ids.get(meta['parent']), meta['parent']
439
+ if other is None and name in failed_ids and failed_id_kinds[name] & {None, 'note'}:
440
+ cascade.append(f'{rel}: parent {name} names a page that failed validation')
441
+ elif other is None or other['meta']['type'] != 'note':
442
+ errors.append(f'{rel}: parent {name} is not the id of an existing note')
443
+ elif other is note:
444
+ errors.append(f'{rel}: parent must name a different note')
445
+ elif other['meta'].get('parent'):
446
+ errors.append(f"{rel}: parent {name} is itself a part of {other['meta']['parent']}; a part belongs to a feature, not to another part")
447
+ elif other['meta']['areas'][:1] != meta['areas'][:1]:
448
+ errors.append(f"{rel}: parent {name} is in another area; a part has the same first area as its feature")
449
+ else:
450
+ edges.add((other['meta']['id'], meta['id']))
451
+ parts.append((other, note))
422
452
  if meta['type'] == 'gap':
423
453
  note['state'] = gap_state(meta)
424
454
  if meta['type'] == 'gap' and meta.get('duplicate_of'):
@@ -495,6 +525,7 @@ def load_vault(repo, vault, freshness=True, dirty=None, report=None):
495
525
  "if that part needs its own link, make it its own note and list it here")
496
526
  else:
497
527
  edges.add((meta['id'], choices[0]['meta']['id']))
528
+ linked.add((meta['id'], choices[0]['meta']['id']))
498
529
  if meta['type'] == 'area':
499
530
  # An area page is a map of its notes. A table on it is behavior written in the wrong place.
500
531
  tables = sum(1 for line in text.split('\n') if TABLE_RULE.match(line))
@@ -541,6 +572,9 @@ def load_vault(repo, vault, freshness=True, dirty=None, report=None):
541
572
  if credential_path(path.resolve().relative_to(repo).as_posix()):
542
573
  raise ValueError('credential source is forbidden')
543
574
  if not path.is_file():
575
+ if absent is not None and not path.exists():
576
+ absent.append(source)
577
+ continue
544
578
  if closed:
545
579
  warnings.append(f'{rel}: missing source {source}')
546
580
  note['stale'] = True
@@ -564,6 +598,10 @@ def load_vault(repo, vault, freshness=True, dirty=None, report=None):
564
598
  note['stale'] = True
565
599
  except (OSError, ValueError) as exc:
566
600
  errors.append(f'{rel}: {source}: {exc}')
601
+ for feature, part in parts:
602
+ # The viewer nests a part under its feature from `parent`; a plain Markdown reader has only the link.
603
+ if (feature['meta']['id'], part['meta']['id']) not in linked:
604
+ errors.append(f"{part['path']}: its feature {feature['path']} does not link to it; list every part on the feature's page")
567
605
  if report is not None:
568
606
  report['failed'] = failed
569
607
  report['cascade'] = cascade
@@ -634,7 +672,7 @@ def node_of(note):
634
672
  'kind': m.get('kind', ''), 'recheck': m.get('recheck', ''), 'recheckedBy': m.get('rechecked_by', ''),
635
673
  'recheckedOn': m.get('rechecked_on', ''), 'decision': m.get('decision', ''), 'decidedBy': m.get('decided_by', ''),
636
674
  'decidedOn': m.get('decided_on', ''), 'priority': m.get('priority', ''), 'confidence': m.get('confidence', ''),
637
- 'duplicateOf': m.get('duplicate_of', ''), 'recheckOutdated': gap and (m['recheck'] == 'outdated' or recheck_marker(note)), 'state': note.get('state', ''), 'gap': m.get('gap', ''),
675
+ 'parent': m.get('parent', ''), 'duplicateOf': m.get('duplicate_of', ''), 'recheckOutdated': gap and (m['recheck'] == 'outdated' or recheck_marker(note)), 'state': note.get('state', ''), 'gap': m.get('gap', ''),
638
676
  'layer': LAYERS[m['type']]}
639
677
 
640
678
 
@@ -689,6 +727,17 @@ def front_value(line):
689
727
  return value if isinstance(value, str) else ''
690
728
 
691
729
 
730
+ def shallow_commits(repo):
731
+ """Commits whose parents a shallow clone left out. Empty in a full clone and outside Git."""
732
+ found = run_git(repo, ['rev-parse', '--git-path', 'shallow'])
733
+ if found.returncode:
734
+ return set()
735
+ try:
736
+ return set((repo / os.fsdecode(found.stdout).strip()).read_text(encoding='ascii').split())
737
+ except (OSError, UnicodeDecodeError):
738
+ return set()
739
+
740
+
692
741
  def vault_history(repo, vault):
693
742
  """Recent commits that touched vault pages, newest first: sha, date, subject and what happened to each page.
694
743
 
@@ -698,29 +747,31 @@ def vault_history(repo, vault):
698
747
  spec, prefix = page_pathspec(repo, vault)
699
748
  if spec is None:
700
749
  return []
701
- head = subprocess.run(['git', '-C', str(repo), 'rev-parse', 'HEAD'], capture_output=True)
750
+ head = run_git(repo, ['rev-parse', 'HEAD'])
702
751
  if head.returncode:
703
752
  return []
704
753
  key = (str(repo), str(vault), head.stdout.strip())
705
754
  if key in _history_cache:
706
755
  return _history_cache[key]
707
756
  # Git prints paths from the top of the work tree, which may be above the repository folder given here.
708
- top = subprocess.run(['git', '-C', str(repo), 'rev-parse', '--show-prefix'], capture_output=True).stdout.decode(errors='replace').strip()
757
+ top = run_git(repo, ['rev-parse', '--show-prefix']).stdout.decode(errors='replace').strip()
709
758
  prefix = top + prefix
710
759
  # A marker nobody could have written into a commit message or a page beforehand separates the commits.
711
760
  marker = '\x01' + os.urandom(12).hex() + '\x01'
712
761
  try:
713
- log = subprocess.run(['git', '-C', str(repo), '-c', 'core.quotepath=off', 'log', f'-n{HISTORY_COMMITS}', '--date=short', '--no-color',
762
+ log = run_git(repo, ['-c', 'core.quotepath=off', 'log', f'-n{HISTORY_COMMITS}', '--date=short', '--no-color',
714
763
  '--no-renames', '-p', '-U0', f'--format={marker}%H%x02%ad%x02%s', '--', *spec],
715
- capture_output=True, timeout=20)
764
+ timeout=20)
716
765
  except subprocess.TimeoutExpired:
717
766
  return []
718
767
  if log.returncode:
719
768
  return []
720
- commits = []
769
+ commits, boundary = [], shallow_commits(repo)
721
770
  for chunk in log.stdout.decode('utf-8', errors='replace').split(marker)[1:]:
722
771
  header, _, body = chunk.partition('\n')
723
772
  sha, date, subject = (header.split('\x02', 2) + ['', ''])[:3]
773
+ if sha in boundary: # its parents were not fetched, so Git would show every page as added
774
+ continue
724
775
  subject = CONTROL.sub('', subject)
725
776
  pages, page = [], None
726
777
  for line in body.split('\n'):
@@ -755,11 +806,10 @@ def vault_pending(repo, vault):
755
806
  spec, prefix = page_pathspec(repo, vault)
756
807
  if spec is None:
757
808
  return []
758
- status = subprocess.run(['git', '-C', str(repo), '-c', 'core.quotepath=off', 'status', '--porcelain=v1', '-z', '--untracked-files=all', '--no-renames', '--', *spec],
759
- capture_output=True)
809
+ status = run_git(repo, ['-c', 'core.quotepath=off', 'status', '--porcelain=v1', '-z', '--untracked-files=all', '--no-renames', '--', *spec])
760
810
  if status.returncode:
761
811
  return []
762
- top = subprocess.run(['git', '-C', str(repo), 'rev-parse', '--show-prefix'], capture_output=True).stdout.decode(errors='replace').strip()
812
+ top = run_git(repo, ['rev-parse', '--show-prefix']).stdout.decode(errors='replace').strip()
763
813
  pending = []
764
814
  for item in status.stdout.split(b'\0'):
765
815
  text = os.fsdecode(item)
@@ -772,6 +822,13 @@ def vault_pending(repo, vault):
772
822
  return sorted(pending, key=lambda p: p['path'])
773
823
 
774
824
 
825
+ def page_payload(repo, vault, notes, edges, warnings, sources=True):
826
+ files = cited_files(repo, vault, notes)[0] if sources else {}
827
+ return graph_of(notes, edges, warnings) | {
828
+ 'vaultPath': Path(os.path.relpath(vault, repo)).as_posix(), 'files': files,
829
+ 'history': vault_history(repo, vault), 'pending': vault_pending(repo, vault)}
830
+
831
+
775
832
  def live_page(repo, vault):
776
833
  """The viewer for the vault as it is on disk right now, or a page listing what stops it from loading."""
777
834
  try:
@@ -785,10 +842,29 @@ def live_page(repo, vault):
785
842
  f'<h1>The spec has {len(errors)} error{"s" if len(errors) != 1 else ""}</h1>'
786
843
  '<p>Fix the pages below (the <code>check</code> command prints the same list), then reload.</p>'
787
844
  f'<ul>{items}</ul></body>')
788
- files, _ = cited_files(repo, vault, notes)
789
- return viewer_html(graph_of(notes, edges, warnings) | {
790
- 'live': True, 'vaultPath': Path(os.path.relpath(vault, repo)).as_posix(), 'files': files,
791
- 'history': vault_history(repo, vault), 'pending': vault_pending(repo, vault)})
845
+ return viewer_html(page_payload(repo, vault, notes, edges, warnings) | {'live': True})
846
+
847
+
848
+ def source_base(value, commit):
849
+ """The address `--source-url` names, with `{commit}` filled in. Raises ValueError with the reason when it cannot be used."""
850
+ if CONTROL.search(value) or ' ' in value or not re.match(r'https?://[^/]', value, re.I):
851
+ raise ValueError('--source-url must be an http(s) address, e.g. https://github.com/OWNER/REPO/blob/{commit}')
852
+ if '{commit}' in value and not commit:
853
+ raise ValueError('--source-url names {commit}, but --repo is not a Git work tree with a commit')
854
+ return value.replace('{commit}', commit).rstrip('/')
855
+
856
+
857
+ def built_page(repo, vault, notes, edges, warnings, with_sources=False, source_url=''):
858
+ """The viewer as one file to put on a server: the vault as it is now, with nothing read later.
859
+
860
+ The text of cited repository files is left out unless asked for, because the file may be shown to people
861
+ who cannot read the repository. Returns (html, payload).
862
+ """
863
+ head = run_git(repo, ['rev-parse', 'HEAD'])
864
+ commit = '' if head.returncode else head.stdout.decode('ascii', errors='replace').strip()
865
+ payload = page_payload(repo, vault, notes, edges, warnings, with_sources) | {
866
+ 'built': True, 'commit': commit, 'sourceUrl': source_base(source_url, commit) if source_url else ''}
867
+ return viewer_html(payload), payload
792
868
 
793
869
 
794
870
  def make_server(repo, vault, port):
@@ -820,12 +896,17 @@ def make_server(repo, vault, port):
820
896
 
821
897
  def main(argv=None):
822
898
  parser = argparse.ArgumentParser()
823
- parser.add_argument('command', choices=['check', 'snapshot', 'serve'])
899
+ parser.add_argument('command', choices=['check', 'snapshot', 'serve', 'build'])
824
900
  parser.add_argument('--repo', required=True, type=Path)
825
901
  parser.add_argument('--vault', type=Path, help='default: the single vault under knowledge/ in the repository')
826
902
  parser.add_argument('--strict', action='store_true', help='fail check when freshness warnings exist')
827
903
  parser.add_argument('--port', type=int, default=8757, help='serve: port on 127.0.0.1 (another free port is used if it is taken)')
828
904
  parser.add_argument('--open', action='store_true', help='serve: open the page in the default browser')
905
+ parser.add_argument('--out', type=Path, help='build: the HTML file to write, outside the vault')
906
+ parser.add_argument('--with-sources', action='store_true', help='build: include the text of the repository files the pages cite')
907
+ parser.add_argument('--source-url', default='', help='build: where a cited file can be read, e.g. https://github.com/OWNER/REPO/blob/{commit}')
908
+ parser.add_argument('--allow-missing-sources', action='store_true',
909
+ help='build: do not stop when a cited file is not there, for a build that sees only part of the repository')
829
910
  parser.add_argument('--ignore-dirty', action='store_true',
830
911
  help='check: do not count freshness warnings for sources with uncommitted changes (needs a Git work tree)')
831
912
  group = parser.add_mutually_exclusive_group()
@@ -838,6 +919,10 @@ def main(argv=None):
838
919
  parser.error('--ignore-dirty is only for check')
839
920
  if args.open and args.command != 'serve':
840
921
  parser.error('--open is only for serve')
922
+ if args.command == 'build' and not args.out:
923
+ parser.error('build requires --out FILE')
924
+ if args.command != 'build' and (args.out or args.with_sources or args.source_url or args.allow_missing_sources):
925
+ parser.error('--out, --with-sources, --source-url and --allow-missing-sources are only for build')
841
926
  if args.command == 'snapshot' and not (args.all or args.notes):
842
927
  parser.error('snapshot requires --all or --notes')
843
928
  if args.command != 'snapshot' and (args.all or args.notes):
@@ -871,7 +956,8 @@ def main(argv=None):
871
956
  if dirty is None:
872
957
  parser.error('--ignore-dirty needs --repo to be a Git work tree')
873
958
  report = {}
874
- notes, edges, errors, warnings, baseline = load_vault(repo, vault, args.command != 'snapshot', dirty, report)
959
+ absent = [] if args.allow_missing_sources else None
960
+ notes, edges, errors, warnings, baseline = load_vault(repo, vault, args.command != 'snapshot', dirty, report, absent)
875
961
  ignored = []
876
962
  if args.command == 'snapshot' and args.notes:
877
963
  # Only the selected pages must be valid; errors elsewhere are counted, not blocking.
@@ -904,6 +990,26 @@ def main(argv=None):
904
990
  return 1
905
991
  if ignored:
906
992
  print(f'{args.command}: {len(ignored)} errors on {len({error_page(m) for m in ignored if error_page(m)})} other pages do not block this snapshot (run check to see them)', file=sys.stderr)
993
+ if args.command == 'build':
994
+ out = Path(os.path.abspath(args.out))
995
+ if out.is_dir() or out.resolve().is_relative_to(vault):
996
+ parser.error('--out must be a file outside the vault: nothing is generated into the vault')
997
+ try:
998
+ page, payload = built_page(repo, vault, notes, edges, warnings, args.with_sources, args.source_url)
999
+ except ValueError as exc:
1000
+ parser.error(str(exc))
1001
+ atomic_write(out, page)
1002
+ files = len(payload['files'])
1003
+ print(f'build: wrote {out} ({len(page.encode("utf-8")) // 1024} kB), {len(notes)} notes, {len(warnings)} warnings'
1004
+ + (f', commit {payload["commit"][:7]}' if payload['commit'] else ''))
1005
+ print('build: ' + (f'the text of {files} cited source files is inside the file; show it only to people who may read the repository'
1006
+ if args.with_sources else 'cited source files are named but their text is not included (--with-sources adds it)'))
1007
+ if absent:
1008
+ names = sorted(set(absent))
1009
+ print(f'build: {len(names)} cited files are not in this folder, so nobody compared them with the pages: ' + ', '.join(names[:8]) + (' ...' if len(names) > 8 else ''))
1010
+ if shallow_commits(repo):
1011
+ print('build: this is a shallow clone, so the Changes page starts at the first commit that was fetched in full; fetch the whole history to show more')
1012
+ return 0
907
1013
  if args.command == 'snapshot':
908
1014
  for note in notes:
909
1015
  meta = note['meta']