project-spec 0.7.0 → 0.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +52 -0
- package/bin/project-spec.mjs +3 -1
- package/package.json +1 -1
- package/skills/project-knowledge/SKILL.md +1 -1
- package/skills/project-knowledge/assets/viewer.html +53 -11
- package/skills/project-knowledge/references/format.md +3 -1
- package/skills/project-knowledge/scripts/knowledge.py +69 -6
- package/skills/project-knowledge/scripts/test_knowledge.py +80 -0
package/README.md
CHANGED
|
@@ -88,6 +88,57 @@ alias spec='python3 ~/.claude/skills/project-knowledge/scripts/knowledge.py serv
|
|
|
88
88
|
|
|
89
89
|
The vault is found by itself when the repository has one under `knowledge/`; otherwise add `--vault PATH`. The page is served on port 8757, or another free port when that one is taken; `--port N` picks one. `project-spec serve` opens the browser unless you add `--no-open`; the helper opens it only with `--open`. If the vault has errors, the page lists them.
|
|
90
90
|
|
|
91
|
+
### Show it on a server
|
|
92
|
+
|
|
93
|
+
`serve` is for your own machine. To let the rest of the team read the spec on a development or staging server, write the viewer as one file and let your application return it:
|
|
94
|
+
|
|
95
|
+
```sh
|
|
96
|
+
project-spec build --out build/spec.html \
|
|
97
|
+
--source-url 'https://github.com/OWNER/REPO/blob/{commit}'
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
Without the package the command is `python3 <skill>/scripts/knowledge.py build --repo . --out build/spec.html`. The file is the whole viewer: the pages, the search, the map, the gap queue and the history, with no other files and no network requests. It is a copy of the vault at the moment of the build, so build it again on every deploy.
|
|
101
|
+
|
|
102
|
+
| Option | What it does |
|
|
103
|
+
| --- | --- |
|
|
104
|
+
| `--out FILE` | Where to write the page. It must be outside the vault: nothing is generated into the vault. |
|
|
105
|
+
| `--source-url URL` | Where a cited file can be read. The file's path in the repository is added to this address, and `{commit}` becomes the commit the page was built from. Without it, cited files are shown as names. |
|
|
106
|
+
| `--with-sources` | Put the text of the cited files into the page, as `serve` does, so they open inside the viewer. Leave it out when people who may not read the repository can open the page. |
|
|
107
|
+
|
|
108
|
+
The spec describes how the product works inside, and with `--with-sources` it carries source code. So treat the file like the repository:
|
|
109
|
+
|
|
110
|
+
- Keep it out of folders the web server publishes by itself (`public/`, `static/`). Return it from a route that checks who is asking.
|
|
111
|
+
- Build it only for the environments that should have it, and make the route answer 404 everywhere else.
|
|
112
|
+
- If your application sets a `Content-Security-Policy`, the page needs inline scripts and styles and nothing else.
|
|
113
|
+
|
|
114
|
+
For example, a route in a Next.js application (a sketch: put your own sign-in check where `signedIn()` is):
|
|
115
|
+
|
|
116
|
+
```ts
|
|
117
|
+
// src/app/development/spec/route.ts
|
|
118
|
+
import { readFile } from "node:fs/promises";
|
|
119
|
+
import path from "node:path";
|
|
120
|
+
|
|
121
|
+
export const dynamic = "force-dynamic";
|
|
122
|
+
|
|
123
|
+
export async function GET() {
|
|
124
|
+
if (process.env.SPEC_PAGE !== "on" || !(await signedIn())) {
|
|
125
|
+
return new Response("Not found", { status: 404 });
|
|
126
|
+
}
|
|
127
|
+
const page = await readFile(path.join(process.cwd(), "build/spec.html"));
|
|
128
|
+
return new Response(page, {
|
|
129
|
+
headers: {
|
|
130
|
+
"Content-Type": "text/html; charset=utf-8",
|
|
131
|
+
"Cache-Control": "no-store",
|
|
132
|
+
"X-Robots-Tag": "noindex",
|
|
133
|
+
"Content-Security-Policy":
|
|
134
|
+
"default-src 'none'; script-src 'unsafe-inline'; style-src 'unsafe-inline'; base-uri 'none'; form-action 'none'",
|
|
135
|
+
},
|
|
136
|
+
});
|
|
137
|
+
}
|
|
138
|
+
```
|
|
139
|
+
|
|
140
|
+
The build needs Python and Git, so run it where the application is built (a CI job or a build stage), not inside a running container that has neither. The **Recent changes** page comes from the Git history: in a shallow clone it starts at the first commit that was fetched in full, and the command says so. In GitHub Actions, `fetch-depth: 0` on the checkout step gives it the whole history.
|
|
141
|
+
|
|
91
142
|
## What a feature page says
|
|
92
143
|
|
|
93
144
|
Every feature, screen or flow has one page, and all of them have the same sections in the same order. A section the feature does not have is left out.
|
|
@@ -145,6 +196,7 @@ The package is [`project-spec`](https://www.npmjs.com/package/project-spec) on t
|
|
|
145
196
|
```sh
|
|
146
197
|
npx project-spec install --claude # or --codex, or --dir knowledge/tooling
|
|
147
198
|
npx project-spec serve # the viewer for the project in the current folder
|
|
199
|
+
npx project-spec build --out FILE # the same viewer as one file, to show on a server
|
|
148
200
|
```
|
|
149
201
|
|
|
150
202
|
Then ask your agent to use the `project-knowledge` skill ([example prompts](#example-prompts)).
|
package/bin/project-spec.mjs
CHANGED
|
@@ -13,7 +13,7 @@ const SKILLS = path.join(ROOT, 'skills');
|
|
|
13
13
|
const PACKAGE = JSON.parse(fs.readFileSync(path.join(ROOT, 'package.json'), 'utf8'));
|
|
14
14
|
const MARKER = '.project-spec.json'; // written into every copy: which version it is and what its files looked like
|
|
15
15
|
const SKIP = new Set(['__pycache__', '.DS_Store', MARKER]);
|
|
16
|
-
const KNOWLEDGE = new Set(['serve', 'check', 'snapshot']);
|
|
16
|
+
const KNOWLEDGE = new Set(['serve', 'build', 'check', 'snapshot']);
|
|
17
17
|
const MAINTAIN = new Set(['scan', 'receipt', 'finish']);
|
|
18
18
|
// Skills earlier versions shipped; project-knowledge and spec-change took over what they did.
|
|
19
19
|
const RETIRED = ['spec-audit', 'spec-investigation', 'spec-plan'];
|
|
@@ -25,6 +25,8 @@ const USAGE = `project-spec ${PACKAGE.version}
|
|
|
25
25
|
Read the spec
|
|
26
26
|
project-spec serve [--repo DIR] [--vault DIR] [--port N] [--no-open]
|
|
27
27
|
Open the viewer for the repository (default: the current folder).
|
|
28
|
+
project-spec build --out FILE [--source-url URL] [--with-sources] [--repo DIR] [--vault DIR]
|
|
29
|
+
Write the viewer as one HTML file, to show the spec on a server.
|
|
28
30
|
project-spec check | snapshot [options]
|
|
29
31
|
The knowledge helper; --repo defaults to the current folder.
|
|
30
32
|
project-spec scan | receipt | finish [options]
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "project-spec",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.8.0",
|
|
4
4
|
"description": "Agent skills that read a repository and keep a spec people can read: one page per feature, every business rule numbered. Local viewer and one command to install and update the skills.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"agent-skills",
|
|
@@ -124,7 +124,7 @@ python3 "$SKILL_DIR/scripts/knowledge.py" snapshot --repo "$REPO" --vault "$VAUL
|
|
|
124
124
|
|
|
125
125
|
`check` validates structure and freshness; it cannot judge prose or coverage. `--strict` turns freshness warnings into a failing exit status, for CI. In a working tree shared with someone else's uncommitted work, `--strict --ignore-dirty` still prints warnings for sources with uncommitted changes but does not fail on them when the accepted fingerprint equals the file at `HEAD` (the only drift is uncommitted); committed drift still fails, and it is not for CI ([details](references/format.md)). `serve` shows the vault in a browser and writes nothing. `snapshot` accepts source fingerprints as reviewed; run it only for pages you actually re-read against their sources, never to clear a warning. An initial reviewed pass may use `--all`.
|
|
126
126
|
|
|
127
|
-
To read the result, run `python3 <skill>/scripts/knowledge.py serve --repo <repository> --open` (add `--vault` when the repository has several vaults under `knowledge/`), or open the folder as an Obsidian vault. The command is long, so when you hand over offer to give it a short name in the project: a `spec` script in `package.json` (`project-spec serve` when the project has the `project-spec-skills` package, otherwise the full command), a `spec` target in a `Makefile`, or a shell alias. That file is outside the vault, so add it only when the person agrees. The viewer is one page on `127.0.0.1`; it reads the vault and the files the pages cite from disk on every page load, so it needs no build and shows the current state after a reload. It shows a note as written, under its title and summary, with the source files folded away at the end; an area page shows its notes with their summaries. So the first paragraph and the summary are what most readers see. Its **Copy for AI** turns a page and the reader's own words into a request for an agent that names the page file. It only copies text, so an agent that receives one still follows the skill it names, and a recheck or decision the reader states there is recorded under the reader's name, never as the agent's. If the vault has errors the page lists them instead.
|
|
127
|
+
To read the result, run `python3 <skill>/scripts/knowledge.py serve --repo <repository> --open` (add `--vault` when the repository has several vaults under `knowledge/`), or open the folder as an Obsidian vault. The command is long, so when you hand over offer to give it a short name in the project: a `spec` script in `package.json` (`project-spec serve` when the project has the `project-spec-skills` package, otherwise the full command), a `spec` target in a `Makefile`, or a shell alias. That file is outside the vault, so add it only when the person agrees. The viewer is one page on `127.0.0.1`; it reads the vault and the files the pages cite from disk on every page load, so it needs no build and shows the current state after a reload. It shows a note as written, under its title and summary, with the source files folded away at the end; an area page shows its notes with their summaries. So the first paragraph and the summary are what most readers see. Its **Copy for AI** turns a page and the reader's own words into a request for an agent that names the page file. It only copies text, so an agent that receives one still follows the skill it names, and a recheck or decision the reader states there is recorded under the reader's name, never as the agent's. If the vault has errors the page lists them instead. When the person wants the spec on a server for people without the repository, `knowledge.py build --repo <repository> --out <file outside the vault>` writes that same viewer as one file ([options](references/format.md)); run it only when asked, never commit the file into the vault, and leave `--with-sources` out unless the person says everyone who can open the page may read the code.
|
|
128
128
|
|
|
129
129
|
Before handing over, read what you wrote as the person who will use it, not as its author. Open one area page and two of the notes you wrote or changed, and ask:
|
|
130
130
|
|
|
@@ -1558,8 +1558,9 @@
|
|
|
1558
1558
|
(n) => n && typeof n === "object",
|
|
1559
1559
|
),
|
|
1560
1560
|
edges = Array.isArray(data.edges) ? data.edges : [];
|
|
1561
|
-
// Sent by `knowledge.py serve`
|
|
1562
|
-
// repository, its Git history and its uncommitted pages.
|
|
1561
|
+
// Sent by `knowledge.py serve` and `build`: the text of cited repository files (build leaves it out
|
|
1562
|
+
// unless asked), the vault's place in the repository, its Git history and its uncommitted pages.
|
|
1563
|
+
// A page without them still works.
|
|
1563
1564
|
const files =
|
|
1564
1565
|
data.files && typeof data.files === "object" ? data.files : {},
|
|
1565
1566
|
vaultPath = typeof data.vaultPath === "string" ? data.vaultPath : "",
|
|
@@ -1570,6 +1571,15 @@
|
|
|
1570
1571
|
(p) => p && typeof p === "object",
|
|
1571
1572
|
);
|
|
1572
1573
|
const live = data.live === true;
|
|
1574
|
+
// A page written by `knowledge.py build` to be put on a server: a copy of one moment, read by people
|
|
1575
|
+
// who may not have the repository. `sourceUrl` is where a cited file can be read instead.
|
|
1576
|
+
const built = data.built === true,
|
|
1577
|
+
builtCommit = typeof data.commit === "string" ? data.commit : "",
|
|
1578
|
+
builtOn = typeof data.generatedAt === "string" ? data.generatedAt.slice(0, 10) : "",
|
|
1579
|
+
sourceUrl =
|
|
1580
|
+
built && typeof data.sourceUrl === "string" && /^https?:\/\/[^\s/]/i.test(data.sourceUrl)
|
|
1581
|
+
? data.sourceUrl.replace(/\/+$/, "")
|
|
1582
|
+
: "";
|
|
1573
1583
|
const byId = new Map(nodes.map((n) => [n.id, n])),
|
|
1574
1584
|
byPath = new Map(
|
|
1575
1585
|
nodes.map((n) => [String(n.path || "").replace(/\.md$/i, ""), n]),
|
|
@@ -1694,7 +1704,13 @@
|
|
|
1694
1704
|
const areas = nodes.filter((n) => n.type === "area").sort(byTitle);
|
|
1695
1705
|
$("#project-title").textContent = projectTitle;
|
|
1696
1706
|
$("#project-title").title = projectTitle;
|
|
1697
|
-
$("#header-note").textContent = live
|
|
1707
|
+
$("#header-note").textContent = live
|
|
1708
|
+
? "Spec · live from disk"
|
|
1709
|
+
: built
|
|
1710
|
+
? "Spec" + (builtOn ? " · built " + builtOn : "")
|
|
1711
|
+
: "Spec";
|
|
1712
|
+
// The line is too narrow for the commit as well, so it is in the tooltip.
|
|
1713
|
+
if (built && builtCommit) $("#header-note").title = "Built from commit " + builtCommit;
|
|
1698
1714
|
// Small things kept in this browser only: where you were, what you picked. Never required.
|
|
1699
1715
|
const store = {
|
|
1700
1716
|
key: (name) => "spec-viewer:" + projectTitle + ":" + name,
|
|
@@ -1834,8 +1850,9 @@
|
|
|
1834
1850
|
return out.join("/");
|
|
1835
1851
|
}
|
|
1836
1852
|
const hasFile = (key) => Object.prototype.hasOwnProperty.call(files, key);
|
|
1837
|
-
// A relative Markdown link
|
|
1838
|
-
|
|
1853
|
+
// A relative Markdown link read as a place in the repository: the file and which lines it names.
|
|
1854
|
+
// Null for an address, an anchor or a path that leaves the repository.
|
|
1855
|
+
function repoTarget(raw, n) {
|
|
1839
1856
|
let t = String(raw).trim();
|
|
1840
1857
|
if (t.startsWith("<") && t.endsWith(">")) t = t.slice(1, -1);
|
|
1841
1858
|
if (!t || t[0] === "#" || t[0] === "/" || /^[A-Za-z][A-Za-z0-9+.-]*:/.test(t))
|
|
@@ -1849,10 +1866,27 @@
|
|
|
1849
1866
|
const dir = String(n.path || "").split("/").slice(0, -1).join("/");
|
|
1850
1867
|
// Vault pages link relative to themselves inside the vault; a cited document links relative to itself in the repository.
|
|
1851
1868
|
const key = normPath(n.repoFile ? [dir, t] : [vaultPath, dir, t]);
|
|
1852
|
-
if (!
|
|
1869
|
+
if (!key || key.split("/")[0] === "..") return null;
|
|
1853
1870
|
const m = hash.match(/^L(\d+)(?:-L?(\d+))?$/i);
|
|
1854
1871
|
return { key, from: m ? +m[1] : 0, to: m ? +(m[2] || m[1]) : 0 };
|
|
1855
1872
|
}
|
|
1873
|
+
// The same link when the text of that file came with the page, so it can be opened in place.
|
|
1874
|
+
function fileRef(raw, n) {
|
|
1875
|
+
const ref = repoTarget(raw, n);
|
|
1876
|
+
return ref && hasFile(ref.key) ? ref : null;
|
|
1877
|
+
}
|
|
1878
|
+
// A cited file on the code host a built page was given: a new tab, since the reader leaves the spec.
|
|
1879
|
+
function sourceLink(ref, text) {
|
|
1880
|
+
const a = el("a", text);
|
|
1881
|
+
a.href =
|
|
1882
|
+
sourceUrl +
|
|
1883
|
+
"/" +
|
|
1884
|
+
ref.key.split("/").map(encodeURIComponent).join("/") +
|
|
1885
|
+
(ref.from ? "#L" + ref.from + (ref.to > ref.from ? "-L" + ref.to : "") : "");
|
|
1886
|
+
a.target = "_blank";
|
|
1887
|
+
a.rel = "noopener noreferrer";
|
|
1888
|
+
return a;
|
|
1889
|
+
}
|
|
1856
1890
|
const isMarkdownFile = (key) => /\.(md|markdown)$/i.test(key);
|
|
1857
1891
|
let fileState = null;
|
|
1858
1892
|
function fileButton(ref, text) {
|
|
@@ -1888,7 +1922,7 @@
|
|
|
1888
1922
|
? "Lines " + ref.from + "–" + ref.to
|
|
1889
1923
|
: "Line " + ref.from) + " · "
|
|
1890
1924
|
: "") +
|
|
1891
|
-
(live ? "Read from disk when this page loaded" : "Copy taken when this
|
|
1925
|
+
(live ? "Read from disk when this page loaded" : "Copy taken when this page was built") +
|
|
1892
1926
|
(file.truncated ? " · only the beginning of the file is shown" : "");
|
|
1893
1927
|
actions.replaceChildren();
|
|
1894
1928
|
if (isMarkdownFile(ref.key))
|
|
@@ -1957,9 +1991,16 @@
|
|
|
1957
1991
|
else {
|
|
1958
1992
|
const parts = token.match(/^\[([^\]]+)\]\(([^)]+)\)$/),
|
|
1959
1993
|
ref = parts && fileRef(parts[2], n),
|
|
1960
|
-
|
|
1994
|
+
// On a built page a relative link would lead into the site that hosts it, not to the file.
|
|
1995
|
+
away = parts && !ref && built && !n.repoFile ? repoTarget(parts[2], n) : null,
|
|
1996
|
+
href = parts && !ref && !away && !n.repoFile && safeHref(parts[2], n);
|
|
1961
1997
|
if (ref) parent.append(fileButton(ref, parts[1]));
|
|
1962
|
-
else if (
|
|
1998
|
+
else if (away && sourceUrl) parent.append(sourceLink(away, parts[1]));
|
|
1999
|
+
else if (away) {
|
|
2000
|
+
const name = el("span", parts[1]);
|
|
2001
|
+
name.title = away.key;
|
|
2002
|
+
parent.append(name);
|
|
2003
|
+
} else if (href) {
|
|
1963
2004
|
const a = el("a", parts[1]);
|
|
1964
2005
|
a.href = href;
|
|
1965
2006
|
parent.append(a);
|
|
@@ -2464,7 +2505,8 @@
|
|
|
2464
2505
|
};
|
|
2465
2506
|
const sourceLine = (src) => {
|
|
2466
2507
|
const key = normPath([src]);
|
|
2467
|
-
|
|
2508
|
+
const ref = { key, from: 0, to: 0 };
|
|
2509
|
+
return hasFile(key) ? fileButton(ref, src) : sourceUrl && key ? sourceLink(ref, src) : el("span", src);
|
|
2468
2510
|
};
|
|
2469
2511
|
|
|
2470
2512
|
// --- Views ---------------------------------------------------------------------------------------
|
|
@@ -2891,7 +2933,7 @@
|
|
|
2891
2933
|
"p",
|
|
2892
2934
|
historyLog.length
|
|
2893
2935
|
? "Nothing in the history matches this filter."
|
|
2894
|
-
: live
|
|
2936
|
+
: live || built
|
|
2895
2937
|
? "No Git history was found for this vault."
|
|
2896
2938
|
: "History is shown when the spec is opened with the serve command.",
|
|
2897
2939
|
"empty",
|
|
@@ -166,7 +166,7 @@ The validator enforces these body rules:
|
|
|
166
166
|
|
|
167
167
|
Three scripts in `$SKILL_DIR/scripts/`; see [SKILL.md](../SKILL.md) for the variables.
|
|
168
168
|
|
|
169
|
-
`knowledge.py {check,snapshot,serve} --repo PATH [--vault PATH]`
|
|
169
|
+
`knowledge.py {check,snapshot,serve,build} --repo PATH [--vault PATH]`
|
|
170
170
|
|
|
171
171
|
- `check`: read-only, exit 1 for structural errors; freshness warnings normally leave exit status 0. Add `--strict` to also fail on warnings. Page-level errors are printed first, and a failing `check` or `snapshot` ends with a line giving the number of errors and the number of pages they are on. A page whose frontmatter fails validation still counts as a target for links, embeds, area membership, a plan's `gap` and `duplicate_of`, so one broken page yields its own errors only; fix it and nothing else needs touching. Only the "missing" error is hidden this way: an ambiguous link (two or more candidates), a link to a page of the wrong type (a spec page linking a gap, or an `areas` entry naming a page known to be another type) and a run page's link are still reported. `snapshot --notes X` refuses X while X's area references, links, `gap` or `duplicate_of` point at a page that failed validation. Missing source files are errors except on explicitly `stale` or `superseded` notes (warnings). Changed or unbaselined evidence warns. A source that differs from its baseline only by line endings (typically a Windows checkout) warns "line endings differ in source X"; do not snapshot on that checkout, because it would record the converted bytes. Nothing is rewritten.
|
|
172
172
|
|
|
@@ -204,6 +204,8 @@ Canonical Markdown must not contain raw merge conflicts. Validation rejects comp
|
|
|
204
204
|
|
|
205
205
|
`serve` shows the vault in a browser: `knowledge.py serve --repo REPO [--vault VAULT] [--port N] [--open]`. Without `--vault`, this and every other command use the single vault under `knowledge/` (`knowledge/00-index.md` or `knowledge/*/00-index.md`) and stop with a message when there is none or several. It listens on `127.0.0.1` only (port 8757, or a free one when that is taken), answers only `/` and only when addressed as this machine, never writes, and builds the page again from disk on every request, so a reload shows the current files. When the vault has errors the page lists them. The page data is the graph above plus `live: true`, `vaultPath` (the vault's path relative to the repository), `history`, `pending` and `files`. `history` is the last 30 commits that touched vault pages (not `.knowledge/`), newest first, each `{"sha", "date", "subject", "pages", "more"}`; a page entry is `{"path", "status"}` with `status` `added`, `changed` or `removed`, plus `was` and `now` when the commit changed the page's `summary` (or, failing that, its `title`); at most 80 pages are listed per commit and `more` counts the rest. `pending` lists vault pages with uncommitted changes in the same `{"path", "status"}` shape. Both are empty outside a Git work tree. `files` is the text of the repository files that pages cite in `sources` or link to, keyed by repository-relative path, each `{"text", "truncated"}`. A file is included only when it is inside the repository and outside the vault, is not a credential path, and is UTF-8 text; one file is cut at 300,000 characters and all files together at 4,000,000, after which the rest stay ordinary links. The viewer opens a link to an included file in place: Markdown rendered, anything else as numbered lines, with the lines of a `#L10` or `#L10-L20` fragment marked.
|
|
206
206
|
|
|
207
|
+
`build` writes the same viewer as one file, for a server that shows the spec to people who do not have the repository: `knowledge.py build --repo REPO [--vault VAULT] --out FILE [--source-url URL] [--with-sources]`. `--out` must be a file outside the vault. The command stops with the `check` errors when the vault has any, and writes nothing then; freshness warnings do not stop it. The page data is the graph plus `built: true`, `commit` (the full `HEAD`, empty outside Git), `sourceUrl`, `vaultPath`, `history`, `pending` and `files`. `files` is empty unless `--with-sources` is given, because the text of cited files is source code. `--source-url` must be an `http(s)` address; `{commit}` in it is replaced by `HEAD`, and it is an error to name `{commit}` outside a Git work tree. On a built page a cited file whose text is not included is a link to `sourceUrl/<repository path>` (with `#L10-L20` when lines are named) in a new tab, or plain text when there is no `sourceUrl`; it is never a relative link, which would lead into the site that hosts the page. In a shallow clone the commit whose parents were not fetched is left out of `history` (by `serve` too), since Git would show every page as added there, and `build` says that the history is short.
|
|
208
|
+
|
|
207
209
|
`assets/viewer.html` has exactly one `__KNOWLEDGE_DATA__` placeholder for script-safe JSON. Treat contents as untrusted text: no raw Markdown HTML execution, unsafe URL schemes, external scripts, fonts, analytics or fetches. Render readable basic Markdown, pipe tables and wikilinks; leave unsupported syntax as text. The viewer has a tree of areas with their notes (a note is listed under its first area; superseded notes are not listed), a search over pages, gaps and the cited files, and five places: the overview, a page, the gap queue, the map and recent changes. A note shows its title, `summary`, the page text and the gaps about it that are not closed, then its source files folded; an area shows its notes with their summaries and folds its own text. The pages linked to or from the page are a short list beside it. Navigation keeps a trail for Back and Forward and writes the place into the address (`#/page/notes/example`, `#/gaps`), so a reload or a copied link returns to it. The map draws one frame per primary area (the area page, or the first entry of `areas`) with its pages inside; an edge between a page and its primary area is not drawn because the frame says it, links inside one area are solid and links across areas dashed; `nav` pages are not drawn. Recent changes reads `history`, `pending` and the "changed source" lines of `warnings`. "Copy for AI" only composes text from the page, a choice of what the agent should do and the reader's own words, and copies it to the clipboard. The reader's name, the recently viewed pages and the last commit seen are kept in the browser's `localStorage`, and the viewer works without it.
|
|
208
210
|
|
|
209
211
|
## Format references
|
|
@@ -689,6 +689,17 @@ def front_value(line):
|
|
|
689
689
|
return value if isinstance(value, str) else ''
|
|
690
690
|
|
|
691
691
|
|
|
692
|
+
def shallow_commits(repo):
|
|
693
|
+
"""Commits whose parents a shallow clone left out. Empty in a full clone and outside Git."""
|
|
694
|
+
found = subprocess.run(['git', '-C', str(repo), 'rev-parse', '--git-path', 'shallow'], capture_output=True)
|
|
695
|
+
if found.returncode:
|
|
696
|
+
return set()
|
|
697
|
+
try:
|
|
698
|
+
return set((repo / os.fsdecode(found.stdout).strip()).read_text(encoding='ascii').split())
|
|
699
|
+
except (OSError, UnicodeDecodeError):
|
|
700
|
+
return set()
|
|
701
|
+
|
|
702
|
+
|
|
692
703
|
def vault_history(repo, vault):
|
|
693
704
|
"""Recent commits that touched vault pages, newest first: sha, date, subject and what happened to each page.
|
|
694
705
|
|
|
@@ -717,10 +728,12 @@ def vault_history(repo, vault):
|
|
|
717
728
|
return []
|
|
718
729
|
if log.returncode:
|
|
719
730
|
return []
|
|
720
|
-
commits = []
|
|
731
|
+
commits, boundary = [], shallow_commits(repo)
|
|
721
732
|
for chunk in log.stdout.decode('utf-8', errors='replace').split(marker)[1:]:
|
|
722
733
|
header, _, body = chunk.partition('\n')
|
|
723
734
|
sha, date, subject = (header.split('\x02', 2) + ['', ''])[:3]
|
|
735
|
+
if sha in boundary: # its parents were not fetched, so Git would show every page as added
|
|
736
|
+
continue
|
|
724
737
|
subject = CONTROL.sub('', subject)
|
|
725
738
|
pages, page = [], None
|
|
726
739
|
for line in body.split('\n'):
|
|
@@ -772,6 +785,13 @@ def vault_pending(repo, vault):
|
|
|
772
785
|
return sorted(pending, key=lambda p: p['path'])
|
|
773
786
|
|
|
774
787
|
|
|
788
|
+
def page_payload(repo, vault, notes, edges, warnings, sources=True):
|
|
789
|
+
files = cited_files(repo, vault, notes)[0] if sources else {}
|
|
790
|
+
return graph_of(notes, edges, warnings) | {
|
|
791
|
+
'vaultPath': Path(os.path.relpath(vault, repo)).as_posix(), 'files': files,
|
|
792
|
+
'history': vault_history(repo, vault), 'pending': vault_pending(repo, vault)}
|
|
793
|
+
|
|
794
|
+
|
|
775
795
|
def live_page(repo, vault):
|
|
776
796
|
"""The viewer for the vault as it is on disk right now, or a page listing what stops it from loading."""
|
|
777
797
|
try:
|
|
@@ -785,10 +805,29 @@ def live_page(repo, vault):
|
|
|
785
805
|
f'<h1>The spec has {len(errors)} error{"s" if len(errors) != 1 else ""}</h1>'
|
|
786
806
|
'<p>Fix the pages below (the <code>check</code> command prints the same list), then reload.</p>'
|
|
787
807
|
f'<ul>{items}</ul></body>')
|
|
788
|
-
|
|
789
|
-
|
|
790
|
-
|
|
791
|
-
|
|
808
|
+
return viewer_html(page_payload(repo, vault, notes, edges, warnings) | {'live': True})
|
|
809
|
+
|
|
810
|
+
|
|
811
|
+
def source_base(value, commit):
|
|
812
|
+
"""The address `--source-url` names, with `{commit}` filled in. Raises ValueError with the reason when it cannot be used."""
|
|
813
|
+
if CONTROL.search(value) or ' ' in value or not re.match(r'https?://[^/]', value, re.I):
|
|
814
|
+
raise ValueError('--source-url must be an http(s) address, e.g. https://github.com/OWNER/REPO/blob/{commit}')
|
|
815
|
+
if '{commit}' in value and not commit:
|
|
816
|
+
raise ValueError('--source-url names {commit}, but --repo is not a Git work tree with a commit')
|
|
817
|
+
return value.replace('{commit}', commit).rstrip('/')
|
|
818
|
+
|
|
819
|
+
|
|
820
|
+
def built_page(repo, vault, notes, edges, warnings, with_sources=False, source_url=''):
|
|
821
|
+
"""The viewer as one file to put on a server: the vault as it is now, with nothing read later.
|
|
822
|
+
|
|
823
|
+
The text of cited repository files is left out unless asked for, because the file may be shown to people
|
|
824
|
+
who cannot read the repository. Returns (html, payload).
|
|
825
|
+
"""
|
|
826
|
+
head = subprocess.run(['git', '-C', str(repo), 'rev-parse', 'HEAD'], capture_output=True)
|
|
827
|
+
commit = '' if head.returncode else head.stdout.decode('ascii', errors='replace').strip()
|
|
828
|
+
payload = page_payload(repo, vault, notes, edges, warnings, with_sources) | {
|
|
829
|
+
'built': True, 'commit': commit, 'sourceUrl': source_base(source_url, commit) if source_url else ''}
|
|
830
|
+
return viewer_html(payload), payload
|
|
792
831
|
|
|
793
832
|
|
|
794
833
|
def make_server(repo, vault, port):
|
|
@@ -820,12 +859,15 @@ def make_server(repo, vault, port):
|
|
|
820
859
|
|
|
821
860
|
def main(argv=None):
|
|
822
861
|
parser = argparse.ArgumentParser()
|
|
823
|
-
parser.add_argument('command', choices=['check', 'snapshot', 'serve'])
|
|
862
|
+
parser.add_argument('command', choices=['check', 'snapshot', 'serve', 'build'])
|
|
824
863
|
parser.add_argument('--repo', required=True, type=Path)
|
|
825
864
|
parser.add_argument('--vault', type=Path, help='default: the single vault under knowledge/ in the repository')
|
|
826
865
|
parser.add_argument('--strict', action='store_true', help='fail check when freshness warnings exist')
|
|
827
866
|
parser.add_argument('--port', type=int, default=8757, help='serve: port on 127.0.0.1 (another free port is used if it is taken)')
|
|
828
867
|
parser.add_argument('--open', action='store_true', help='serve: open the page in the default browser')
|
|
868
|
+
parser.add_argument('--out', type=Path, help='build: the HTML file to write, outside the vault')
|
|
869
|
+
parser.add_argument('--with-sources', action='store_true', help='build: include the text of the repository files the pages cite')
|
|
870
|
+
parser.add_argument('--source-url', default='', help='build: where a cited file can be read, e.g. https://github.com/OWNER/REPO/blob/{commit}')
|
|
829
871
|
parser.add_argument('--ignore-dirty', action='store_true',
|
|
830
872
|
help='check: do not count freshness warnings for sources with uncommitted changes (needs a Git work tree)')
|
|
831
873
|
group = parser.add_mutually_exclusive_group()
|
|
@@ -838,6 +880,10 @@ def main(argv=None):
|
|
|
838
880
|
parser.error('--ignore-dirty is only for check')
|
|
839
881
|
if args.open and args.command != 'serve':
|
|
840
882
|
parser.error('--open is only for serve')
|
|
883
|
+
if args.command == 'build' and not args.out:
|
|
884
|
+
parser.error('build requires --out FILE')
|
|
885
|
+
if args.command != 'build' and (args.out or args.with_sources or args.source_url):
|
|
886
|
+
parser.error('--out, --with-sources and --source-url are only for build')
|
|
841
887
|
if args.command == 'snapshot' and not (args.all or args.notes):
|
|
842
888
|
parser.error('snapshot requires --all or --notes')
|
|
843
889
|
if args.command != 'snapshot' and (args.all or args.notes):
|
|
@@ -904,6 +950,23 @@ def main(argv=None):
|
|
|
904
950
|
return 1
|
|
905
951
|
if ignored:
|
|
906
952
|
print(f'{args.command}: {len(ignored)} errors on {len({error_page(m) for m in ignored if error_page(m)})} other pages do not block this snapshot (run check to see them)', file=sys.stderr)
|
|
953
|
+
if args.command == 'build':
|
|
954
|
+
out = Path(os.path.abspath(args.out))
|
|
955
|
+
if out.is_dir() or out.resolve().is_relative_to(vault):
|
|
956
|
+
parser.error('--out must be a file outside the vault: nothing is generated into the vault')
|
|
957
|
+
try:
|
|
958
|
+
page, payload = built_page(repo, vault, notes, edges, warnings, args.with_sources, args.source_url)
|
|
959
|
+
except ValueError as exc:
|
|
960
|
+
parser.error(str(exc))
|
|
961
|
+
atomic_write(out, page)
|
|
962
|
+
files = len(payload['files'])
|
|
963
|
+
print(f'build: wrote {out} ({len(page.encode("utf-8")) // 1024} kB), {len(notes)} notes, {len(warnings)} warnings'
|
|
964
|
+
+ (f', commit {payload["commit"][:7]}' if payload['commit'] else ''))
|
|
965
|
+
print('build: ' + (f'the text of {files} cited source files is inside the file; show it only to people who may read the repository'
|
|
966
|
+
if args.with_sources else 'cited source files are named but their text is not included (--with-sources adds it)'))
|
|
967
|
+
if shallow_commits(repo):
|
|
968
|
+
print('build: this is a shallow clone, so the Changes page starts at the first commit that was fetched in full; fetch the whole history to show more')
|
|
969
|
+
return 0
|
|
907
970
|
if args.command == 'snapshot':
|
|
908
971
|
for note in notes:
|
|
909
972
|
meta = note['meta']
|
|
@@ -583,6 +583,86 @@ class GapLayerTests(Fixture):
|
|
|
583
583
|
self.assertEqual(k.vault_pending(inner.resolve(), (inner / 'vault').resolve()), [])
|
|
584
584
|
k._history_cache.clear()
|
|
585
585
|
|
|
586
|
+
def built(self, *args):
|
|
587
|
+
out = self.root / 'site/spec.html'
|
|
588
|
+
status, printed, err = self.run_cli('build', '--out', str(out), *args)
|
|
589
|
+
data = json.loads(re.search(r'<script id="knowledge-data" type="application/json">\s*(.*?)\s*</script>', out.read_text(encoding='utf-8'), re.S).group(1)) if out.is_file() else None
|
|
590
|
+
return status, printed, err, data
|
|
591
|
+
|
|
592
|
+
def refused(self, *args):
|
|
593
|
+
"""What a command that stops with a usage error says."""
|
|
594
|
+
err = StringIO()
|
|
595
|
+
with self.assertRaises(SystemExit), redirect_stdout(StringIO()), redirect_stderr(err):
|
|
596
|
+
k.main([*args, '--repo', str(self.repo), '--vault', str(self.vault)])
|
|
597
|
+
return err.getvalue()
|
|
598
|
+
|
|
599
|
+
def test_build_writes_one_page_for_a_server_without_source_text(self):
|
|
600
|
+
status, printed, err, data = self.built()
|
|
601
|
+
self.assertEqual((status, err), (0, 'WARNING: notes/n.md: unbaselined source source.txt\n'))
|
|
602
|
+
self.assertIn('build: wrote ' + str(self.root / 'site/spec.html'), printed)
|
|
603
|
+
self.assertIn('their text is not included', printed)
|
|
604
|
+
self.assertEqual((data['built'], data.get('live'), data['files'], data['sourceUrl'], data['commit']), (True, None, {}, '', ''))
|
|
605
|
+
self.assertEqual({n['id'] for n in data['nodes']}, {'a', 'index', 'n'})
|
|
606
|
+
self.assertNotIn('evidence', (self.root / 'site/spec.html').read_text(encoding='utf-8'))
|
|
607
|
+
# nothing appeared in the vault, and no temporary file was left beside the page
|
|
608
|
+
self.assertFalse((self.vault / '.knowledge').exists())
|
|
609
|
+
self.assertEqual([f.name for f in (self.root / 'site').iterdir()], ['spec.html'])
|
|
610
|
+
# the text of cited files only when asked for, with a warning about who may see it
|
|
611
|
+
status, printed, _, data = self.built('--with-sources')
|
|
612
|
+
self.assertEqual((status, data['files']['source.txt']['text']), (0, 'evidence'))
|
|
613
|
+
self.assertIn('the text of 1 cited source files is inside the file', printed)
|
|
614
|
+
|
|
615
|
+
def test_build_links_cited_files_to_the_commit_it_was_built_from(self):
|
|
616
|
+
k._history_cache.clear()
|
|
617
|
+
self.vault.rename(self.repo / 'vault')
|
|
618
|
+
self.vault = self.repo / 'vault'
|
|
619
|
+
out = str(self.root / 'site/spec.html')
|
|
620
|
+
self.assertIn('--repo is not a Git work tree with a commit', self.refused('build', '--out', out, '--source-url', 'https://github.com/o/r/blob/{commit}'))
|
|
621
|
+
self.git('init', '-q')
|
|
622
|
+
self.git('add', '-A')
|
|
623
|
+
self.git('commit', '-q', '-m', 'First')
|
|
624
|
+
head = self.git('rev-parse', 'HEAD').strip()
|
|
625
|
+
status, printed, _, data = self.built('--source-url', 'https://github.com/o/r/blob/{commit}/')
|
|
626
|
+
self.assertEqual((status, data['commit'], data['sourceUrl']), (0, head, 'https://github.com/o/r/blob/' + head))
|
|
627
|
+
self.assertIn('commit ' + head[:7], printed)
|
|
628
|
+
self.assertEqual([c['subject'] for c in data['history']], ['First'])
|
|
629
|
+
self.assertNotIn('shallow clone', printed)
|
|
630
|
+
for bad in ('javascript:alert(1)', 'github.com/o/r', 'https://', 'https://x.example/a b'):
|
|
631
|
+
with self.subTest(url=bad):
|
|
632
|
+
self.assertIn('--source-url must be an http(s) address', self.refused('build', '--out', out, '--source-url', bad))
|
|
633
|
+
|
|
634
|
+
def test_build_refuses_a_broken_vault_and_a_file_inside_the_vault(self):
|
|
635
|
+
for out in (self.vault / 'spec.html', self.vault / 'notes/deep/spec.html', self.root):
|
|
636
|
+
with self.subTest(out=out):
|
|
637
|
+
self.assertIn('--out must be a file outside the vault', self.refused('build', '--out', str(out)))
|
|
638
|
+
self.assertFalse((self.vault / 'spec.html').exists() or (self.vault / 'notes/deep').exists())
|
|
639
|
+
for args in (['build'], ['check', '--out', 'x.html'], ['serve', '--with-sources'], ['check', '--source-url', 'https://x.example']):
|
|
640
|
+
with self.subTest(args=args):
|
|
641
|
+
self.assertIn('build', self.refused(*args))
|
|
642
|
+
self.note('notes/bad.md', 'bad', sources=['source.txt'], body='[[<absent>]]')
|
|
643
|
+
status, _, err, data = self.built()
|
|
644
|
+
self.assertEqual((status, data), (1, None))
|
|
645
|
+
self.assertIn('missing wikilink <absent>', err)
|
|
646
|
+
|
|
647
|
+
def test_history_of_a_shallow_clone_leaves_out_the_commit_without_parents(self):
|
|
648
|
+
k._history_cache.clear()
|
|
649
|
+
self.vault.rename(self.repo / 'vault')
|
|
650
|
+
self.vault = self.repo / 'vault'
|
|
651
|
+
self.git('init', '-q')
|
|
652
|
+
self.git('add', '-A')
|
|
653
|
+
self.git('commit', '-q', '-m', 'First')
|
|
654
|
+
self.note('notes/n.md', 'n', sources=['source.txt'], summary='Second line.')
|
|
655
|
+
self.git('commit', '-q', '-am', 'Second')
|
|
656
|
+
self.note('notes/n.md', 'n', sources=['source.txt'], summary='Third line.')
|
|
657
|
+
self.git('commit', '-q', '-am', 'Third')
|
|
658
|
+
clone = self.root / 'clone'
|
|
659
|
+
subprocess.run(['git', 'clone', '-q', '--depth', '2', self.repo.resolve().as_uri(), str(clone)], check=True, capture_output=True)
|
|
660
|
+
history = k.vault_history(clone.resolve(), (clone / 'vault').resolve())
|
|
661
|
+
self.assertEqual([(c['subject'], [p['status'] for p in c['pages']]) for c in history], [('Third', ['changed'])],
|
|
662
|
+
'Second is where the clone stops: Git would show every page as added there')
|
|
663
|
+
self.repo, self.vault = clone, clone / 'vault'
|
|
664
|
+
self.assertIn('build: this is a shallow clone', self.built()[1])
|
|
665
|
+
|
|
586
666
|
def test_serve_finds_the_single_vault_and_asks_when_unsure(self):
|
|
587
667
|
with self.assertRaisesRegex(ValueError, 'no vault found under knowledge/; pass --vault'):
|
|
588
668
|
k.find_vault(self.repo)
|