project-spec 0.7.0 → 0.8.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -88,6 +88,57 @@ alias spec='python3 ~/.claude/skills/project-knowledge/scripts/knowledge.py serv
88
88
 
89
89
  The vault is found by itself when the repository has one under `knowledge/`; otherwise add `--vault PATH`. The page is served on port 8757, or another free port when that one is taken; `--port N` picks one. `project-spec serve` opens the browser unless you add `--no-open`; the helper opens it only with `--open`. If the vault has errors, the page lists them.
90
90
 
91
+ ### Show it on a server
92
+
93
+ `serve` is for your own machine. To let the rest of the team read the spec on a development or staging server, write the viewer as one file and let your application return it:
94
+
95
+ ```sh
96
+ project-spec build --out build/spec.html \
97
+ --source-url 'https://github.com/OWNER/REPO/blob/{commit}'
98
+ ```
99
+
100
+ Without the package the command is `python3 <skill>/scripts/knowledge.py build --repo . --out build/spec.html`. The file is the whole viewer: the pages, the search, the map, the gap queue and the history, with no other files and no network requests. It is a copy of the vault at the moment of the build, so build it again on every deploy.
101
+
102
+ | Option | What it does |
103
+ | --- | --- |
104
+ | `--out FILE` | Where to write the page. It must be outside the vault: nothing is generated into the vault. |
105
+ | `--source-url URL` | Where a cited file can be read. The file's path in the repository is added to this address, and `{commit}` becomes the commit the page was built from. Without it, cited files are shown as names. |
106
+ | `--with-sources` | Put the text of the cited files into the page, as `serve` does, so they open inside the viewer. Leave it out when people who may not read the repository can open the page. |
107
+
108
+ The spec describes how the product works inside, and with `--with-sources` it carries source code. So treat the file like the repository:
109
+
110
+ - Keep it out of folders the web server publishes by itself (`public/`, `static/`). Return it from a route that checks who is asking.
111
+ - Build it only for the environments that should have it, and make the route answer 404 everywhere else.
112
+ - If your application sets a `Content-Security-Policy`, the page needs inline scripts and styles and nothing else.
113
+
114
+ For example, a route in a Next.js application (a sketch: put your own sign-in check where `signedIn()` is):
115
+
116
+ ```ts
117
+ // src/app/development/spec/route.ts
118
+ import { readFile } from "node:fs/promises";
119
+ import path from "node:path";
120
+
121
+ export const dynamic = "force-dynamic";
122
+
123
+ export async function GET() {
124
+ if (process.env.SPEC_PAGE !== "on" || !(await signedIn())) {
125
+ return new Response("Not found", { status: 404 });
126
+ }
127
+ const page = await readFile(path.join(process.cwd(), "build/spec.html"));
128
+ return new Response(page, {
129
+ headers: {
130
+ "Content-Type": "text/html; charset=utf-8",
131
+ "Cache-Control": "no-store",
132
+ "X-Robots-Tag": "noindex",
133
+ "Content-Security-Policy":
134
+ "default-src 'none'; script-src 'unsafe-inline'; style-src 'unsafe-inline'; base-uri 'none'; form-action 'none'",
135
+ },
136
+ });
137
+ }
138
+ ```
139
+
140
+ The build needs Python and Git, so run it where the application is built (a CI job or a build stage), not inside a running container that has neither. The **Recent changes** page comes from the Git history: in a shallow clone it starts at the first commit that was fetched in full, and the command says so. In GitHub Actions, `fetch-depth: 0` on the checkout step gives it the whole history.
141
+
91
142
  ## What a feature page says
92
143
 
93
144
  Every feature, screen or flow has one page, and all of them have the same sections in the same order. A section the feature does not have is left out.
@@ -145,6 +196,7 @@ The package is [`project-spec`](https://www.npmjs.com/package/project-spec) on t
145
196
  ```sh
146
197
  npx project-spec install --claude # or --codex, or --dir knowledge/tooling
147
198
  npx project-spec serve # the viewer for the project in the current folder
199
+ npx project-spec build --out FILE # the same viewer as one file, to show on a server
148
200
  ```
149
201
 
150
202
  Then ask your agent to use the `project-knowledge` skill ([example prompts](#example-prompts)).
@@ -13,7 +13,7 @@ const SKILLS = path.join(ROOT, 'skills');
13
13
  const PACKAGE = JSON.parse(fs.readFileSync(path.join(ROOT, 'package.json'), 'utf8'));
14
14
  const MARKER = '.project-spec.json'; // written into every copy: which version it is and what its files looked like
15
15
  const SKIP = new Set(['__pycache__', '.DS_Store', MARKER]);
16
- const KNOWLEDGE = new Set(['serve', 'check', 'snapshot']);
16
+ const KNOWLEDGE = new Set(['serve', 'build', 'check', 'snapshot']);
17
17
  const MAINTAIN = new Set(['scan', 'receipt', 'finish']);
18
18
  // Skills earlier versions shipped; project-knowledge and spec-change took over what they did.
19
19
  const RETIRED = ['spec-audit', 'spec-investigation', 'spec-plan'];
@@ -25,6 +25,8 @@ const USAGE = `project-spec ${PACKAGE.version}
25
25
  Read the spec
26
26
  project-spec serve [--repo DIR] [--vault DIR] [--port N] [--no-open]
27
27
  Open the viewer for the repository (default: the current folder).
28
+ project-spec build --out FILE [--source-url URL] [--with-sources] [--repo DIR] [--vault DIR]
29
+ Write the viewer as one HTML file, to show the spec on a server.
28
30
  project-spec check | snapshot [options]
29
31
  The knowledge helper; --repo defaults to the current folder.
30
32
  project-spec scan | receipt | finish [options]
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "project-spec",
3
- "version": "0.7.0",
3
+ "version": "0.8.0",
4
4
  "description": "Agent skills that read a repository and keep a spec people can read: one page per feature, every business rule numbered. Local viewer and one command to install and update the skills.",
5
5
  "keywords": [
6
6
  "agent-skills",
@@ -124,7 +124,7 @@ python3 "$SKILL_DIR/scripts/knowledge.py" snapshot --repo "$REPO" --vault "$VAUL
124
124
 
125
125
  `check` validates structure and freshness; it cannot judge prose or coverage. `--strict` turns freshness warnings into a failing exit status, for CI. In a working tree shared with someone else's uncommitted work, `--strict --ignore-dirty` still prints warnings for sources with uncommitted changes but does not fail on them when the accepted fingerprint equals the file at `HEAD` (the only drift is uncommitted); committed drift still fails, and it is not for CI ([details](references/format.md)). `serve` shows the vault in a browser and writes nothing. `snapshot` accepts source fingerprints as reviewed; run it only for pages you actually re-read against their sources, never to clear a warning. An initial reviewed pass may use `--all`.
126
126
 
127
- To read the result, run `python3 <skill>/scripts/knowledge.py serve --repo <repository> --open` (add `--vault` when the repository has several vaults under `knowledge/`), or open the folder as an Obsidian vault. The command is long, so when you hand over offer to give it a short name in the project: a `spec` script in `package.json` (`project-spec serve` when the project has the `project-spec-skills` package, otherwise the full command), a `spec` target in a `Makefile`, or a shell alias. That file is outside the vault, so add it only when the person agrees. The viewer is one page on `127.0.0.1`; it reads the vault and the files the pages cite from disk on every page load, so it needs no build and shows the current state after a reload. It shows a note as written, under its title and summary, with the source files folded away at the end; an area page shows its notes with their summaries. So the first paragraph and the summary are what most readers see. Its **Copy for AI** turns a page and the reader's own words into a request for an agent that names the page file. It only copies text, so an agent that receives one still follows the skill it names, and a recheck or decision the reader states there is recorded under the reader's name, never as the agent's. If the vault has errors the page lists them instead.
127
+ To read the result, run `python3 <skill>/scripts/knowledge.py serve --repo <repository> --open` (add `--vault` when the repository has several vaults under `knowledge/`), or open the folder as an Obsidian vault. The command is long, so when you hand over offer to give it a short name in the project: a `spec` script in `package.json` (`project-spec serve` when the project has the `project-spec-skills` package, otherwise the full command), a `spec` target in a `Makefile`, or a shell alias. That file is outside the vault, so add it only when the person agrees. The viewer is one page on `127.0.0.1`; it reads the vault and the files the pages cite from disk on every page load, so it needs no build and shows the current state after a reload. It shows a note as written, under its title and summary, with the source files folded away at the end; an area page shows its notes with their summaries. So the first paragraph and the summary are what most readers see. Its **Copy for AI** turns a page and the reader's own words into a request for an agent that names the page file. It only copies text, so an agent that receives one still follows the skill it names, and a recheck or decision the reader states there is recorded under the reader's name, never as the agent's. If the vault has errors the page lists them instead. When the person wants the spec on a server for people without the repository, `knowledge.py build --repo <repository> --out <file outside the vault>` writes that same viewer as one file ([options](references/format.md)); run it only when asked, never commit the file into the vault, and leave `--with-sources` out unless the person says everyone who can open the page may read the code.
128
128
 
129
129
  Before handing over, read what you wrote as the person who will use it, not as its author. Open one area page and two of the notes you wrote or changed, and ask:
130
130
 
@@ -1558,8 +1558,9 @@
1558
1558
  (n) => n && typeof n === "object",
1559
1559
  ),
1560
1560
  edges = Array.isArray(data.edges) ? data.edges : [];
1561
- // Sent by `knowledge.py serve` only: the text of cited repository files, the vault's place in the
1562
- // repository, its Git history and its uncommitted pages. A page without them still works.
1561
+ // Sent by `knowledge.py serve` and `build`: the text of cited repository files (build leaves it out
1562
+ // unless asked), the vault's place in the repository, its Git history and its uncommitted pages.
1563
+ // A page without them still works.
1563
1564
  const files =
1564
1565
  data.files && typeof data.files === "object" ? data.files : {},
1565
1566
  vaultPath = typeof data.vaultPath === "string" ? data.vaultPath : "",
@@ -1570,6 +1571,15 @@
1570
1571
  (p) => p && typeof p === "object",
1571
1572
  );
1572
1573
  const live = data.live === true;
1574
+ // A page written by `knowledge.py build` to be put on a server: a copy of one moment, read by people
1575
+ // who may not have the repository. `sourceUrl` is where a cited file can be read instead.
1576
+ const built = data.built === true,
1577
+ builtCommit = typeof data.commit === "string" ? data.commit : "",
1578
+ builtOn = typeof data.generatedAt === "string" ? data.generatedAt.slice(0, 10) : "",
1579
+ sourceUrl =
1580
+ built && typeof data.sourceUrl === "string" && /^https?:\/\/[^\s/]/i.test(data.sourceUrl)
1581
+ ? data.sourceUrl.replace(/\/+$/, "")
1582
+ : "";
1573
1583
  const byId = new Map(nodes.map((n) => [n.id, n])),
1574
1584
  byPath = new Map(
1575
1585
  nodes.map((n) => [String(n.path || "").replace(/\.md$/i, ""), n]),
@@ -1694,7 +1704,13 @@
1694
1704
  const areas = nodes.filter((n) => n.type === "area").sort(byTitle);
1695
1705
  $("#project-title").textContent = projectTitle;
1696
1706
  $("#project-title").title = projectTitle;
1697
- $("#header-note").textContent = live ? "Spec · live from disk" : "Spec";
1707
+ $("#header-note").textContent = live
1708
+ ? "Spec · live from disk"
1709
+ : built
1710
+ ? "Spec" + (builtOn ? " · built " + builtOn : "")
1711
+ : "Spec";
1712
+ // The line is too narrow for the commit as well, so it is in the tooltip.
1713
+ if (built && builtCommit) $("#header-note").title = "Built from commit " + builtCommit;
1698
1714
  // Small things kept in this browser only: where you were, what you picked. Never required.
1699
1715
  const store = {
1700
1716
  key: (name) => "spec-viewer:" + projectTitle + ":" + name,
@@ -1834,8 +1850,9 @@
1834
1850
  return out.join("/");
1835
1851
  }
1836
1852
  const hasFile = (key) => Object.prototype.hasOwnProperty.call(files, key);
1837
- // A relative Markdown link to a cited repository file: where it is in `files` and which lines it names.
1838
- function fileRef(raw, n) {
1853
+ // A relative Markdown link read as a place in the repository: the file and which lines it names.
1854
+ // Null for an address, an anchor or a path that leaves the repository.
1855
+ function repoTarget(raw, n) {
1839
1856
  let t = String(raw).trim();
1840
1857
  if (t.startsWith("<") && t.endsWith(">")) t = t.slice(1, -1);
1841
1858
  if (!t || t[0] === "#" || t[0] === "/" || /^[A-Za-z][A-Za-z0-9+.-]*:/.test(t))
@@ -1849,10 +1866,27 @@
1849
1866
  const dir = String(n.path || "").split("/").slice(0, -1).join("/");
1850
1867
  // Vault pages link relative to themselves inside the vault; a cited document links relative to itself in the repository.
1851
1868
  const key = normPath(n.repoFile ? [dir, t] : [vaultPath, dir, t]);
1852
- if (!hasFile(key)) return null;
1869
+ if (!key || key.split("/")[0] === "..") return null;
1853
1870
  const m = hash.match(/^L(\d+)(?:-L?(\d+))?$/i);
1854
1871
  return { key, from: m ? +m[1] : 0, to: m ? +(m[2] || m[1]) : 0 };
1855
1872
  }
1873
+ // The same link when the text of that file came with the page, so it can be opened in place.
1874
+ function fileRef(raw, n) {
1875
+ const ref = repoTarget(raw, n);
1876
+ return ref && hasFile(ref.key) ? ref : null;
1877
+ }
1878
+ // A cited file on the code host a built page was given: a new tab, since the reader leaves the spec.
1879
+ function sourceLink(ref, text) {
1880
+ const a = el("a", text);
1881
+ a.href =
1882
+ sourceUrl +
1883
+ "/" +
1884
+ ref.key.split("/").map(encodeURIComponent).join("/") +
1885
+ (ref.from ? "#L" + ref.from + (ref.to > ref.from ? "-L" + ref.to : "") : "");
1886
+ a.target = "_blank";
1887
+ a.rel = "noopener noreferrer";
1888
+ return a;
1889
+ }
1856
1890
  const isMarkdownFile = (key) => /\.(md|markdown)$/i.test(key);
1857
1891
  let fileState = null;
1858
1892
  function fileButton(ref, text) {
@@ -1888,7 +1922,7 @@
1888
1922
  ? "Lines " + ref.from + "–" + ref.to
1889
1923
  : "Line " + ref.from) + " · "
1890
1924
  : "") +
1891
- (live ? "Read from disk when this page loaded" : "Copy taken when this map was built") +
1925
+ (live ? "Read from disk when this page loaded" : "Copy taken when this page was built") +
1892
1926
  (file.truncated ? " · only the beginning of the file is shown" : "");
1893
1927
  actions.replaceChildren();
1894
1928
  if (isMarkdownFile(ref.key))
@@ -1957,9 +1991,16 @@
1957
1991
  else {
1958
1992
  const parts = token.match(/^\[([^\]]+)\]\(([^)]+)\)$/),
1959
1993
  ref = parts && fileRef(parts[2], n),
1960
- href = parts && !ref && !n.repoFile && safeHref(parts[2], n);
1994
+ // On a built page a relative link would lead into the site that hosts it, not to the file.
1995
+ away = parts && !ref && built && !n.repoFile ? repoTarget(parts[2], n) : null,
1996
+ href = parts && !ref && !away && !n.repoFile && safeHref(parts[2], n);
1961
1997
  if (ref) parent.append(fileButton(ref, parts[1]));
1962
- else if (href) {
1998
+ else if (away && sourceUrl) parent.append(sourceLink(away, parts[1]));
1999
+ else if (away) {
2000
+ const name = el("span", parts[1]);
2001
+ name.title = away.key;
2002
+ parent.append(name);
2003
+ } else if (href) {
1963
2004
  const a = el("a", parts[1]);
1964
2005
  a.href = href;
1965
2006
  parent.append(a);
@@ -2464,7 +2505,8 @@
2464
2505
  };
2465
2506
  const sourceLine = (src) => {
2466
2507
  const key = normPath([src]);
2467
- return hasFile(key) ? fileButton({ key, from: 0, to: 0 }, src) : el("span", src);
2508
+ const ref = { key, from: 0, to: 0 };
2509
+ return hasFile(key) ? fileButton(ref, src) : sourceUrl && key ? sourceLink(ref, src) : el("span", src);
2468
2510
  };
2469
2511
 
2470
2512
  // --- Views ---------------------------------------------------------------------------------------
@@ -2891,7 +2933,7 @@
2891
2933
  "p",
2892
2934
  historyLog.length
2893
2935
  ? "Nothing in the history matches this filter."
2894
- : live
2936
+ : live || built
2895
2937
  ? "No Git history was found for this vault."
2896
2938
  : "History is shown when the spec is opened with the serve command.",
2897
2939
  "empty",
@@ -166,7 +166,7 @@ The validator enforces these body rules:
166
166
 
167
167
  Three scripts in `$SKILL_DIR/scripts/`; see [SKILL.md](../SKILL.md) for the variables.
168
168
 
169
- `knowledge.py {check,snapshot,serve} --repo PATH [--vault PATH]`
169
+ `knowledge.py {check,snapshot,serve,build} --repo PATH [--vault PATH]`
170
170
 
171
171
  - `check`: read-only, exit 1 for structural errors; freshness warnings normally leave exit status 0. Add `--strict` to also fail on warnings. Page-level errors are printed first, and a failing `check` or `snapshot` ends with a line giving the number of errors and the number of pages they are on. A page whose frontmatter fails validation still counts as a target for links, embeds, area membership, a plan's `gap` and `duplicate_of`, so one broken page yields its own errors only; fix it and nothing else needs touching. Only the "missing" error is hidden this way: an ambiguous link (two or more candidates), a link to a page of the wrong type (a spec page linking a gap, or an `areas` entry naming a page known to be another type) and a run page's link are still reported. `snapshot --notes X` refuses X while X's area references, links, `gap` or `duplicate_of` point at a page that failed validation. Missing source files are errors except on explicitly `stale` or `superseded` notes (warnings). Changed or unbaselined evidence warns. A source that differs from its baseline only by line endings (typically a Windows checkout) warns "line endings differ in source X"; do not snapshot on that checkout, because it would record the converted bytes. Nothing is rewritten.
172
172
 
@@ -204,6 +204,8 @@ Canonical Markdown must not contain raw merge conflicts. Validation rejects comp
204
204
 
205
205
  `serve` shows the vault in a browser: `knowledge.py serve --repo REPO [--vault VAULT] [--port N] [--open]`. Without `--vault`, this and every other command use the single vault under `knowledge/` (`knowledge/00-index.md` or `knowledge/*/00-index.md`) and stop with a message when there is none or several. It listens on `127.0.0.1` only (port 8757, or a free one when that is taken), answers only `/` and only when addressed as this machine, never writes, and builds the page again from disk on every request, so a reload shows the current files. When the vault has errors the page lists them. The page data is the graph above plus `live: true`, `vaultPath` (the vault's path relative to the repository), `history`, `pending` and `files`. `history` is the last 30 commits that touched vault pages (not `.knowledge/`), newest first, each `{"sha", "date", "subject", "pages", "more"}`; a page entry is `{"path", "status"}` with `status` `added`, `changed` or `removed`, plus `was` and `now` when the commit changed the page's `summary` (or, failing that, its `title`); at most 80 pages are listed per commit and `more` counts the rest. `pending` lists vault pages with uncommitted changes in the same `{"path", "status"}` shape. Both are empty outside a Git work tree. `files` is the text of the repository files that pages cite in `sources` or link to, keyed by repository-relative path, each `{"text", "truncated"}`. A file is included only when it is inside the repository and outside the vault, is not a credential path, and is UTF-8 text; one file is cut at 300,000 characters and all files together at 4,000,000, after which the rest stay ordinary links. The viewer opens a link to an included file in place: Markdown rendered, anything else as numbered lines, with the lines of a `#L10` or `#L10-L20` fragment marked.
206
206
 
207
+ `build` writes the same viewer as one file, for a server that shows the spec to people who do not have the repository: `knowledge.py build --repo REPO [--vault VAULT] --out FILE [--source-url URL] [--with-sources]`. `--out` must be a file outside the vault. The command stops with the `check` errors when the vault has any, and writes nothing then; freshness warnings do not stop it. The page data is the graph plus `built: true`, `commit` (the full `HEAD`, empty outside Git), `sourceUrl`, `vaultPath`, `history`, `pending` and `files`. `files` is empty unless `--with-sources` is given, because the text of cited files is source code. `--source-url` must be an `http(s)` address; `{commit}` in it is replaced by `HEAD`, and it is an error to name `{commit}` outside a Git work tree. On a built page a cited file whose text is not included is a link to `sourceUrl/<repository path>` (with `#L10-L20` when lines are named) in a new tab, or plain text when there is no `sourceUrl`; it is never a relative link, which would lead into the site that hosts the page. In a shallow clone the commit whose parents were not fetched is left out of `history` (by `serve` too), since Git would show every page as added there, and `build` says that the history is short.
208
+
207
209
  `assets/viewer.html` has exactly one `__KNOWLEDGE_DATA__` placeholder for script-safe JSON. Treat contents as untrusted text: no raw Markdown HTML execution, unsafe URL schemes, external scripts, fonts, analytics or fetches. Render readable basic Markdown, pipe tables and wikilinks; leave unsupported syntax as text. The viewer has a tree of areas with their notes (a note is listed under its first area; superseded notes are not listed), a search over pages, gaps and the cited files, and five places: the overview, a page, the gap queue, the map and recent changes. A note shows its title, `summary`, the page text and the gaps about it that are not closed, then its source files folded; an area shows its notes with their summaries and folds its own text. The pages linked to or from the page are a short list beside it. Navigation keeps a trail for Back and Forward and writes the place into the address (`#/page/notes/example`, `#/gaps`), so a reload or a copied link returns to it. The map draws one frame per primary area (the area page, or the first entry of `areas`) with its pages inside; an edge between a page and its primary area is not drawn because the frame says it, links inside one area are solid and links across areas dashed; `nav` pages are not drawn. Recent changes reads `history`, `pending` and the "changed source" lines of `warnings`. "Copy for AI" only composes text from the page, a choice of what the agent should do and the reader's own words, and copies it to the clipboard. The reader's name, the recently viewed pages and the last commit seen are kept in the browser's `localStorage`, and the viewer works without it.
208
210
 
209
211
  ## Format references
@@ -689,6 +689,17 @@ def front_value(line):
689
689
  return value if isinstance(value, str) else ''
690
690
 
691
691
 
692
+ def shallow_commits(repo):
693
+ """Commits whose parents a shallow clone left out. Empty in a full clone and outside Git."""
694
+ found = subprocess.run(['git', '-C', str(repo), 'rev-parse', '--git-path', 'shallow'], capture_output=True)
695
+ if found.returncode:
696
+ return set()
697
+ try:
698
+ return set((repo / os.fsdecode(found.stdout).strip()).read_text(encoding='ascii').split())
699
+ except (OSError, UnicodeDecodeError):
700
+ return set()
701
+
702
+
692
703
  def vault_history(repo, vault):
693
704
  """Recent commits that touched vault pages, newest first: sha, date, subject and what happened to each page.
694
705
 
@@ -717,10 +728,12 @@ def vault_history(repo, vault):
717
728
  return []
718
729
  if log.returncode:
719
730
  return []
720
- commits = []
731
+ commits, boundary = [], shallow_commits(repo)
721
732
  for chunk in log.stdout.decode('utf-8', errors='replace').split(marker)[1:]:
722
733
  header, _, body = chunk.partition('\n')
723
734
  sha, date, subject = (header.split('\x02', 2) + ['', ''])[:3]
735
+ if sha in boundary: # its parents were not fetched, so Git would show every page as added
736
+ continue
724
737
  subject = CONTROL.sub('', subject)
725
738
  pages, page = [], None
726
739
  for line in body.split('\n'):
@@ -772,6 +785,13 @@ def vault_pending(repo, vault):
772
785
  return sorted(pending, key=lambda p: p['path'])
773
786
 
774
787
 
788
+ def page_payload(repo, vault, notes, edges, warnings, sources=True):
789
+ files = cited_files(repo, vault, notes)[0] if sources else {}
790
+ return graph_of(notes, edges, warnings) | {
791
+ 'vaultPath': Path(os.path.relpath(vault, repo)).as_posix(), 'files': files,
792
+ 'history': vault_history(repo, vault), 'pending': vault_pending(repo, vault)}
793
+
794
+
775
795
  def live_page(repo, vault):
776
796
  """The viewer for the vault as it is on disk right now, or a page listing what stops it from loading."""
777
797
  try:
@@ -785,10 +805,29 @@ def live_page(repo, vault):
785
805
  f'<h1>The spec has {len(errors)} error{"s" if len(errors) != 1 else ""}</h1>'
786
806
  '<p>Fix the pages below (the <code>check</code> command prints the same list), then reload.</p>'
787
807
  f'<ul>{items}</ul></body>')
788
- files, _ = cited_files(repo, vault, notes)
789
- return viewer_html(graph_of(notes, edges, warnings) | {
790
- 'live': True, 'vaultPath': Path(os.path.relpath(vault, repo)).as_posix(), 'files': files,
791
- 'history': vault_history(repo, vault), 'pending': vault_pending(repo, vault)})
808
+ return viewer_html(page_payload(repo, vault, notes, edges, warnings) | {'live': True})
809
+
810
+
811
+ def source_base(value, commit):
812
+ """The address `--source-url` names, with `{commit}` filled in. Raises ValueError with the reason when it cannot be used."""
813
+ if CONTROL.search(value) or ' ' in value or not re.match(r'https?://[^/]', value, re.I):
814
+ raise ValueError('--source-url must be an http(s) address, e.g. https://github.com/OWNER/REPO/blob/{commit}')
815
+ if '{commit}' in value and not commit:
816
+ raise ValueError('--source-url names {commit}, but --repo is not a Git work tree with a commit')
817
+ return value.replace('{commit}', commit).rstrip('/')
818
+
819
+
820
+ def built_page(repo, vault, notes, edges, warnings, with_sources=False, source_url=''):
821
+ """The viewer as one file to put on a server: the vault as it is now, with nothing read later.
822
+
823
+ The text of cited repository files is left out unless asked for, because the file may be shown to people
824
+ who cannot read the repository. Returns (html, payload).
825
+ """
826
+ head = subprocess.run(['git', '-C', str(repo), 'rev-parse', 'HEAD'], capture_output=True)
827
+ commit = '' if head.returncode else head.stdout.decode('ascii', errors='replace').strip()
828
+ payload = page_payload(repo, vault, notes, edges, warnings, with_sources) | {
829
+ 'built': True, 'commit': commit, 'sourceUrl': source_base(source_url, commit) if source_url else ''}
830
+ return viewer_html(payload), payload
792
831
 
793
832
 
794
833
  def make_server(repo, vault, port):
@@ -820,12 +859,15 @@ def make_server(repo, vault, port):
820
859
 
821
860
  def main(argv=None):
822
861
  parser = argparse.ArgumentParser()
823
- parser.add_argument('command', choices=['check', 'snapshot', 'serve'])
862
+ parser.add_argument('command', choices=['check', 'snapshot', 'serve', 'build'])
824
863
  parser.add_argument('--repo', required=True, type=Path)
825
864
  parser.add_argument('--vault', type=Path, help='default: the single vault under knowledge/ in the repository')
826
865
  parser.add_argument('--strict', action='store_true', help='fail check when freshness warnings exist')
827
866
  parser.add_argument('--port', type=int, default=8757, help='serve: port on 127.0.0.1 (another free port is used if it is taken)')
828
867
  parser.add_argument('--open', action='store_true', help='serve: open the page in the default browser')
868
+ parser.add_argument('--out', type=Path, help='build: the HTML file to write, outside the vault')
869
+ parser.add_argument('--with-sources', action='store_true', help='build: include the text of the repository files the pages cite')
870
+ parser.add_argument('--source-url', default='', help='build: where a cited file can be read, e.g. https://github.com/OWNER/REPO/blob/{commit}')
829
871
  parser.add_argument('--ignore-dirty', action='store_true',
830
872
  help='check: do not count freshness warnings for sources with uncommitted changes (needs a Git work tree)')
831
873
  group = parser.add_mutually_exclusive_group()
@@ -838,6 +880,10 @@ def main(argv=None):
838
880
  parser.error('--ignore-dirty is only for check')
839
881
  if args.open and args.command != 'serve':
840
882
  parser.error('--open is only for serve')
883
+ if args.command == 'build' and not args.out:
884
+ parser.error('build requires --out FILE')
885
+ if args.command != 'build' and (args.out or args.with_sources or args.source_url):
886
+ parser.error('--out, --with-sources and --source-url are only for build')
841
887
  if args.command == 'snapshot' and not (args.all or args.notes):
842
888
  parser.error('snapshot requires --all or --notes')
843
889
  if args.command != 'snapshot' and (args.all or args.notes):
@@ -904,6 +950,23 @@ def main(argv=None):
904
950
  return 1
905
951
  if ignored:
906
952
  print(f'{args.command}: {len(ignored)} errors on {len({error_page(m) for m in ignored if error_page(m)})} other pages do not block this snapshot (run check to see them)', file=sys.stderr)
953
+ if args.command == 'build':
954
+ out = Path(os.path.abspath(args.out))
955
+ if out.is_dir() or out.resolve().is_relative_to(vault):
956
+ parser.error('--out must be a file outside the vault: nothing is generated into the vault')
957
+ try:
958
+ page, payload = built_page(repo, vault, notes, edges, warnings, args.with_sources, args.source_url)
959
+ except ValueError as exc:
960
+ parser.error(str(exc))
961
+ atomic_write(out, page)
962
+ files = len(payload['files'])
963
+ print(f'build: wrote {out} ({len(page.encode("utf-8")) // 1024} kB), {len(notes)} notes, {len(warnings)} warnings'
964
+ + (f', commit {payload["commit"][:7]}' if payload['commit'] else ''))
965
+ print('build: ' + (f'the text of {files} cited source files is inside the file; show it only to people who may read the repository'
966
+ if args.with_sources else 'cited source files are named but their text is not included (--with-sources adds it)'))
967
+ if shallow_commits(repo):
968
+ print('build: this is a shallow clone, so the Changes page starts at the first commit that was fetched in full; fetch the whole history to show more')
969
+ return 0
907
970
  if args.command == 'snapshot':
908
971
  for note in notes:
909
972
  meta = note['meta']
@@ -583,6 +583,86 @@ class GapLayerTests(Fixture):
583
583
  self.assertEqual(k.vault_pending(inner.resolve(), (inner / 'vault').resolve()), [])
584
584
  k._history_cache.clear()
585
585
 
586
+ def built(self, *args):
587
+ out = self.root / 'site/spec.html'
588
+ status, printed, err = self.run_cli('build', '--out', str(out), *args)
589
+ data = json.loads(re.search(r'<script id="knowledge-data" type="application/json">\s*(.*?)\s*</script>', out.read_text(encoding='utf-8'), re.S).group(1)) if out.is_file() else None
590
+ return status, printed, err, data
591
+
592
+ def refused(self, *args):
593
+ """What a command that stops with a usage error says."""
594
+ err = StringIO()
595
+ with self.assertRaises(SystemExit), redirect_stdout(StringIO()), redirect_stderr(err):
596
+ k.main([*args, '--repo', str(self.repo), '--vault', str(self.vault)])
597
+ return err.getvalue()
598
+
599
+ def test_build_writes_one_page_for_a_server_without_source_text(self):
600
+ status, printed, err, data = self.built()
601
+ self.assertEqual((status, err), (0, 'WARNING: notes/n.md: unbaselined source source.txt\n'))
602
+ self.assertIn('build: wrote ' + str(self.root / 'site/spec.html'), printed)
603
+ self.assertIn('their text is not included', printed)
604
+ self.assertEqual((data['built'], data.get('live'), data['files'], data['sourceUrl'], data['commit']), (True, None, {}, '', ''))
605
+ self.assertEqual({n['id'] for n in data['nodes']}, {'a', 'index', 'n'})
606
+ self.assertNotIn('evidence', (self.root / 'site/spec.html').read_text(encoding='utf-8'))
607
+ # nothing appeared in the vault, and no temporary file was left beside the page
608
+ self.assertFalse((self.vault / '.knowledge').exists())
609
+ self.assertEqual([f.name for f in (self.root / 'site').iterdir()], ['spec.html'])
610
+ # the text of cited files only when asked for, with a warning about who may see it
611
+ status, printed, _, data = self.built('--with-sources')
612
+ self.assertEqual((status, data['files']['source.txt']['text']), (0, 'evidence'))
613
+ self.assertIn('the text of 1 cited source files is inside the file', printed)
614
+
615
+ def test_build_links_cited_files_to_the_commit_it_was_built_from(self):
616
+ k._history_cache.clear()
617
+ self.vault.rename(self.repo / 'vault')
618
+ self.vault = self.repo / 'vault'
619
+ out = str(self.root / 'site/spec.html')
620
+ self.assertIn('--repo is not a Git work tree with a commit', self.refused('build', '--out', out, '--source-url', 'https://github.com/o/r/blob/{commit}'))
621
+ self.git('init', '-q')
622
+ self.git('add', '-A')
623
+ self.git('commit', '-q', '-m', 'First')
624
+ head = self.git('rev-parse', 'HEAD').strip()
625
+ status, printed, _, data = self.built('--source-url', 'https://github.com/o/r/blob/{commit}/')
626
+ self.assertEqual((status, data['commit'], data['sourceUrl']), (0, head, 'https://github.com/o/r/blob/' + head))
627
+ self.assertIn('commit ' + head[:7], printed)
628
+ self.assertEqual([c['subject'] for c in data['history']], ['First'])
629
+ self.assertNotIn('shallow clone', printed)
630
+ for bad in ('javascript:alert(1)', 'github.com/o/r', 'https://', 'https://x.example/a b'):
631
+ with self.subTest(url=bad):
632
+ self.assertIn('--source-url must be an http(s) address', self.refused('build', '--out', out, '--source-url', bad))
633
+
634
+ def test_build_refuses_a_broken_vault_and_a_file_inside_the_vault(self):
635
+ for out in (self.vault / 'spec.html', self.vault / 'notes/deep/spec.html', self.root):
636
+ with self.subTest(out=out):
637
+ self.assertIn('--out must be a file outside the vault', self.refused('build', '--out', str(out)))
638
+ self.assertFalse((self.vault / 'spec.html').exists() or (self.vault / 'notes/deep').exists())
639
+ for args in (['build'], ['check', '--out', 'x.html'], ['serve', '--with-sources'], ['check', '--source-url', 'https://x.example']):
640
+ with self.subTest(args=args):
641
+ self.assertIn('build', self.refused(*args))
642
+ self.note('notes/bad.md', 'bad', sources=['source.txt'], body='[[<absent>]]')
643
+ status, _, err, data = self.built()
644
+ self.assertEqual((status, data), (1, None))
645
+ self.assertIn('missing wikilink <absent>', err)
646
+
647
+ def test_history_of_a_shallow_clone_leaves_out_the_commit_without_parents(self):
648
+ k._history_cache.clear()
649
+ self.vault.rename(self.repo / 'vault')
650
+ self.vault = self.repo / 'vault'
651
+ self.git('init', '-q')
652
+ self.git('add', '-A')
653
+ self.git('commit', '-q', '-m', 'First')
654
+ self.note('notes/n.md', 'n', sources=['source.txt'], summary='Second line.')
655
+ self.git('commit', '-q', '-am', 'Second')
656
+ self.note('notes/n.md', 'n', sources=['source.txt'], summary='Third line.')
657
+ self.git('commit', '-q', '-am', 'Third')
658
+ clone = self.root / 'clone'
659
+ subprocess.run(['git', 'clone', '-q', '--depth', '2', self.repo.resolve().as_uri(), str(clone)], check=True, capture_output=True)
660
+ history = k.vault_history(clone.resolve(), (clone / 'vault').resolve())
661
+ self.assertEqual([(c['subject'], [p['status'] for p in c['pages']]) for c in history], [('Third', ['changed'])],
662
+ 'Second is where the clone stops: Git would show every page as added there')
663
+ self.repo, self.vault = clone, clone / 'vault'
664
+ self.assertIn('build: this is a shallow clone', self.built()[1])
665
+
586
666
  def test_serve_finds_the_single_vault_and_asks_when_unsure(self):
587
667
  with self.assertRaisesRegex(ValueError, 'no vault found under knowledge/; pass --vault'):
588
668
  k.find_vault(self.repo)