@danielsimonjr/repo-tools 2.0.0 → 2.2.0

package/CHANGELOG.md

@@ -6,6 +6,193 @@ All notable changes to this project are recorded in this file. The format follow
 
 ## [Unreleased]
 
+## [2.2.0] - 2026-10-02
+
+`repo-tools docs` is the new gate of the doc comments of a repository. It ports `code_docs.py` of
+the code-docs skill. `docs/parity-docs.md` lists the differences from the Python tool. The
+release also adds the audit of the fix ledger and the check for dangling references to the
+design document.
+
+### Added
+
+- `repo-tools docs`, the gate of the doc comments. The command is a port of `code_docs.py` of the
+  code-docs skill, and it needs no Python. `docs scan` measures the coverage of the exported
+  symbols and writes `COVERAGE.md` and `coverage.json`. `docs check` is the gate: it exits 1 on a
+  MUST issue (M1 to M5) and on a file that does not parse. `--paths` and `--paths-from=<file>`
+  gate the changed files of a pull request only. `docs stub` plans a skeleton comment with the
+  marker `TODO:` for each exported symbol that has no comment. The default is a dry run until
+  `--apply`. After a change, `stub` parses the file again, and it keeps the line ends of the
+  file. The command reads TypeScript, JavaScript and Python with tree-sitter. A
+  `.code-docs.json` file at the root excludes paths, and each entry needs a reason. Section 16
+  of `docs/design.md` describes the command. `docs/parity-docs.md` compares the command with the
+  Python tool on 9 public repositories: 4665 files, 22329 symbols, 690 MUST issues and 11208
+  SHOULD issues. The reports are equal, except for the order of the symbols in 40 files and the
+  doc names of 29 symbols of one repository. The command gives 3 MUST issues more than the Python
+  tool, for stale names in optional `[name]` tags. The record lists the differences. Some are
+  defects of the Python tool. The defects are a closing `*/` in the summary, comment delimiters in
+  the STE check, and the parameter names in a doc. The others are the parser of Python, the exit
+  codes, and the files that `stub` changes.
+- `repo-tools docs stub` never writes through a link. The command skips a file that is a link or
+  resolves outside the root, and names the file. The command does not write a TypeScript block
+  when another token precedes the declaration on its line, for example the end of a template. The
+  end of a Python signature ignores the brackets in strings and a comment after the colon. A root
+  with no source file gives exit code 1, as for `scan` and `check`. `--path=.` selects the whole
+  repository.
+- `docs check --paths-from` reads the quoted names that `git diff --name-only` writes for a
+  non-ASCII file name. The analysers read a JSDoc parameter name with `$`, and the optional forms
+  `[name]` and `[name=default]`. They read a reST field with a type, for example
+  `:param int x:`. A destructured parameter binds the names on the left of a default value. The
+  analysers read each name from the source as written, so a parameter named `abstract` keeps its
+  name. The Python tool has each of these defects, and design section 16.6 lists them.
+- The smoke test has a step for the doc gate. A run of `docs check` reads a TypeScript, a TSX and
+  a Python file, so one run loads the three grammars of the product. The test passes documented
+  source and fails a new undocumented export.
+- `bun run audit:ledger`, the audit of the fix ledger. For each fix (F1 to F44, M1 and R1), the
+  script restores the 1.x defect in a scratch copy of the repository. Then it runs the tests.
+  It runs the test file of the fix first, and then the whole suite. A test of the fix file must
+  fail. Each test has a 30 s limit (`--test-timeout-s`). A run in which only tests hit their
+  time limit is an error, because load can cause it.
+  The script exits 0 only when this holds for every mutation. Section 12 of
+  `docs/design.md` describes the audit. `scripts/fix-ledger-mutations.ts` holds the 77
+  mutations. `docs/fix-ledger-audit.md` holds the result: 77 of 77 mutations fail a test of
+  their own fix. The tests of the audit make their temporary repositories with a git identity
+  that the privacy check accepts.
+- Tests for the fixes that no test of their own locked. F12: the module map of
+  `dependency-layers.json`. F26: components sorted by smallest member. F30: the map resolver
+  picks a `.tsx` file and a directory index. F34: a workspace package folder that is a link.
+  F43: the `src/<sub>/index.ts` fallback of an `exports` subpath. The tests of F17, F41 and F43
+  assert more. They assert the `src/index.ts` fallback root, an excluded package that has an
+  entry, and the edges of a self-import.
+- `bun run check:design`, the check for dangling references to `docs/design.md`. A comment, a
+  test title or a workflow can name a part of the design, for example `design 11.4`. The script
+  finds each such reference and looks for a heading with that number. It prints each reference
+  that has no heading, and it exits 1. The unit tests run it on this repository. The check
+  proves that the number exists. It cannot prove that the heading has the meaning that the
+  reference gives.
+
+### Changed
+
+- The grammar loader reads a third grammar, TSX. The Node bundle and the npm package now hold four
+  `.wasm` files and not three, and the compiled executable holds the TSX grammar too. `docs` uses
+  it for `.tsx` and `.jsx` files. `map` still reads these files with the TypeScript grammar.
+- The directory readers of the map engine read through `readdirHook`. The readers are discovery,
+  the two `.csproj` walks and the folder list of a workspace pattern. The F2 test replaces
+  `readdirHook`. Before, the map never called the reader that the F2 test reversed. The test
+  passed with or without the sort in discovery.
+- `docs/fix-ledger-2.0.0.md` describes what each test locks. The rows for F9 to F12, F19 to F21,
+  F23, F24, F27 and F29 to F31 described other behavior. The rows for F13, F36, F41 and F44 sat
+  in the section "No 2.0.0 meaning", although tests lock them. The test for M1 said that
+  `--reachable-only` removes a file. It asserts that `map` rejects the flag.
+- `docs/design.md` now holds the parts that the code names. Section 11 has the subsections 11.1
+  to 11.4: the Node bundle, the compiled executables, the workflows and the release. Section 12
+  has 12.1, the steps of the smoke test. Section 14.2 describes the API-surface flags.
+- The check found 50 references in comments, test titles and workflows to a part that the
+  document never had. Each reference now names an existing section. Four more references named
+  section 4 for the chunk and compress modules. Section 4 holds the output rules, so they now
+  name section 8.
+
+### Maintenance
+
+- The root `bun` Dependabot entry is removed from `.github/dependabot.yml`. It cannot succeed.
+  The bun updater of Dependabot reads `bun.lock` lockfileVersion 1, and Bun 1.4 writes 2. The
+  `npm` alternative aborts during file fetching on a bun-managed project. Each error recommends
+  the other. Measured here: the `bun in /.` updater job failed on 2026-09-24 and again on
+  2026-10-01, while `github_actions in /.` succeeded on both days. `github-actions` updates are
+  unchanged. The file now records the two errors, the cost and a checkable revisit condition.
+  **Unlike the sibling repositories in this sweep, this one also has Dependabot alerts
+  disabled.** Thus nothing reports a vulnerable dependency either. A bump is entirely a human
+  action: `bun outdated`, `bun update <pkg>`, then commit the regenerated `bun.lock`.
+
+## [2.1.0] - 2026-10-01
+
+`repo-tools check` is the new drift gate for the Markdown documents of a repository. It ports
+`check.py` of the architecture-docs skill. The release also makes a literal `import()` a graph
+edge, removes the 1.x depgraph pipeline, and adds an accept-list to the privacy check.
+`docs/parity-check.md` lists the differences from the Python tool.
+
+### Added
+
+- `repo-tools check --docs=<dir>`, the drift gate of the architecture documents. The command is
+  a port of `check.py` of the architecture-docs skill. The check reads the `## Verification`
+  tables of the Markdown files in the folder. The check builds a fresh graph, and the check
+  compares each claim with the graph. Exit 0 means that every claim matched. Exit 1 lists every
+  problem on standard error, one per line. A missing or empty docs folder is a problem. A
+  document without a Verification section is a problem. So are a section without a row, an
+  unknown claim, a count that the graph build declared unreliable, and a value that differs. A
+  document opts out when one of its lines is exactly `<!-- repo-map:no-verification -->`. A relative `--docs` path is
+  relative to the root, and an absolute path is used as given. The check reads
+  `map.duplicateAllowlist` and `map.verificationMarker` from the config file. Section 15 of
+  `docs/design.md` lists the differences from the Python tool. The smoke test has a `check` step.
+  `docs/parity-check.md` records the comparison with the Python tool on 11 repositories.
+- The privacy check has an accept-list, `scripts/privacy-accepted.txt`, for a reviewed finding in
+  the commit history. A commit message cannot change after a merge, so a new commit cannot clear
+  a finding in it. An entry holds a full commit sha, one finding kind, one line of the message and
+  a reason. The entry accepts that finding only: no prefix, no wildcard, no whole-commit waiver,
+  and never a finding in a tracked file. An entry that accepts no finding is a finding
+  (`accepted-stale`). The check prints each accepted finding with its reason.
+
+### Changed
+
+- A literal relative `import()` is an edge of the core graph. This reverses the known difference
+  in 2.0.0. A runtime call is a namespace use (`*`). A type-position `import('./c').Name` records
+  that name. `typeof import()` records no names. A template that contains `${`, and a specifier
+  that is not relative, are not edges. The parity record gives the merge rules. The verdict
+  against the Python tool is `repo_map-wrong`.
+- A namespace import (`import * as ns`, `export * as ns`) counts every export of its target as
+  used, as a runtime `import()` does. The Python tool lists those exports as unused. A name that
+  a file mentions only in a comment is not an in-module reference. The Python tool counts the
+  comment. The parity record gives the measured effect. A test locks the namespace rule.
+- The opt-out marker of `repo-tools check` must stand alone on a line. `check.py` opts a document
+  out when the marker is anywhere in its text. A document can quote the marker in a sentence, a
+  code span or a table cell. That document then skips its own table, and a stale claim passes in
+  silence.
+  `repo-tools check` opts out only when a line, trimmed, equals the marker. This changes the
+  result for a document that opted out from inside a sentence or a table cell: the check now
+  reads its claims. `repo-tools map` writes the marker alone on a line, so generated reports
+  still opt out. A blank `map.verificationMarker` matches no line. `docs/parity-check.md` records
+  the difference. The verdict is fail-closed.
+- A bodiless `export function f(): T;` is not an export, as in the Python tool
+  (`repo_map-kept`). An overload plus an implementation records the name once.
+- The 1.x depgraph pipeline is removed: the runner, the regex parser, the scanner, the inventory
+  reporter and the 1.x goldens. `repo-tools depgraph` stays an alias of `repo-tools map`.
+  `docs/fix-ledger-2.0.0.md` records which 1.x fixes the `map` tests lock, and which fixes have
+  no 2.0.0 meaning.
+
+### Fixed
+
+- `src/check/claims.ts` and one test held the byte-order mark as an invisible U+FEFF character.
+  Both files now spell it as an escape sequence. A new test fails when a file under `src/`,
+  `tests/` or `scripts/` holds a literal U+FEFF.
+- A named `export { name } from` writes one edge for that statement. The statement does not add a
+  second empty-imports edge to the same file.
+- `write` refuses a symbolic link inside the output folder that points outside it.
+- A skip-list folder under `src/` (`dist`, `build`, `coverage`, `node_modules`, `.git`) that
+  holds a source file produces a warning. The files stay out of the census. A root `build/` or
+  `dist/` folder stays silent.
+- `--check-census` names the configured regenerate command. The test-coverage note names the
+  configured coverage-policy path.
+- The config file accepts a top-level `$schema` key.
+- The walk skip list is not module state. Each walk builds its own skip set.
+- A type-position `import('./c').C` records the name `C`.
+- In single-package mode, `main` and `exports` targets are roots after `dist/` maps to `src/`,
+  including when the file is not `src/index.ts`. A subpath root is the source of the export
+  target.
+- Test coverage follows an import by package name, including the side-effect imports of the
+  resolved file.
+- A workspace subpath uses the `exports` target, then `<sub>.ts`, then `<sub>/index.ts`.
+- A package-name import is an edge in cycle detection, including the cyclic components in
+  `dependency-layers.json`.
+- The comment mask treats `a++ / b / c` as division, and `if (x) /re/` as a regular expression.
+- A bare `import './x'` together with `import('./x')` is one side-effect edge whose names are
+  `["*"]`.
+- A dangling symbolic link is listed in `skippedLinks`.
+- A `new URL` in a root config, and a `new URL('./x.js', import.meta.url)` launch, seed roots.
+- A `.test.tsx` or `.spec.tsx` file is in the `tests` area.
+- The repository scan of the privacy check passes again. A test fixture used an e-mail address
+  that the check rejects. Two trailers of the merge commit of the engine change hold a co-author
+  address and a name. The fixture now uses an allowed address. The accept-list records the two
+  trailers.
+
 ## [2.0.0] - 2026-09-26
 
 The unified engine. `repo-tools map` replaces the Python `repo_map.py` and depgraph 1.x with one
@@ -16,10 +203,10 @@ flags have no effect: `map` exits 1 on them, and the alias gives a warning.
 
 ### Known differences
 
-- `map` makes no graph edge for a dynamic `import()`, as the Python tool does. depgraph 1.x
-  made one. So a cycle that closes only through `import()` is not a cycle in 2.0.0. On
-  memoryjs, one type-only cyclic component has 3 files, not 7. Test coverage still counts a
-  literal `import()` in a test file as a load.
+- The 2.0.0 release made no graph edge for a dynamic `import()`. That choice is reversed in
+  Unreleased: a literal relative `import()` is now an edge. On memoryjs, the release counted
+  one type-only cyclic component as 3 files, not 7. Test coverage counted a literal `import()`
+  in a test file as a load, and it still does.
 
 ### Added
 

package/README.md

@@ -6,16 +6,18 @@
 |---|---|
 | `map` | Writes the dependency graph and the architecture reports of a TypeScript/JavaScript, Python, C# or Rust repository. |
 | `depgraph` | The deprecated alias of `map` (through 2.x). |
+| `check` | Checks the `## Verification` tables of Markdown documents against a fresh graph, and fails on drift. |
+| `docs` | Measures the doc comments of TypeScript, JavaScript and Python files, plans the missing ones, and gates the result. |
 | `chunk` | Splits a large file into chunks, merges the chunks back, and shows which chunks changed. |
 | `compress` | Writes a compact copy of a file for a model context, and restores it. |
 | `query` | Answers structural questions from the graph of `map`, and writes two derived reports. |
 | `ste` | Checks Markdown or docstring prose against Simplified Technical English. |
 
-Status: version 2.0.0.
+Status: version 2.2.0.
 
-- `map` and `query` read their settings from `repo-tools.config.json` in the root. The section
-  of `map` is `map` (its old name, `depgraph`, is also read). `--config=<file>` makes either
-  subcommand read another file.
+- `map`, `check` and `query` read their settings from `repo-tools.config.json` in the root. The
+  section of `map` is `map` (its old name, `depgraph`, is also read). `--config=<file>` makes any
+  of the three read another file.
 - `repo-tools depgraph` prints one deprecation line and runs `map`. It accepts the 1.x
   scan-scope flags (`--src`, `--tests`, `--exclude`, `--also-exclude`, `--all`,
   `--reachable-only`, `--include-tests`) with a warning, because they have no effect in 2.0.0.
@@ -39,7 +41,7 @@ repo-tools --version
 
 The executable holds the tool and the Bun runtime. It needs no Bun, Node or `node_modules`.
 Download the file for your platform and `SHA256SUMS` from the
-[GitHub release](https://github.com/danielsimonjr/repo-tools/releases/tag/v2.0.0):
+[GitHub release](https://github.com/danielsimonjr/repo-tools/releases/tag/v2.2.0):
 
 | Platform | File |
 |---|---|
@@ -139,7 +141,13 @@ a "GENERATED FILE -- do not edit by hand" banner.
 The census is the set of source files that git tracks (outside a git work tree, a pruned walk).
 The language with the most files decides how `map` reads the repository. Input files never come
 from the output folder. The duplicate allowlist, the duplicate baseline and the coverage policy
-are in `docs/architecture/`, or at the path that the config gives.
+are in `docs/architecture/`, or at the path that the config gives. A repository that already has
+its own allowlist must set `map.duplicateAllowlist` (the old key `depgraph.duplicateAllowlist`
+is also read) to that file. Mathts has 283 allowlisted names. With the default path, the
+duplicate gate fails on those names.
+
+A literal relative `import()` is a graph edge. The Python `repo_map.py` records none. The parity
+record in `docs/parity-2.0.0.md` gives the rules and the verdict.
 
 | File | What it answers |
 |---|---|
@@ -166,6 +174,94 @@ depgraph 1.x.
 | `dependency-reverse.json` | Which files import each file? The reverse edges of the graph. |
 | `node-safety.json` | Which packages are browser-safe, which files import a `node:` builtin, and which of those files does the `.` entry of each browser-safe package reach? |
 
+## Check the documents against the graph
+
+`repo-tools check --docs=<dir>` reads every `*.md` file of the folder. The check reads the
+`## Verification` table of each document. Each row `| claim | value | source |` names a metric of
+the graph (`totalFiles`, `orphanedFiles`, `runtimeCircularDeps` and so on). The value is the
+number that the document states. The check builds the graph again, so a stale report cannot hide
+drift. A relative `--docs` path is relative to the root.
+
+```md
+## Verification
+
+| Claim | Value | Source |
+|---|---|---|
+| totalFiles | 243 | file-inventory.json |
+```
+
+```sh
+repo-tools check --docs=docs/architecture
+# check passed: every claim matches the graph (1 claim in 1 document).
+```
+
+When a value drifts, the exit code is 1, and standard error lists every problem, one per line:
+
+```text
+OVERVIEW.md: totalFiles claims 243 but actual is 244
+```
+
+A document passes without a check when one of its lines is exactly
+`<!-- repo-map:no-verification -->`. The marker must stand alone on its line: a marker inside a
+sentence, a code span or a table cell does not count.
+`repo-tools map` writes that line at the top of each generated report. Every other gap is a
+failure. A missing docs folder, a folder with no `*.md` file, a document without a Verification
+section, and a section without a row are failures. An unknown claim name is a failure too. So is a
+metric that the graph build declared unreliable. `docs/design.md` (section 15) gives the rules and the
+differences from the Python tool. `docs/parity-check.md` records the comparison with that tool on
+11 repositories.
+
+## Gate the documentation comments
+
+`repo-tools docs` reads the TypeScript, JavaScript and Python files that git tracks. It finds each
+function, class, interface, type alias and method, and the doc comment of each one. It needs no
+Python on the computer.
+
+```sh
+repo-tools docs scan .                      # writes docs/code-docs/COVERAGE.md and coverage.json
+repo-tools docs check .                     # the gate: exit 1 on a MUST issue
+repo-tools docs stub . --path=src/api       # a dry run: lists the stubs and writes nothing
+repo-tools docs stub . --path=src/api --apply
+```
+
+The rules are:
+
+| Rule | Tier | The rule |
+|---|---|---|
+| M1 | MUST | An exported symbol has a doc comment. A test file is exempt. |
+| M2 | MUST | The doc comment has a summary line. |
+| M3 | MUST | The doc comment names no parameter that the signature lacks. |
+| M4 | MUST | The doc comment holds no `TODO:` marker. |
+| M5 | MUST | A file uses one doc dialect: `google`, `numpy` or `rest` for Python, `tsdoc` or `jsdoc` for TypeScript. |
+| S2 | SHOULD | The summary ends with a full stop. |
+| S5 | SHOULD | The doc comment does not repeat a type that the signature holds. |
+| S6 | SHOULD | The prose follows Simplified Technical English (the rules of `repo-tools ste`). |
+
+Only a MUST issue changes the exit code of `check`. A file that does not parse is also a failure,
+because an unknown file is not a clean file. A repository with no source file fails too.
+
+`stub` is a dry run until you pass `--apply`. Each stub holds the marker `TODO:`, so `check` fails
+until you replace the stub with real prose. The command parses a file again after the change.
+A file that would no longer parse stays unchanged. The command skips a file that is a link or
+resolves outside the root, and it names the skipped file.
+
+To gate a pull request and not the whole backlog, give `check` the changed files:
+
+```sh
+git diff --name-only origin/main... > changed.txt
+repo-tools docs check . --paths-from=changed.txt
+```
+
+A `.code-docs.json` file at the root excludes paths. Each entry needs a reason, and the output
+names each exclusion:
+
+```json
+{ "exclude": [ { "path": "assembly/", "reason": "AssemblyScript" } ] }
+```
+
+`docs/design.md` (section 16) gives the rules, the exit codes and the differences from the Python
+tool. `docs/parity-docs.md` records the comparison with that tool on 9 repositories.
+
 ## Query the graph
 
 `repo-tools query` answers questions from `dependency-graph.json` (and
@@ -225,6 +321,8 @@ the config key `query.out`, else from `map.out`, else from `docs/architecture`.
 | `bun run smoke -- <command...>` | Runs the smoke test against one way to run the tool. |
 | `bun run privacy` | Runs the privacy check on the tracked files and the commit messages. |
 | `bun run hooks` | Installs the `commit-msg` hook that runs the privacy check on each message. |
+| `bun run audit:ledger` | Reverts each fix of `docs/fix-ledger-2.0.0.md` in a scratch copy and checks that a test of that fix fails. |
+| `bun run check:design` | Finds each reference to a section of `docs/design.md` that has no heading with that number. |
 
 ## Design