@danielsimonjr/repo-tools 2.2.6 → 2.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -6,6 +6,44 @@ All notable changes to this project are recorded in this file. The format follow
6
6
 
7
7
  ## [Unreleased]
8
8
 
9
+ ## [2.3.0] - 2026-10-22
10
+
11
+ ### Added
12
+
13
+ - `repo-tools clones` lists the functions of a TypeScript or JavaScript repository and finds the functions that copy
14
+ each other. It reads the source and writes files only when `--out` or the `clones.out` config key names a folder.
15
+ A unit is a named function with at least 50 labels (`--min-nodes`). The command reports exact groups (the same label
16
+ sequence) and near pairs (shared runs of labels, scored by Jaccard and containment). It prints a summary, and
17
+ with an output folder it writes `inventory.jsonl` and `clones.json`. Every limit is a flag and a key of the new
18
+ `clones` section of `repo-tools.config.json`. A flag overrides the file, and the file overrides the default.
19
+ Exit code 0 means the run finished, 1 means the root holds no source file to read, and 2 means a usage or config
20
+ error. `docs/design.md` section 17 gives the rules.
21
+ - `docs/parity-clones.md` compares the command with the prototype of the memoryjs duplicate-function audit on 292
22
+ files. 1,368 units are the same. The command finds all 20 exact groups of the prototype. Every unit that only one
23
+ side reports lies at the cutoff of 50 labels.
24
+ - The smoke test runs `clones` on a small repository that holds a planted copy, so the shipped bundle proves the
25
+ command and its tree-sitter loader.
26
+
27
+ ## [2.2.7] - 2026-10-22
28
+
29
+ ### Fixed
30
+
31
+ - `repo-tools ste` exits 2 when any path in the argument list does not exist, and prints `ERROR: path does not exist:
32
+ <path>` on stderr for each such path. Before, a mistyped path among valid paths was ignored and the run exited 0.
33
+ The report of each valid path is still printed.
34
+ - The warning that `repo-tools map` writes for a `package.json` entry that does not resolve is shorter (15 words).
35
+ It had 30 words, so `repo-tools ste` flagged the generated `FILE_INVENTORY.md`. A test now checks the word limit
36
+ of every warning and that the generated report passes `repo-tools ste`.
37
+ - `repo-tools map` reads a folder named `test/` as a tests folder, as version 1.x did. Before, only `tests/` counted,
38
+ so a file such as `test/setup.ts` got the area `src` in `FILE_INVENTORY.md` and counted as a source file in
39
+ `TEST_COVERAGE.md`. The disposition legend and the `--tests` message name both folders.
40
+ - The `TEST_COVERAGE.md` note on the coverage policy no longer links `COVERAGE_POLICY.md`, a file that the tool does
41
+ not write, and no longer names mathjs or AssemblyScript.
42
+ - The `Expected test` hint in `TEST_COVERAGE.md` names `tests/unit/<module>/`. It appears only when the repository
43
+ keeps tests in `tests/unit/`. Other repositories get the plain list of untested files.
44
+ - The text of `duplicate-symbols.md` and of the `duplicate-symbols.json` note no longer names `Bucket C` or
45
+ `AssemblyScript`.
46
+
9
47
  ## [2.2.6] - 2026-10-22
10
48
 
11
49
  ### Fixed
package/README.md CHANGED
@@ -11,13 +11,16 @@
11
11
  | `chunk` | Splits a large file into chunks, merges the chunks back, and shows which chunks changed. |
12
12
  | `compress` | Writes a compact copy of a file for a model context, and restores it. |
13
13
  | `query` | Answers structural questions from the graph of `map`, and writes two derived reports. |
14
+ | `clones` | Lists the functions of a TypeScript or JavaScript repository, and finds the functions that copy each other. |
14
15
  | `ste` | Checks Markdown or docstring prose against Simplified Technical English. |
15
16
 
16
- Status: version 2.2.6.
17
+ Status: version 2.3.0.
17
18
 
18
19
  - `map`, `check` and `query` read their settings from `repo-tools.config.json` in the root. The
19
20
  section of `map` is `map` (its old name, `depgraph`, is also read). `--config=<file>` makes any
20
21
  of the three read another file.
22
+ - `clones` reads its settings from the `clones` section of the same file. A flag overrides the
23
+ file, and the file overrides the default.
21
24
  - `repo-tools depgraph` prints one deprecation line and runs `map`. It accepts the 1.x
22
25
  scan-scope flags (`--src`, `--tests`, `--exclude`, `--also-exclude`, `--all`,
23
26
  `--reachable-only`, `--include-tests`) with a warning, because they have no effect in 2.0.0.
@@ -41,7 +44,7 @@ repo-tools --version
41
44
 
42
45
  The executable holds the tool and the Bun runtime. It needs no Bun, Node or `node_modules`.
43
46
  Download the file for your platform and `SHA256SUMS` from the
44
- [GitHub release](https://github.com/danielsimonjr/repo-tools/releases/tag/v2.2.6):
47
+ [GitHub release](https://github.com/danielsimonjr/repo-tools/releases/tag/v2.3.0):
45
48
 
46
49
  | Platform | File |
47
50
  |---|---|
@@ -311,6 +314,38 @@ repo-tools query --emit
311
314
  the config key `query.out`, else from `map.out`, else from `docs/architecture`. Run
312
315
  `repo-tools query --help` for every option.
313
316
 
317
+ ## Find the functions that copy each other
318
+
319
+ `repo-tools clones` lists the functions of a TypeScript or JavaScript repository. It finds the
320
+ functions that copy each other. The command only reads the repository. It writes files only when
321
+ you name an output folder.
322
+
323
+ ```sh
324
+ repo-tools clones . # prints the summary and writes no file
325
+ repo-tools clones . --src=src --out=reports/clones
326
+ repo-tools clones . --min-nodes=30 --near=0.7
327
+ ```
328
+
329
+ A function is a unit when it has a name and its body has at least 50 labels. A label is one node
330
+ of the body. Names, literals, types and comments give no detail, so a copy with other names still
331
+ matches. The command finds two kinds of result:
332
+
333
+ | Result | Meaning |
334
+ |---|---|
335
+ | Exact group | Units with the same label sequence. |
336
+ | Near pair | Two units that share enough runs of 6 labels. |
337
+
338
+ `--src=<path>...` names the folders to read. The default `auto` reads the files of the `src` area.
339
+ `--out=<dir>` writes `inventory.jsonl` (one line for each unit) and `clones.json` (the groups and the
340
+ pairs) into a folder. Give the folder as a path relative to the root. Run `repo-tools clones --help` for every limit.
341
+
342
+ Exit codes: 0 when the run finished, even if it found no copy. 1 when the root holds no source file
343
+ to read. 2 on a usage error, a bad config file or a root that is not a folder.
344
+
345
+ The result is a list of candidates. Read both bodies of a pair before you change anything.
346
+ `docs/design.md` (section 17) gives the rules. `docs/parity-clones.md` records the comparison with the
347
+ prototype of the memoryjs audit.
348
+
314
349
  ## Develop
315
350
 
316
351
  | Command | Job |
package/dist/cli.js CHANGED
@@ -2,7 +2,7 @@
2
2
  // package.json
3
3
  var package_default = {
4
4
  name: "@danielsimonjr/repo-tools",
5
- version: "2.2.6",
5
+ version: "2.3.0",
6
6
  description: "One CLI for repository analysis: dependency graphs creation and query with per-export API facts, file chunking, and context compression. TypeScript on Bun, compiled to a single executable.",
7
7
  type: "module",
8
8
  license: "MIT",
@@ -851,6 +851,16 @@ function checkValue(key, kind, value, fail) {
851
851
  return text(value, "a non-empty string");
852
852
  case "string-or-null":
853
853
  return value === null ? null : text(value, "a non-empty string or null");
854
+ case "count":
855
+ if (typeof value !== "number" || !Number.isInteger(value) || value < 1) {
856
+ throw fail(`'${key}' must be a whole number of 1 or more`);
857
+ }
858
+ return value;
859
+ case "fraction":
860
+ if (typeof value !== "number" || !(value >= 0 && value <= 1)) {
861
+ throw fail(`'${key}' must be a number from 0 to 1`);
862
+ }
863
+ return value;
854
864
  }
855
865
  }
856
866
  function checkSection(prefix, object, keys, fail) {
@@ -867,6 +877,20 @@ var QUERY_KEYS = {
867
877
  out: "path",
868
878
  nodeRuntimes: "names"
869
879
  };
880
+ var CLONES_KEYS = {
881
+ src: "src",
882
+ out: "path-or-null",
883
+ minNodes: "count",
884
+ shingleLength: "count",
885
+ nearThreshold: "fraction",
886
+ containmentThreshold: "fraction",
887
+ containmentMinShingles: "count",
888
+ rareShingleLimit: "count",
889
+ minSharedShingles: "count"
890
+ };
891
+ function checkClonesFlag(label, key, value) {
892
+ return checkValue(label, CLONES_KEYS[key], value, (message) => new Error(message));
893
+ }
870
894
  function parseConfigSections(parsed, label) {
871
895
  const fail = (message) => new Error(`config ${label}: ${message}`);
872
896
  if (!isObject(parsed))
@@ -874,7 +898,7 @@ function parseConfigSections(parsed, label) {
874
898
  for (const key of Object.keys(parsed)) {
875
899
  if (key === "$schema")
876
900
  continue;
877
- if (key !== "depgraph" && key !== "map" && key !== "query") {
901
+ if (key !== "depgraph" && key !== "map" && key !== "query" && key !== "clones") {
878
902
  throw fail(`unknown key '${key}'`);
879
903
  }
880
904
  }
@@ -882,7 +906,7 @@ function parseConfigSections(parsed, label) {
882
906
  throw fail("use the section 'map' or its old name 'depgraph', not both");
883
907
  }
884
908
  const sectionName = parsed.map !== undefined ? "map" : "depgraph";
885
- const sections = { depgraph: {}, query: {} };
909
+ const sections = { depgraph: {}, query: {}, clones: {} };
886
910
  const section = parsed[sectionName];
887
911
  if (section !== undefined) {
888
912
  if (!isObject(section))
@@ -901,6 +925,12 @@ function parseConfigSections(parsed, label) {
901
925
  throw fail("'query' must be an object");
902
926
  sections.query = checkSection("query.", query, QUERY_KEYS, fail);
903
927
  }
928
+ const clones = parsed.clones;
929
+ if (clones !== undefined) {
930
+ if (!isObject(clones))
931
+ throw fail("'clones' must be an object");
932
+ sections.clones = checkSection("clones.", clones, CLONES_KEYS, fail);
933
+ }
904
934
  return sections;
905
935
  }
906
936
  function loadConfigSections(root, configPath) {
@@ -911,7 +941,7 @@ function loadConfigSections(root, configPath) {
911
941
  const label = `<root>/${rel.replace(/\\/g, "/")}`;
912
942
  const path = resolveUnderRoot(root, rel);
913
943
  if (configPath === undefined && !existsSync2(path))
914
- return { depgraph: {}, query: {} };
944
+ return { depgraph: {}, query: {}, clones: {} };
915
945
  let text;
916
946
  try {
917
947
  text = readFileSync(path, "utf8");
@@ -974,7 +1004,7 @@ function mergeDepgraphConfig(cli, file) {
974
1004
  // src/depgraph/args.ts
975
1005
  var SCAN_SCOPE_FLAGS = {
976
1006
  "--src": "the census finds the source files itself; the area of a file comes from its path",
977
- "--tests": "a test file is found by its name or its tests/ folder",
1007
+ "--tests": "a test file is found by its name or by its test/ or tests/ folder",
978
1008
  "--exclude": "the census reads the files that git tracks; list a file in .gitignore instead",
979
1009
  "--also-exclude": "the census reads the files that git tracks; list a file in .gitignore instead",
980
1010
  "--all": "the graph always holds every file; the disposition of each file shows reachability",
@@ -1372,7 +1402,7 @@ var TYPE_DUP_CATEGORIES = [
1372
1402
  { cat: "type", key: "types" },
1373
1403
  { cat: "enum", key: "enums" }
1374
1404
  ];
1375
- var DUPLICATE_SYMBOLS_NOTE = "This report groups names by OWN definition, not by call graph, then classifies each " + "flagged name (see DupEntryTag): TRUE_DUPLICATE (the actionable merge targets), " + "DISPATCH_VARIANT (>=2 mathTyped(...) registrations of the same public name — distinct " + "dispatch surfaces, Bucket C delegation candidates, not copy-paste bodies), " + "ALIAS_DELEGATION (a const-alias forward to an imported symbol, not an independent body — " + "excluded once fewer than 2 real bodies remain), and ALLOWLISTED (matches " + "duplicate-allowlist.json: hot-path is* guards, AssemblyScript mirrors, per-package " + "VERSION strings). NOT detected: same-file typed-dispatch overload polymorphism for " + "different argument shapes within one registration — a human still triages TRUE_DUPLICATE " + "entries using the defining files + public flags before merging anything.";
1405
+ var DUPLICATE_SYMBOLS_NOTE = "This report groups names by OWN definition, not by call graph, then classifies each " + "flagged name (see DupEntryTag): TRUE_DUPLICATE (the actionable merge targets), " + "DISPATCH_VARIANT (>=2 mathTyped(...) registrations of the same public name — distinct " + "dispatch surfaces, not copy-paste bodies), " + "ALIAS_DELEGATION (a const-alias forward to an imported symbol, not an independent body — " + "excluded once fewer than 2 real bodies remain), and ALLOWLISTED (matches " + "duplicate-allowlist.json: for example hot-path is* guards, mirrors in another language, " + "per-package VERSION strings). NOT detected: same-file typed-dispatch overload polymorphism for " + "different argument shapes within one registration — a human still triages TRUE_DUPLICATE " + "entries using the defining files + public flags before merging anything.";
1376
1406
  var DUP_ENTRY_TAG_SORT_ORDER = {
1377
1407
  TRUE_DUPLICATE: 0,
1378
1408
  DISPATCH_VARIANT: 1,
@@ -6378,7 +6408,7 @@ function classifyArea(relPath) {
6378
6408
  }
6379
6409
  if (CONFIG_PATTERN.test(relPath))
6380
6410
  return "config";
6381
- if (isTestFile || /(^|\/)tests\//.test(relPath))
6411
+ if (isTestFile || /(^|\/)tests?\//.test(relPath))
6382
6412
  return "tests";
6383
6413
  if (/^(bench|benchmarks)\//.test(relPath) || /\.bench\.\w+$/.test(relPath))
6384
6414
  return "benchmarks";
@@ -12435,7 +12465,7 @@ function packageJsonRoots(root, known) {
12435
12465
  continue;
12436
12466
  const mapped = mapDistToSrc(spec, known) ?? launcherEntry(root, spec, known) ?? declaredSource(spec, known);
12437
12467
  if (mapped === null) {
12438
- warnings.push(`package.json entry ${pyRepr(spec)} did not resolve to a known source file -- ` + "ignored; if no other declared entry resolves either, the entry-point root falls back to " + "a conventional src/index.* file instead");
12468
+ warnings.push(`package.json entry ${pyRepr(spec)} did not resolve to a known source file. The entry is ignored.`);
12439
12469
  continue;
12440
12470
  }
12441
12471
  if (!roots.includes(mapped))
@@ -13631,7 +13661,7 @@ function generateTestCoverageMarkdown(coverage, options = {}) {
13631
13661
  lines.push("");
13632
13662
  lines.push(...basisNote);
13633
13663
  const policyPath = options.policyPath ?? "docs/architecture/coverage-policy.json";
13634
- lines.push("> The raw figure counts every source file the CDG tool finds, including code that is " + "intentionally not direct-imported by a vitest `*.test.ts` (synced mathjs categories, " + "AssemblyScript sources, type-only barrels, …). The **effective** figure excludes those " + `per \`${policyPath}\` so the number reflects the genuinely-active hand-written code only. ` + "See [`COVERAGE_POLICY.md`](./COVERAGE_POLICY.md) for the policy.");
13664
+ lines.push("> The raw figure counts every source file the map finds, including files that no test " + "imports directly by design (for example generated code, type-only barrels and code in " + `another language). The **effective** figure excludes the files that \`${policyPath}\` ` + "classifies, so it shows the active code only.");
13635
13665
  lines.push("");
13636
13666
  lines.push("### Untested-file breakdown by category");
13637
13667
  lines.push("");
@@ -14227,7 +14257,7 @@ function emitTestCoverage(graph, root, outDir, options = {}) {
14227
14257
  const jsonPath = join16(outDir, "test-coverage.json");
14228
14258
  const namespaceImportBasis = !importsNameSymbols(graph.language);
14229
14259
  const markdown = generateTestCoverageMarkdown(coverage, {
14230
- expectedTestHint: graph.language === "typescript",
14260
+ expectedTestHint: graph.language === "typescript" && tests.some((t) => t.path.startsWith("tests/unit/")),
14231
14261
  policyPath: options.policyNote ?? "docs/architecture/coverage-policy.json",
14232
14262
  namespaceImportBasis
14233
14263
  });
@@ -14287,17 +14317,14 @@ function generateDuplicateSymbolsMarkdown(report) {
14287
14317
  md += "Each name carries one tag (see `DupEntryTag`), so the actionable subset is clear:\n\n";
14288
14318
  md += "- `TRUE_DUPLICATE`: a real merge target.\n";
14289
14319
  md += "- `DISPATCH_VARIANT`: two or more `mathTyped(...)` registrations of the same public name. ";
14290
- md += "The registrations are distinct dispatch surfaces and Bucket C delegation candidates, ";
14291
- md += `not copy-paste bodies.
14320
+ md += `The registrations are distinct dispatch surfaces, not copy-paste bodies.
14292
14321
  `;
14293
14322
  md += "- `ALIAS_DELEGATION`: a `const X = importedY` forward. ";
14294
14323
  md += `The entry is excluded once fewer than two real bodies remain.
14295
14324
  `;
14296
14325
  md += "- `ALLOWLISTED`: the name matches `duplicate-allowlist.json`. ";
14297
- md += "Examples are hot-path `is*` guards, AssemblyScript mirrors and per-package `VERSION` ";
14298
- md += `strings.
14299
-
14300
- `;
14326
+ md += "Examples are hot-path `is*` guards, mirrors in another language and per-package ";
14327
+ md += "`VERSION` strings.\n\n";
14301
14328
  md += `> **Note:** ${report.note}
14302
14329
 
14303
14330
  `;
@@ -14313,7 +14340,7 @@ function generateDuplicateSymbolsMarkdown(report) {
14313
14340
 
14314
14341
  `;
14315
14342
  md += renderTaggedSection("TRUE_DUPLICATE — actionable merge targets", report.runtime, "TRUE_DUPLICATE");
14316
- md += renderTaggedSection("DISPATCH_VARIANT — distinct public typed-dispatch surfaces (Bucket C candidates)", report.runtime, "DISPATCH_VARIANT");
14343
+ md += renderTaggedSection("DISPATCH_VARIANT — distinct public typed-dispatch surfaces", report.runtime, "DISPATCH_VARIANT");
14317
14344
  md += renderTaggedSection("ALIAS_DELEGATION — const-alias forwards (not independent bodies)", report.runtime, "ALIAS_DELEGATION");
14318
14345
  md += renderTaggedSection("ALLOWLISTED — accepted layering (see duplicate-allowlist.json)", report.runtime, "ALLOWLISTED");
14319
14346
  md += `## Type duplicates (lower priority)
@@ -14332,7 +14359,7 @@ var DISPOSITION_LEGEND = [
14332
14359
  ["build-entry", "A `src`-area file that is an entry root."],
14333
14360
  ["test-only", "A `src`-area file that only a test reaches."],
14334
14361
  ["orphan", "A `src`-area file that nothing reaches. Delete it, or wire it to a root."],
14335
- ["test", "A file in a `tests/` folder, or a `*.test.ts` or `*.spec.ts` file."],
14362
+ ["test", "A file in a `test/` or `tests/` folder, or a `*.test.ts` or `*.spec.ts` file."],
14336
14363
  ["tool", "A file in a `tools/` or `scripts/` folder outside `src/`."],
14337
14364
  ["config", "A `*.config.*` file of TypeScript or JavaScript."],
14338
14365
  ["example", "A file in `examples/` or `docs/`."],
@@ -16567,9 +16594,655 @@ ${CHUNK_HELP}`);
16567
16594
  }
16568
16595
  }
16569
16596
 
16597
+ // src/clones/index.ts
16598
+ import { join as join22, resolve as resolve8 } from "node:path";
16599
+
16600
+ // src/clones/detect.ts
16601
+ function fnv1a(text) {
16602
+ let h = 2166136261;
16603
+ for (let i2 = 0;i2 < text.length; i2++) {
16604
+ h ^= text.charCodeAt(i2);
16605
+ h = Math.imul(h, 16777619);
16606
+ }
16607
+ return h >>> 0;
16608
+ }
16609
+ function groupsBy(units, key) {
16610
+ const groups = new Map;
16611
+ units.forEach((unit, id) => {
16612
+ const ids = groups.get(key(unit));
16613
+ if (ids)
16614
+ ids.push(id);
16615
+ else
16616
+ groups.set(key(unit), [id]);
16617
+ });
16618
+ return [...groups.values()].filter((ids) => ids.length > 1);
16619
+ }
16620
+ function shinglesOf(labels, length) {
16621
+ const set = new Set;
16622
+ for (let i2 = 0;i2 + length <= labels.length; i2++) {
16623
+ set.add(fnv1a(labels.slice(i2, i2 + length).join("|")));
16624
+ }
16625
+ return set;
16626
+ }
16627
+ function nested(a, b) {
16628
+ if (a.file !== b.file)
16629
+ return false;
16630
+ const aInB = b.startIndex <= a.startIndex && a.endIndex <= b.endIndex;
16631
+ const bInA = a.startIndex <= b.startIndex && b.endIndex <= a.endIndex;
16632
+ return aInB || bInA;
16633
+ }
16634
+ function overlap(small, big) {
16635
+ let shared = 0;
16636
+ for (const item of small)
16637
+ if (big.has(item))
16638
+ shared++;
16639
+ return shared;
16640
+ }
16641
+ function round3(value) {
16642
+ return Number(value.toFixed(3));
16643
+ }
16644
+ function detect(units, params) {
16645
+ const exactGroups = groupsBy(units, (u) => u.exactHash);
16646
+ const textGroups = groupsBy(units, (u) => u.textHash);
16647
+ const shingles = units.map((u) => shinglesOf(u.labels, params.shingleLength));
16648
+ const index = new Map;
16649
+ shingles.forEach((set, id) => {
16650
+ for (const shingle of set) {
16651
+ const ids = index.get(shingle);
16652
+ if (ids)
16653
+ ids.push(id);
16654
+ else
16655
+ index.set(shingle, [id]);
16656
+ }
16657
+ });
16658
+ const near = [];
16659
+ units.forEach((unit, id) => {
16660
+ const mine = shingles[id];
16661
+ const shared = new Map;
16662
+ for (const shingle of mine) {
16663
+ const ids = index.get(shingle);
16664
+ if (ids.length > params.rareShingleLimit)
16665
+ continue;
16666
+ for (const other of ids)
16667
+ if (other > id)
16668
+ shared.set(other, (shared.get(other) ?? 0) + 1);
16669
+ }
16670
+ for (const [other, count] of shared) {
16671
+ if (count < params.minSharedShingles)
16672
+ continue;
16673
+ const peer = units[other];
16674
+ if (nested(unit, peer))
16675
+ continue;
16676
+ const theirs = shingles[other];
16677
+ const [small, big] = mine.size <= theirs.size ? [mine, theirs] : [theirs, mine];
16678
+ const inter = overlap(small, big);
16679
+ const jaccard = inter / (mine.size + theirs.size - inter);
16680
+ const contain = inter / small.size;
16681
+ const byContainment = contain >= params.containmentThreshold && small.size >= params.containmentMinShingles;
16682
+ if (jaccard < params.nearThreshold && !byContainment)
16683
+ continue;
16684
+ const common = overlap(unit.names, peer.names);
16685
+ const union = unit.names.size + peer.names.size - common;
16686
+ near.push({
16687
+ a: id,
16688
+ b: other,
16689
+ jStruct: round3(jaccard),
16690
+ contain: round3(contain),
16691
+ jNames: round3(common / (union || 1))
16692
+ });
16693
+ }
16694
+ });
16695
+ near.sort((x, y) => y.jStruct - x.jStruct || x.a - y.a || x.b - y.b);
16696
+ return { exactGroups, textGroups, near };
16697
+ }
16698
+
16699
+ // src/clones/report.ts
16700
+ function inventoryRecord(unit, id) {
16701
+ return {
16702
+ id,
16703
+ file: unit.file,
16704
+ name: unit.name,
16705
+ container: unit.container,
16706
+ kind: unit.kind,
16707
+ exported: unit.exported,
16708
+ startLine: unit.startLine,
16709
+ endLine: unit.endLine,
16710
+ loc: unit.loc,
16711
+ params: unit.params,
16712
+ async: unit.async,
16713
+ nodes: unit.nodes,
16714
+ doc: unit.doc,
16715
+ exactHash: unit.exactHash,
16716
+ textHash: unit.textHash
16717
+ };
16718
+ }
16719
+ function inventoryJsonl(units) {
16720
+ return units.map((unit, id) => `${JSON.stringify(inventoryRecord(unit, id))}
16721
+ `).join("");
16722
+ }
16723
+ function listJson(items) {
16724
+ if (items.length === 0)
16725
+ return "[]";
16726
+ return `[
16727
+ ${items.map((item) => ` ${JSON.stringify(item)}`).join(`,
16728
+ `)}
16729
+ ]`;
16730
+ }
16731
+ function clonesJson(config, detection) {
16732
+ const { out: _out, ...params } = config;
16733
+ return [
16734
+ "{",
16735
+ ` "params": ${JSON.stringify(params)},`,
16736
+ ` "exactGroups": ${listJson(detection.exactGroups)},`,
16737
+ ` "textGroups": ${listJson(detection.textGroups)},`,
16738
+ ` "near": ${listJson(detection.near)}`,
16739
+ "}",
16740
+ ""
16741
+ ].join(`
16742
+ `);
16743
+ }
16744
+ var SUMMARY_BANDS = [
16745
+ ["0.9 or more", 0.9, Number.POSITIVE_INFINITY],
16746
+ ["0.8 to 0.9", 0.8, 0.9],
16747
+ ["0.7 to 0.8", 0.7, 0.8],
16748
+ ["0.6 to 0.7", 0.6, 0.7],
16749
+ ["below 0.6 (containment)", 0, 0.6]
16750
+ ];
16751
+ function folderOf(file) {
16752
+ const parts2 = file.split("/").slice(0, -1);
16753
+ return parts2.length === 0 ? "." : parts2.slice(0, 2).join("/");
16754
+ }
16755
+ function percentile(sorted, p) {
16756
+ return sorted[Math.min(sorted.length - 1, Math.floor(sorted.length * p))] ?? 0;
16757
+ }
16758
+ function summaryText(config, stats, units, detection) {
16759
+ const sizes = units.map((u) => u.nodes).sort((a, b) => a - b);
16760
+ const inGroups = detection.exactGroups.reduce((sum, group) => sum + group.length, 0);
16761
+ const bands = SUMMARY_BANDS.map(([label, lo, hi]) => `${label}: ${detection.near.filter((p) => p.jStruct >= lo && p.jStruct < hi).length}`);
16762
+ const byFolder = new Map;
16763
+ for (const unit of units) {
16764
+ const key = folderOf(unit.file);
16765
+ byFolder.set(key, (byFolder.get(key) ?? 0) + 1);
16766
+ }
16767
+ const folders = [...byFolder.entries()].sort(([a], [b]) => a < b ? -1 : a > b ? 1 : 0).map(([key, count]) => `${key}=${count}`).join(" ");
16768
+ const lines = [
16769
+ `files=${stats.files} parseErrors=${stats.parseErrors} units(minNodes=${config.minNodes})=${units.length}`,
16770
+ `nodes p50=${percentile(sizes, 0.5)} p90=${percentile(sizes, 0.9)} p99=${percentile(sizes, 0.99)} max=${sizes[sizes.length - 1] ?? 0}`,
16771
+ `exact groups=${detection.exactGroups.length} (units in them=${inGroups}) text-identical groups=${detection.textGroups.length}`,
16772
+ `near pairs=${detection.near.length} by jStruct: ${bands.join(", ")}`,
16773
+ `units by folder: ${folders}`
16774
+ ];
16775
+ return `${lines.join(`
16776
+ `)}
16777
+ `;
16778
+ }
16779
+
16780
+ // src/clones/settings.ts
16781
+ var CLONES_DEFAULTS = Object.freeze({
16782
+ src: "auto",
16783
+ out: null,
16784
+ minNodes: 50,
16785
+ shingleLength: 6,
16786
+ nearThreshold: 0.6,
16787
+ containmentThreshold: 0.9,
16788
+ containmentMinShingles: 20,
16789
+ rareShingleLimit: 60,
16790
+ minSharedShingles: 3
16791
+ });
16792
+ function mergeClonesConfig(cli, file) {
16793
+ const d = CLONES_DEFAULTS;
16794
+ const src = cli.src ?? file.src ?? d.src;
16795
+ return {
16796
+ src: src === "auto" ? src : [...src],
16797
+ out: cli.out !== undefined ? cli.out : file.out !== undefined ? file.out : d.out,
16798
+ minNodes: cli.minNodes ?? file.minNodes ?? d.minNodes,
16799
+ shingleLength: cli.shingleLength ?? file.shingleLength ?? d.shingleLength,
16800
+ nearThreshold: cli.nearThreshold ?? file.nearThreshold ?? d.nearThreshold,
16801
+ containmentThreshold: cli.containmentThreshold ?? file.containmentThreshold ?? d.containmentThreshold,
16802
+ containmentMinShingles: cli.containmentMinShingles ?? file.containmentMinShingles ?? d.containmentMinShingles,
16803
+ rareShingleLimit: cli.rareShingleLimit ?? file.rareShingleLimit ?? d.rareShingleLimit,
16804
+ minSharedShingles: cli.minSharedShingles ?? file.minSharedShingles ?? d.minSharedShingles
16805
+ };
16806
+ }
16807
+
16808
+ // src/clones/units.ts
16809
+ import { createHash as createHash2 } from "node:crypto";
16810
+
16811
+ // src/clones/normalise.ts
16812
+ var IDENTIFIERS = new Set([
16813
+ "identifier",
16814
+ "property_identifier",
16815
+ "shorthand_property_identifier",
16816
+ "shorthand_property_identifier_pattern",
16817
+ "private_property_identifier",
16818
+ "statement_identifier",
16819
+ "undefined"
16820
+ ]);
16821
+ var LITERALS = new Set(["string", "number", "regex", "true", "false", "null"]);
16822
+ var TYPE_SYNTAX = new Set([
16823
+ "type_annotation",
16824
+ "type_arguments",
16825
+ "type_parameters",
16826
+ "type_alias_declaration",
16827
+ "interface_declaration",
16828
+ "enum_declaration",
16829
+ "ambient_declaration",
16830
+ "asserts_annotation",
16831
+ "opting_type_annotation",
16832
+ "omitting_type_annotation",
16833
+ "type_predicate_annotation",
16834
+ "accessibility_modifier",
16835
+ "override_modifier",
16836
+ "decorator",
16837
+ "comment"
16838
+ ]);
16839
+ var TRANSPARENT = new Set([
16840
+ "parenthesized_expression",
16841
+ "as_expression",
16842
+ "satisfies_expression",
16843
+ "non_null_expression",
16844
+ "type_assertion",
16845
+ "instantiation_expression"
16846
+ ]);
16847
+ var TEMPLATE_TEXT = new Set(["string_fragment", "escape_sequence"]);
16848
+ function wrapped(node) {
16849
+ const children = node.namedChildren;
16850
+ return (node.type === "type_assertion" ? children[children.length - 1] : children[0]) ?? null;
16851
+ }
16852
+ function walk2(node, out2) {
16853
+ if (!node.isNamed)
16854
+ return;
16855
+ const type = node.type;
16856
+ if (TYPE_SYNTAX.has(type))
16857
+ return;
16858
+ if (TRANSPARENT.has(type)) {
16859
+ const inner = wrapped(node);
16860
+ if (inner)
16861
+ walk2(inner, out2);
16862
+ return;
16863
+ }
16864
+ if (IDENTIFIERS.has(type)) {
16865
+ out2.labels.push("ID");
16866
+ out2.names.add(node.text.toLowerCase());
16867
+ return;
16868
+ }
16869
+ if (LITERALS.has(type)) {
16870
+ out2.labels.push("LIT");
16871
+ return;
16872
+ }
16873
+ if (TEMPLATE_TEXT.has(type)) {
16874
+ out2.labels.push("TPL");
16875
+ return;
16876
+ }
16877
+ if (type !== "template_substitution") {
16878
+ const operator = node.childForFieldName("operator");
16879
+ out2.labels.push(operator ? `${type}:${operator.type}` : type);
16880
+ }
16881
+ for (const child of node.children)
16882
+ walk2(child, out2);
16883
+ }
16884
+ function normalise(body2) {
16885
+ const out2 = { labels: [], names: new Set };
16886
+ walk2(body2, out2);
16887
+ return out2;
16888
+ }
16889
+
16890
+ // src/clones/units.ts
16891
+ var DOC_LIMIT = 160;
16892
+ var FUNCTION_VALUES = new Set([
16893
+ "arrow_function",
16894
+ "function_expression",
16895
+ "function",
16896
+ "generator_function"
16897
+ ]);
16898
+ var CLASSES = new Set(["class_declaration", "abstract_class_declaration", "class"]);
16899
+ var ANONYMOUS_CLASS = "(class)";
16900
+ function sha1(text) {
16901
+ return createHash2("sha1").update(text).digest("hex").slice(0, 16);
16902
+ }
16903
+ function keyName(key) {
16904
+ if (!key)
16905
+ return null;
16906
+ return key.type === "string" ? key.text.slice(1, -1) : key.text;
16907
+ }
16908
+ function hasToken(node, token) {
16909
+ return node.children.some((c) => !c.isNamed && c.type === token);
16910
+ }
16911
+ function anchorOf(node) {
16912
+ let anchor = node;
16913
+ if (node.type === "variable_declarator" && node.parent)
16914
+ anchor = node.parent;
16915
+ if (anchor.parent?.type === "export_statement")
16916
+ anchor = anchor.parent;
16917
+ return anchor;
16918
+ }
16919
+ function declared(node, container) {
16920
+ switch (node.type) {
16921
+ case "function_declaration":
16922
+ case "generator_function_declaration": {
16923
+ const name2 = keyName(node.childForFieldName("name"));
16924
+ if (!name2)
16925
+ return null;
16926
+ return { fn: node, name: name2, kind: container ? "local" : "function", anchor: anchorOf(node) };
16927
+ }
16928
+ case "method_definition": {
16929
+ const name2 = keyName(node.childForFieldName("name"));
16930
+ if (!name2)
16931
+ return null;
16932
+ const inClass = node.parent?.type === "class_body";
16933
+ let kind = "prop";
16934
+ if (inClass) {
16935
+ if (name2 === "constructor")
16936
+ kind = "ctor";
16937
+ else if (hasToken(node, "get"))
16938
+ kind = "get";
16939
+ else if (hasToken(node, "set"))
16940
+ kind = "set";
16941
+ else
16942
+ kind = "method";
16943
+ }
16944
+ return { fn: node, name: name2, kind, anchor: node };
16945
+ }
16946
+ case "public_field_definition": {
16947
+ const value = node.childForFieldName("value");
16948
+ const name2 = keyName(node.childForFieldName("name"));
16949
+ if (!value || !name2 || !FUNCTION_VALUES.has(value.type))
16950
+ return null;
16951
+ return { fn: value, name: name2, kind: "field", anchor: node };
16952
+ }
16953
+ case "variable_declarator": {
16954
+ const value = node.childForFieldName("value");
16955
+ const id = node.childForFieldName("name");
16956
+ if (!value || !id || id.type !== "identifier" || !FUNCTION_VALUES.has(value.type)) {
16957
+ return null;
16958
+ }
16959
+ return {
16960
+ fn: value,
16961
+ name: id.text,
16962
+ kind: container ? "local" : "function",
16963
+ anchor: anchorOf(node)
16964
+ };
16965
+ }
16966
+ case "pair": {
16967
+ const value = node.childForFieldName("value");
16968
+ const name2 = keyName(node.childForFieldName("key"));
16969
+ if (!value || !name2 || !FUNCTION_VALUES.has(value.type))
16970
+ return null;
16971
+ return { fn: value, name: name2, kind: "prop", anchor: node };
16972
+ }
16973
+ default:
16974
+ return null;
16975
+ }
16976
+ }
16977
+ function paramCount(fn) {
16978
+ const list = fn.childForFieldName("parameters");
16979
+ if (list)
16980
+ return list.namedChildren.filter((c) => c.type !== "comment").length;
16981
+ return fn.childForFieldName("parameter") ? 1 : 0;
16982
+ }
16983
+ function docOf(anchor, source) {
16984
+ const comment = anchor.previousSibling;
16985
+ if (comment?.type !== "comment" || !comment.text.startsWith("/**"))
16986
+ return "";
16987
+ if (source.slice(comment.endIndex, anchor.startIndex).trim() !== "")
16988
+ return "";
16989
+ const lines = comment.text.slice(3, -2).split(`
16990
+ `).map((line) => line.replace(/^\s*\*\s?/, "").trim());
16991
+ const tag = lines.findIndex((line) => line.startsWith("@"));
16992
+ const text = (tag === -1 ? lines : lines.slice(0, tag)).filter((line) => line !== "");
16993
+ return text.join(" ").slice(0, DOC_LIMIT);
16994
+ }
16995
+ function buildUnit(d, container, source, file, minNodes) {
16996
+ const body2 = d.fn.childForFieldName("body");
16997
+ if (!body2)
16998
+ return null;
16999
+ const { labels, names } = normalise(body2);
17000
+ if (labels.length < minNodes)
17001
+ return null;
17002
+ const startLine = d.anchor.startPosition.row + 1;
17003
+ const endLine = d.fn.endPosition.row + 1;
17004
+ return {
17005
+ file,
17006
+ name: d.name,
17007
+ container,
17008
+ kind: d.kind,
17009
+ exported: d.anchor.type === "export_statement",
17010
+ startLine,
17011
+ endLine,
17012
+ loc: endLine - startLine + 1,
17013
+ params: paramCount(d.fn),
17014
+ async: hasToken(d.fn, "async"),
17015
+ nodes: labels.length,
17016
+ doc: docOf(d.anchor, source),
17017
+ exactHash: sha1(labels.join(" ")),
17018
+ textHash: sha1(source.slice(body2.startIndex, body2.endIndex).replace(/\s+/g, "")),
17019
+ startIndex: d.fn.startIndex,
17020
+ endIndex: d.fn.endIndex,
17021
+ labels,
17022
+ names
17023
+ };
17024
+ }
17025
+ function unitsOf(root, source, file, minNodes) {
17026
+ const units = [];
17027
+ const visit = (node, container) => {
17028
+ const found = declared(node, container);
17029
+ if (found) {
17030
+ const unit = buildUnit(found, container, source, file, minNodes);
17031
+ if (unit)
17032
+ units.push(unit);
17033
+ }
17034
+ let inner = container;
17035
+ if (CLASSES.has(node.type)) {
17036
+ inner = keyName(node.childForFieldName("name")) ?? (container || ANONYMOUS_CLASS);
17037
+ }
17038
+ for (const child of node.namedChildren)
17039
+ visit(child, inner);
17040
+ };
17041
+ visit(root, "");
17042
+ return units;
17043
+ }
17044
+
17045
+ // src/clones/index.ts
17046
+ var CLONES_HELP = `Usage: repo-tools clones [root] [--out=<dir>] [--config=<path>] [--src=<path>...]
17047
+ [--min-nodes=<n>] [--shingle=<n>] [--near=<f>]
17048
+ [--containment=<f>] [--containment-min=<n>]
17049
+ [--rare=<n>] [--min-shared=<n>]
17050
+
17051
+ List the functions of a TypeScript or JavaScript repository and find the functions
17052
+ that copy each other. The command only reads the repository. The result is a list
17053
+ of candidates: read both bodies of a pair before you change anything.
17054
+
17055
+ A function is a unit when it has a name and its body has at least --min-nodes
17056
+ labels. A label is one node of the body, with names, literals, types and comments
17057
+ erased. Units with the same label sequence are an exact group. Two units whose
17058
+ shingles (runs of --shingle labels) overlap enough are a near pair.
17059
+
17060
+ --out=<dir> Write inventory.jsonl and clones.json to this folder.
17061
+ Without it the command writes no file.
17062
+ --config=<path> The config file. The default is repo-tools.config.json at
17063
+ the root. The settings are in its "clones" section.
17064
+ --src=<path>... The folders to read. The word "auto" (the default) reads
17065
+ the files of the src area. The flag takes every argument
17066
+ up to the next flag.
17067
+ --min-nodes=<n> The smallest number of labels of a unit. Default 50.
17068
+ --shingle=<n> The labels in one shingle. Default 6.
17069
+ --near=<f> The smallest Jaccard score of a near pair. Default 0.6.
17070
+ --containment=<f> The smallest containment of a near pair. Default 0.9.
17071
+ --containment-min=<n> The smallest shingle count of the smaller unit for the
17072
+ containment rule. Default 20.
17073
+ --rare=<n> A shingle in more units than this does not pick candidate
17074
+ pairs. Default 60.
17075
+ --min-shared=<n> The rare shingles that a candidate pair must share.
17076
+ Default 3.
17077
+ --root=<path> The root. The default is the current folder.
17078
+ --help, -h Show this help.
17079
+
17080
+ A flag overrides the config file, and the config file overrides the default.
17081
+
17082
+ Exit codes: 0 when the run finished. 1 when the root holds no source file to read.
17083
+ 2 on a usage error, a bad config file or a root that is not a folder.
17084
+ `;
17085
+
17086
+ class NothingToRead extends Error {
17087
+ }
17088
+ var VALUE_FLAGS2 = {
17089
+ "--out": "out",
17090
+ "--min-nodes": "minNodes",
17091
+ "--shingle": "shingleLength",
17092
+ "--near": "nearThreshold",
17093
+ "--containment": "containmentThreshold",
17094
+ "--containment-min": "containmentMinShingles",
17095
+ "--rare": "rareShingleLimit",
17096
+ "--min-shared": "minSharedShingles"
17097
+ };
17098
+ var NUMERIC = new Set([
17099
+ "minNodes",
17100
+ "shingleLength",
17101
+ "nearThreshold",
17102
+ "containmentThreshold",
17103
+ "containmentMinShingles",
17104
+ "rareShingleLimit",
17105
+ "minSharedShingles"
17106
+ ]);
17107
+ function parseArgs2(argv) {
17108
+ const options = { root: ".", config: undefined, settings: {} };
17109
+ const seen = new Set;
17110
+ let rootSet = false;
17111
+ const setRoot = (value) => {
17112
+ if (rootSet)
17113
+ throw new Error("the root is set twice (use --root=<path> or one positional path)");
17114
+ rootSet = true;
17115
+ options.root = value;
17116
+ };
17117
+ const once = (flag) => {
17118
+ if (seen.has(flag))
17119
+ throw new Error(`flag ${flag} is set twice`);
17120
+ seen.add(flag);
17121
+ };
17122
+ for (let i2 = 0;i2 < argv.length; i2++) {
17123
+ const arg = argv[i2] ?? "";
17124
+ if (!arg.startsWith("-")) {
17125
+ setRoot(arg);
17126
+ continue;
17127
+ }
17128
+ const eq = arg.indexOf("=");
17129
+ const name2 = eq === -1 ? arg : arg.slice(0, eq);
17130
+ if (name2 === "--src") {
17131
+ once(name2);
17132
+ const paths = [];
17133
+ if (eq !== -1)
17134
+ paths.push(arg.slice(eq + 1));
17135
+ else
17136
+ while (i2 + 1 < argv.length && !(argv[i2 + 1] ?? "").startsWith("-")) {
17137
+ paths.push(argv[++i2] ?? "");
17138
+ }
17139
+ if (paths.length === 0 || paths.some((p) => p === "")) {
17140
+ throw new Error("flag --src needs a value: --src=<path>");
17141
+ }
17142
+ const only = paths.length === 1 ? paths[0] : undefined;
17143
+ options.settings.src = checkClonesFlag(name2, "src", only === "auto" ? "auto" : paths);
17144
+ continue;
17145
+ }
17146
+ if (name2 !== "--root" && name2 !== "--config" && !Object.hasOwn(VALUE_FLAGS2, name2)) {
17147
+ throw new Error(`unknown flag '${name2}' (see repo-tools clones --help)`);
17148
+ }
17149
+ once(name2);
17150
+ let value = eq === -1 ? undefined : arg.slice(eq + 1);
17151
+ if (eq === -1 && i2 + 1 < argv.length && !(argv[i2 + 1] ?? "").startsWith("-")) {
17152
+ value = argv[++i2];
17153
+ }
17154
+ if (value === undefined || value === "") {
17155
+ throw new Error(`flag ${name2} needs a value: ${name2}=<value>`);
17156
+ }
17157
+ if (name2 === "--root")
17158
+ setRoot(value);
17159
+ else if (name2 === "--config")
17160
+ options.config = value;
17161
+ else {
17162
+ const key = VALUE_FLAGS2[name2];
17163
+ const raw = NUMERIC.has(key) ? /^\d+(\.\d+)?$/.test(value) ? Number(value) : NaN : value;
17164
+ Object.assign(options.settings, { [key]: checkClonesFlag(name2, key, raw) });
17165
+ }
17166
+ }
17167
+ return options;
17168
+ }
17169
+ function folderPrefix(path) {
17170
+ const posix = path.replace(/\\/g, "/").replace(/^(\.\/)+/, "").replace(/\/+$/, "");
17171
+ return posix === "." ? "" : posix;
17172
+ }
17173
+ function selectFiles(root, src) {
17174
+ const found = discover(root, "typescript").filter((f) => !f.path.endsWith(".d.ts"));
17175
+ if (src === "auto")
17176
+ return found.filter((f) => f.area === "src").map((f) => f.path);
17177
+ const folders = src.map(folderPrefix);
17178
+ return found.filter((f) => folders.some((d) => d === "" || f.path === d || f.path.startsWith(`${d}/`))).map((f) => f.path);
17179
+ }
17180
+ async function readUnits(root, files, minNodes) {
17181
+ const units = [];
17182
+ let parseErrors = 0;
17183
+ for (const file of files) {
17184
+ const source = readSource(join22(root, file)).replace(/^\uFEFF/, "");
17185
+ const grammar = grammarFor(file);
17186
+ await loadGrammar(grammar);
17187
+ const tree = parserFor(grammar).parse(source);
17188
+ if (!tree) {
17189
+ parseErrors += 1;
17190
+ continue;
17191
+ }
17192
+ try {
17193
+ if (tree.rootNode.hasError)
17194
+ parseErrors += 1;
17195
+ units.push(...unitsOf(tree.rootNode, source, file, minNodes));
17196
+ } finally {
17197
+ tree.delete();
17198
+ }
17199
+ }
17200
+ return { units, parseErrors };
17201
+ }
17202
+ async function execute(root, config, io) {
17203
+ const files = selectFiles(root, config.src);
17204
+ if (files.length === 0) {
17205
+ throw new NothingToRead(config.src === "auto" ? "no TypeScript or JavaScript file in the src area (set clones.src or pass --src=<path>)" : "no TypeScript or JavaScript file below the listed folders");
17206
+ }
17207
+ const { units, parseErrors } = await readUnits(root, files, config.minNodes);
17208
+ const detection = detect(units, config);
17209
+ io.stdout(summaryText(config, { files: files.length, parseErrors }, units, detection));
17210
+ if (config.out !== null) {
17211
+ const out2 = resolve8(root, config.out);
17212
+ try {
17213
+ writeLf(join22(out2, "inventory.jsonl"), inventoryJsonl(units));
17214
+ writeLf(join22(out2, "clones.json"), clonesJson(config, detection));
17215
+ } catch {
17216
+ throw new Error(`cannot write the output folder '${config.out}'`);
17217
+ }
17218
+ const shown = config.out.replace(/\\/g, "/").replace(/\/+$/, "");
17219
+ io.stdout(`wrote ${shown}/inventory.jsonl
17220
+ wrote ${shown}/clones.json
17221
+ `);
17222
+ }
17223
+ return 0;
17224
+ }
17225
+ async function run4(argv, io) {
17226
+ let root;
17227
+ const stderr = (text) => io.stderr(root ? maskRoot(text, root) : text);
17228
+ try {
17229
+ const options = parseArgs2(argv);
17230
+ root = resolve8(options.root);
17231
+ if (!isDirectory(root))
17232
+ throw new Error("the root <root> is not an existing directory");
17233
+ const file = loadConfigSections(root, options.config).clones;
17234
+ return await execute(root, mergeClonesConfig(options.settings, file), io);
17235
+ } catch (error) {
17236
+ const message = error instanceof Error ? error.message : String(error);
17237
+ stderr(`repo-tools clones: ${message}
17238
+ `);
17239
+ return error instanceof NothingToRead ? 1 : 2;
17240
+ }
17241
+ }
17242
+
16570
17243
  // src/compress/index.ts
16571
17244
  import { existsSync as existsSync9, readdirSync as readdirSync3, readFileSync as readFileSync20, statSync as statSync8, writeFileSync as writeFileSync4 } from "node:fs";
16572
- import { basename as basename8, dirname as dirname8, extname as extname4, join as join22 } from "node:path";
17245
+ import { basename as basename8, dirname as dirname8, extname as extname4, join as join23 } from "node:path";
16573
17246
 
16574
17247
  // src/compress/formats.ts
16575
17248
  import { extname as extname3 } from "node:path";
@@ -17202,7 +17875,7 @@ a value, a missing input file and two inputs without --batch are errors. No file
17202
17875
  var defaultDeps = {
17203
17876
  readdir: (dir) => readdirSync3(dir, { withFileTypes: true })
17204
17877
  };
17205
- function parseArgs2(args2) {
17878
+ function parseArgs3(args2) {
17206
17879
  const o = {
17207
17880
  input: "",
17208
17881
  inputs: [],
@@ -17325,7 +17998,7 @@ function findFiles(dir, pattern, recursive, deps = defaultDeps) {
17325
17998
  const scan = (current) => {
17326
17999
  const entries = [...deps.readdir(current)].sort((a, b) => compareCodeUnits(a.name, b.name));
17327
18000
  for (const entry of entries) {
17328
- const full = join22(current, entry.name);
18001
+ const full = join23(current, entry.name);
17329
18002
  if (entry.isDirectory()) {
17330
18003
  if (recursive && !entry.name.startsWith(".") && entry.name !== "node_modules")
17331
18004
  scan(full);
@@ -17342,16 +18015,16 @@ function isDirectory2(path) {
17342
18015
  }
17343
18016
  function compactName(file) {
17344
18017
  const ext = extname4(file);
17345
- return join22(dirname8(file), `${basename8(file, ext)}.compact${ext}`);
18018
+ return join23(dirname8(file), `${basename8(file, ext)}.compact${ext}`);
17346
18019
  }
17347
18020
  function restoredName(file) {
17348
18021
  const base = basename8(file);
17349
18022
  const stripped = base.replace(/\.compact(?=\.[^.]*$|$)/i, "");
17350
- const output = stripped === base ? file : join22(dirname8(file), stripped);
18023
+ const output = stripped === base ? file : join23(dirname8(file), stripped);
17351
18024
  if (output !== file)
17352
18025
  return output;
17353
18026
  const ext = extname4(file);
17354
- return join22(dirname8(file), `${basename8(file, ext)}.restored${ext}`);
18027
+ return join23(dirname8(file), `${basename8(file, ext)}.restored${ext}`);
17355
18028
  }
17356
18029
  function assertCanRestore(output, o) {
17357
18030
  if (!o.dryRun && !o.yes && existsSync9(output)) {
@@ -17511,10 +18184,10 @@ Level: ${o.level}
17511
18184
  }
17512
18185
  return 0;
17513
18186
  }
17514
- async function run4(argv, io, deps = defaultDeps) {
18187
+ async function run5(argv, io, deps = defaultDeps) {
17515
18188
  let o;
17516
18189
  try {
17517
- o = parseArgs2(argv);
18190
+ o = parseArgs3(argv);
17518
18191
  } catch (error) {
17519
18192
  if (!(error instanceof UsageError))
17520
18193
  throw error;
@@ -17565,16 +18238,16 @@ async function run4(argv, io, deps = defaultDeps) {
17565
18238
 
17566
18239
  // src/docs/index.ts
17567
18240
  import { lstatSync as lstatSync5, readFileSync as readFileSync23, realpathSync as realpathSync3, writeFileSync as writeFileSync5 } from "node:fs";
17568
- import { basename as basename9, isAbsolute as isAbsolute4, join as join26, relative as relative5, resolve as resolve8, sep as sep4 } from "node:path";
18241
+ import { basename as basename9, isAbsolute as isAbsolute4, join as join27, relative as relative5, resolve as resolve9, sep as sep4 } from "node:path";
17569
18242
 
17570
18243
  // src/docs/analyse.ts
17571
18244
  import { readFileSync as readFileSync22 } from "node:fs";
17572
- import { join as join24 } from "node:path";
18245
+ import { join as join25 } from "node:path";
17573
18246
 
17574
18247
  // src/docs/discovery.ts
17575
18248
  import { spawnSync as spawnSync2 } from "node:child_process";
17576
18249
  import { existsSync as existsSync10, readdirSync as readdirSync4, readFileSync as readFileSync21, statSync as statSync9 } from "node:fs";
17577
- import { join as join23 } from "node:path";
18250
+ import { join as join24 } from "node:path";
17578
18251
  var PYTHON_SUFFIXES2 = new Set([".py"]);
17579
18252
  var TYPESCRIPT_SUFFIXES = new Set([".ts", ".tsx", ".js", ".jsx", ".mjs", ".cjs"]);
17580
18253
  var SKIP_DIRS2 = new Set([
@@ -17658,8 +18331,8 @@ function walkSource(root) {
17658
18331
  const rel = `${prefix}${entry.name}`;
17659
18332
  if (entry.isDirectory()) {
17660
18333
  if (!SKIP_DIRS2.has(entry.name))
17661
- go(join23(dir, entry.name), `${rel}/`);
17662
- } else if (isSource(rel) && statOk(join23(dir, entry.name))) {
18334
+ go(join24(dir, entry.name), `${rel}/`);
18335
+ } else if (isSource(rel) && statOk(join24(dir, entry.name))) {
17663
18336
  found.push(rel);
17664
18337
  }
17665
18338
  }
@@ -17675,7 +18348,7 @@ function statOk(path) {
17675
18348
  }
17676
18349
  }
17677
18350
  function loadExclusions(root) {
17678
- const file = join23(root, CONFIG_NAME);
18351
+ const file = join24(root, CONFIG_NAME);
17679
18352
  if (!existsSync10(file) || !statOk(file))
17680
18353
  return [];
17681
18354
  let data;
@@ -18287,9 +18960,9 @@ var EXPORT_TYPE_STAR = new RegExp(`(\\bexport${SPACE_CLASS}+)type(${SPACE_CLASS}
18287
18960
  function sanitise(source) {
18288
18961
  return source.replace(ABSTRACT_PROPERTY, "abstrac_$1").replace(EXPORT_TYPE_STAR, "$1 $2");
18289
18962
  }
18290
- var IDENTIFIERS = new Set(["identifier", "shorthand_property_identifier_pattern"]);
18963
+ var IDENTIFIERS2 = new Set(["identifier", "shorthand_property_identifier_pattern"]);
18291
18964
  function boundIdentifiers(node, text) {
18292
- if (IDENTIFIERS.has(node.type))
18965
+ if (IDENTIFIERS2.has(node.type))
18293
18966
  return [text(node)];
18294
18967
  const pattern = node.childForFieldName("pattern");
18295
18968
  if (pattern)
@@ -18307,7 +18980,7 @@ function boundIdentifiers(node, text) {
18307
18980
  case "rest_pattern":
18308
18981
  return into(node.namedChildren[0] ?? null);
18309
18982
  default: {
18310
- const first = node.namedChildren.find((c) => IDENTIFIERS.has(c.type));
18983
+ const first = node.namedChildren.find((c) => IDENTIFIERS2.has(c.type));
18311
18984
  return first ? [text(first)] : [];
18312
18985
  }
18313
18986
  }
@@ -18316,7 +18989,7 @@ function paramsOf2(node, text) {
18316
18989
  const list = node.childForFieldName("parameters");
18317
18990
  return (list?.namedChildren ?? []).flatMap((param) => boundIdentifiers(param, text));
18318
18991
  }
18319
- function anchorOf(node) {
18992
+ function anchorOf2(node) {
18320
18993
  let target = node;
18321
18994
  while (target.parent?.type === "export_statement")
18322
18995
  target = target.parent;
@@ -18473,7 +19146,7 @@ async function analyseTypeScript(path, source) {
18473
19146
  const text = (node) => source.slice(node.startIndex, node.endIndex);
18474
19147
  for (const node of tree.rootNode.descendantsOfType(DECLARATION_TYPES)) {
18475
19148
  const kind = DECLARATIONS[node.type];
18476
- const anchor = anchorOf(node);
19149
+ const anchor = anchorOf2(node);
18477
19150
  const lead = stubAnchor(anchor, source, text);
18478
19151
  const doc = leadingDoc(anchor, node, text);
18479
19152
  const dialect = dialectOf(doc);
@@ -18530,7 +19203,7 @@ async function analyseFile(root, rel) {
18530
19203
  const language = languageOf(rel);
18531
19204
  let source;
18532
19205
  try {
18533
- source = readSourceText2(join24(root, rel));
19206
+ source = readSourceText2(join25(root, rel));
18534
19207
  } catch (error) {
18535
19208
  const report = newReport(rel, language);
18536
19209
  report.error = `unreadable: ${readFailure(error)}`;
@@ -18547,7 +19220,7 @@ async function analyseRepo(root) {
18547
19220
  }
18548
19221
 
18549
19222
  // src/docs/report.ts
18550
- import { join as join25 } from "node:path";
19223
+ import { join as join26 } from "node:path";
18551
19224
  function percent3(part, whole) {
18552
19225
  if (whole === 0)
18553
19226
  return 100;
@@ -18659,8 +19332,8 @@ function symbolJson(s) {
18659
19332
  };
18660
19333
  }
18661
19334
  function writeOutputs(outDir, reports, stats, provenance, rootName) {
18662
- const markdown = join25(outDir, "COVERAGE.md");
18663
- const json = join25(outDir, "coverage.json");
19335
+ const markdown = join26(outDir, "COVERAGE.md");
19336
+ const json = join26(outDir, "coverage.json");
18664
19337
  writeReport(markdown, renderMarkdown(reports, stats, provenance, rootName));
18665
19338
  const payload = {
18666
19339
  summary: statsJson(stats),
@@ -18864,7 +19537,7 @@ Exit codes: 0 when the action succeeded and the gate passed. 1 when the gate
18864
19537
  failed, or when the root holds no source file to measure. 2 on a usage error,
18865
19538
  when the root is not a folder, or when ${CONFIG_NAME} is malformed.
18866
19539
  `;
18867
- var VALUE_FLAGS2 = {
19540
+ var VALUE_FLAGS3 = {
18868
19541
  "--root": "all",
18869
19542
  "--out": "scan",
18870
19543
  "--path": "stub",
@@ -18917,8 +19590,8 @@ function parseDocsArgs(argv) {
18917
19590
  while (i2 + 1 < argv.length && !(argv[i2 + 1] ?? "").startsWith("-"))
18918
19591
  options.paths.push(argv[++i2] ?? "");
18919
19592
  }
18920
- } else if (Object.hasOwn(VALUE_FLAGS2, name2)) {
18921
- forbid(name2, VALUE_FLAGS2[name2]);
19593
+ } else if (Object.hasOwn(VALUE_FLAGS3, name2)) {
19594
+ forbid(name2, VALUE_FLAGS3[name2]);
18922
19595
  let value = eq === -1 ? undefined : arg.slice(eq + 1);
18923
19596
  if (eq === -1 && i2 + 1 < argv.length && !(argv[i2 + 1] ?? "").startsWith("-"))
18924
19597
  value = argv[++i2];
@@ -19012,7 +19685,7 @@ async function runScan(root, options, io) {
19012
19685
  if (reports.length === 0)
19013
19686
  throw new NothingToMeasure(provenance);
19014
19687
  const stats = summarise(reports);
19015
- const outDir = resolve8(options.out ?? join26(root, "docs", "code-docs"));
19688
+ const outDir = resolve9(options.out ?? join27(root, "docs", "code-docs"));
19016
19689
  const written = writeOutputs(outDir, reports, stats, provenance, basename9(root));
19017
19690
  io.stdout(`scanned ${stats.filesScanned} file(s) via ${provenance}
19018
19691
  `);
@@ -19052,7 +19725,7 @@ async function runStub(root, options, io) {
19052
19725
  const targets = report.symbols.filter((s) => s.issues.some((i2) => i2.rule === "M1"));
19053
19726
  if (targets.length === 0)
19054
19727
  continue;
19055
- const file = join26(root, report.path);
19728
+ const file = join27(root, report.path);
19056
19729
  if (leavesRoot(root, file)) {
19057
19730
  io.stdout(` SKIPPED ${report.path}: the path is a link, or it resolves outside the root
19058
19731
  `);
@@ -19178,7 +19851,7 @@ async function runCheck(root, options, io) {
19178
19851
  `);
19179
19852
  return 1;
19180
19853
  }
19181
- async function run5(argv, io) {
19854
+ async function run6(argv, io) {
19182
19855
  if (wantsHelp(argv)) {
19183
19856
  io.stdout(DOCS_HELP);
19184
19857
  return 0;
@@ -19187,7 +19860,7 @@ async function run5(argv, io) {
19187
19860
  const stderr = (text) => io.stderr(root ? maskRoot(text, root) : text);
19188
19861
  try {
19189
19862
  const options = parseDocsArgs(argv);
19190
- root = resolve8(options.root);
19863
+ root = resolve9(options.root);
19191
19864
  if (!isDirectory(root))
19192
19865
  throw new Error("the root <root> is not an existing directory");
19193
19866
  assertReadableLanguage(root);
@@ -19206,14 +19879,14 @@ async function run5(argv, io) {
19206
19879
 
19207
19880
  // src/query/index.ts
19208
19881
  import { existsSync as existsSync12, statSync as statSync10 } from "node:fs";
19209
- import { resolve as resolve9 } from "node:path";
19882
+ import { resolve as resolve10 } from "node:path";
19210
19883
 
19211
19884
  // src/query/args.ts
19212
19885
  var MODE_FLAGS = {
19213
19886
  "--emit": { name: "emit" },
19214
19887
  "--check-browser-safety": { name: "check-browser-safety" }
19215
19888
  };
19216
- var VALUE_FLAGS3 = new Set(["--root", "--config", "--out", "--node-runtime"]);
19889
+ var VALUE_FLAGS4 = new Set(["--root", "--config", "--out", "--node-runtime"]);
19217
19890
  var COMMAND_USAGE = {
19218
19891
  dependents: "<file>",
19219
19892
  "symbol-users": "<symbol>",
@@ -19294,7 +19967,7 @@ function parseQueryArgs(argv, cwd) {
19294
19967
  mode = modeCommand;
19295
19968
  continue;
19296
19969
  }
19297
- if (!VALUE_FLAGS3.has(name2)) {
19970
+ if (!VALUE_FLAGS4.has(name2)) {
19298
19971
  throw new Error(`unknown flag '${name2}' (see repo-tools query --help)`);
19299
19972
  }
19300
19973
  if (value === undefined || value === "") {
@@ -19783,12 +20456,12 @@ report, a path that is not a file of the graph, a browser-safety command on a
19783
20456
  graph that is not TypeScript/JavaScript, or a browser-safety leak with
19784
20457
  --check-browser-safety.
19785
20458
  `;
19786
- async function run6(argv, io) {
20459
+ async function run7(argv, io) {
19787
20460
  let root;
19788
20461
  const stderr = (text) => io.stderr(root ? maskRoot(text, root) : text);
19789
20462
  try {
19790
20463
  const options = parseQueryArgs(argv, process.cwd());
19791
- root = resolve9(options.root);
20464
+ root = resolve10(options.root);
19792
20465
  if (!(existsSync12(root) && statSync10(root).isDirectory())) {
19793
20466
  throw new Error("the root <root> is not an existing directory");
19794
20467
  }
@@ -19824,7 +20497,7 @@ async function run6(argv, io) {
19824
20497
 
19825
20498
  // src/ste/index.ts
19826
20499
  import { existsSync as existsSync13, lstatSync as lstatSync6, readdirSync as readdirSync5, readFileSync as readFileSync26, statSync as statSync11 } from "node:fs";
19827
- import { basename as basename10, join as join28 } from "node:path";
20500
+ import { basename as basename10, join as join29 } from "node:path";
19828
20501
 
19829
20502
  // src/ste/markdown.ts
19830
20503
  function stripNoise(text) {
@@ -19879,7 +20552,7 @@ function wordCount(sentence) {
19879
20552
  const s = sentence.replace(CODE_SPAN, "X").replace(LINK, "$1").replace(EMPHASIS_CHARS, "").replace(LIST_MARKER, "");
19880
20553
  return pySplit(s).filter((w) => Array.from(w).some(isAlnum)).length;
19881
20554
  }
19882
- function join27(parts2, start2) {
20555
+ function join28(parts2, start2) {
19883
20556
  const chunks = [];
19884
20557
  const lineMap = [];
19885
20558
  parts2.forEach(([n, body2], i2) => {
@@ -19909,7 +20582,7 @@ function paragraphs(text) {
19909
20582
  for (const [n, line] of stripNoise(text)) {
19910
20583
  const startsItem = LIST_ITEM.test(line);
19911
20584
  if ((n !== prevLine + 1 || startsItem) && para.length > 0) {
19912
- blocks.push(join27(para, paraStart));
20585
+ blocks.push(join28(para, paraStart));
19913
20586
  para = [];
19914
20587
  }
19915
20588
  if (para.length === 0)
@@ -19918,7 +20591,7 @@ function paragraphs(text) {
19918
20591
  prevLine = n;
19919
20592
  }
19920
20593
  if (para.length > 0)
19921
- blocks.push(join27(para, paraStart));
20594
+ blocks.push(join28(para, paraStart));
19922
20595
  return blocks;
19923
20596
  }
19924
20597
  function checkMarkdown(name2, text) {
@@ -19975,8 +20648,8 @@ identifiers are not checked. A file whose first five lines hold
19975
20648
  --help, -h Show this help.
19976
20649
 
19977
20650
  Exit codes: 0 when no finding (Markdown mode) or always (--prose mode). 1 when a
19978
- Markdown finding exists. 2 on a usage error, when no Markdown file was examined,
19979
- or when a file is not UTF-8.
20651
+ Markdown finding exists. 2 on a usage error, when a path does not exist, when no
20652
+ Markdown file was examined, or when a file is not UTF-8.
19980
20653
  `;
19981
20654
  var HISTORICAL_MARKER = "<!-- ste:historical-record -->";
19982
20655
  function readText(path) {
@@ -20005,7 +20678,7 @@ function targetsFor(path) {
20005
20678
  return;
20006
20679
  }
20007
20680
  for (const entry of entries) {
20008
- const full = join28(dir, entry.name);
20681
+ const full = join29(dir, entry.name);
20009
20682
  if (entry.isDirectory()) {
20010
20683
  if (entry.name !== ".pytest_cache")
20011
20684
  walk(full, [...parts2, entry.name]);
@@ -20018,7 +20691,7 @@ function targetsFor(path) {
20018
20691
  };
20019
20692
  if (existsSync13(path) && lstatSync6(path).isDirectory())
20020
20693
  walk(path, []);
20021
- return found.sort(comparePaths).map((parts2) => join28(path, ...parts2));
20694
+ return found.sort(comparePaths).map((parts2) => join29(path, ...parts2));
20022
20695
  }
20023
20696
  function isHistoricalRecord(path) {
20024
20697
  let text;
@@ -20049,7 +20722,7 @@ function runProse(args2, io) {
20049
20722
  `);
20050
20723
  return 0;
20051
20724
  }
20052
- async function run7(argv, io) {
20725
+ async function run8(argv, io) {
20053
20726
  if (argv[0] === "--prose")
20054
20727
  return runProse(argv.slice(1), io);
20055
20728
  if (argv.length === 0) {
@@ -20060,7 +20733,10 @@ async function run7(argv, io) {
20060
20733
  let total = 0;
20061
20734
  let checked = 0;
20062
20735
  let skipped = 0;
20736
+ const missing = [];
20063
20737
  for (const arg of argv) {
20738
+ if (!existsSync13(arg))
20739
+ missing.push(arg);
20064
20740
  for (const target of targetsFor(arg)) {
20065
20741
  const name2 = basename10(target);
20066
20742
  if (isHistoricalRecord(target)) {
@@ -20093,12 +20769,16 @@ async function run7(argv, io) {
20093
20769
  const tail = skipped > 0 ? `; ${skipped} skipped as historical record(s)` : "";
20094
20770
  io.stdout(`
20095
20771
  TOTAL: ${total} problem(s) in ${checked} file(s)${tail}
20772
+ `);
20773
+ for (const path of missing)
20774
+ io.stderr(`ERROR: path does not exist: ${path}
20096
20775
  `);
20097
20776
  if (checked === 0) {
20098
20777
  io.stderr(`ERROR: no Markdown file was examined. This is a failure, not a clean result -- check the path(s) given.
20099
20778
  `);
20100
- return 2;
20101
20779
  }
20780
+ if (missing.length > 0 || checked === 0)
20781
+ return 2;
20102
20782
  return total > 0 ? 1 : 0;
20103
20783
  }
20104
20784
 
@@ -20124,7 +20804,7 @@ ${MAP_HELP}`,
20124
20804
  docs: {
20125
20805
  summary: "Measure, stub and gate the doc comments of TypeScript, JavaScript and Python source.",
20126
20806
  help: DOCS_HELP,
20127
- run: run5
20807
+ run: run6
20128
20808
  },
20129
20809
  chunk: {
20130
20810
  summary: "Split a large file into chunks, merge the chunks back, or show changed chunks.",
@@ -20134,17 +20814,22 @@ ${MAP_HELP}`,
20134
20814
  compress: {
20135
20815
  summary: "Write a compact copy of a file for a model context, or restore it.",
20136
20816
  help: HELP,
20137
- run: run4
20817
+ run: run5
20138
20818
  },
20139
20819
  query: {
20140
20820
  summary: "Answer structural questions from the depgraph reports, or write derived reports.",
20141
20821
  help: QUERY_HELP,
20142
- run: run6
20822
+ run: run7
20143
20823
  },
20144
20824
  ste: {
20145
20825
  summary: "Check Markdown or docstring prose against Simplified Technical English.",
20146
20826
  help: STE_HELP,
20147
- run: run7
20827
+ run: run8
20828
+ },
20829
+ clones: {
20830
+ summary: "List the functions of a repository and find the functions that copy each other.",
20831
+ help: CLONES_HELP,
20832
+ run: run4
20148
20833
  }
20149
20834
  };
20150
20835
  var SUBCOMMANDS = Object.keys(REGISTRY);
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@danielsimonjr/repo-tools",
3
- "version": "2.2.6",
3
+ "version": "2.3.0",
4
4
  "description": "One CLI for repository analysis: dependency graphs creation and query with per-export API facts, file chunking, and context compression. TypeScript on Bun, compiled to a single executable.",
5
5
  "type": "module",
6
6
  "license": "MIT",