@danielsimonjr/repo-tools 2.2.6 → 2.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +38 -0
- package/README.md +37 -2
- package/dist/cli.js +752 -67
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -6,6 +6,44 @@ All notable changes to this project are recorded in this file. The format follow
|
|
|
6
6
|
|
|
7
7
|
## [Unreleased]
|
|
8
8
|
|
|
9
|
+
## [2.3.0] - 2026-10-22
|
|
10
|
+
|
|
11
|
+
### Added
|
|
12
|
+
|
|
13
|
+
- `repo-tools clones` lists the functions of a TypeScript or JavaScript repository and finds the functions that copy
|
|
14
|
+
each other. It reads the source and writes files only when `--out` or the `clones.out` config key names a folder.
|
|
15
|
+
A unit is a named function with at least 50 labels (`--min-nodes`). The command reports exact groups (the same label
|
|
16
|
+
sequence) and near pairs (shared runs of labels, scored by Jaccard and containment). It prints a summary, and
|
|
17
|
+
with an output folder it writes `inventory.jsonl` and `clones.json`. Every limit is a flag and a key of the new
|
|
18
|
+
`clones` section of `repo-tools.config.json`. A flag overrides the file, and the file overrides the default.
|
|
19
|
+
Exit code 0 means the run finished, 1 means the root holds no source file to read, and 2 means a usage or config
|
|
20
|
+
error. `docs/design.md` section 17 gives the rules.
|
|
21
|
+
- `docs/parity-clones.md` compares the command with the prototype of the memoryjs duplicate-function audit on 292
|
|
22
|
+
files. 1,368 units are the same. The command finds all 20 exact groups of the prototype. Every unit that only one
|
|
23
|
+
side reports lies at the cutoff of 50 labels.
|
|
24
|
+
- The smoke test runs `clones` on a small repository that holds a planted copy, so the shipped bundle proves the
|
|
25
|
+
command and its tree-sitter loader.
|
|
26
|
+
|
|
27
|
+
## [2.2.7] - 2026-10-22
|
|
28
|
+
|
|
29
|
+
### Fixed
|
|
30
|
+
|
|
31
|
+
- `repo-tools ste` exits 2 when any path in the argument list does not exist, and prints `ERROR: path does not exist:
|
|
32
|
+
<path>` on stderr for each such path. Before, a mistyped path among valid paths was ignored and the run exited 0.
|
|
33
|
+
The report of each valid path is still printed.
|
|
34
|
+
- The warning that `repo-tools map` writes for a `package.json` entry that does not resolve is shorter (15 words).
|
|
35
|
+
It had 30 words, so `repo-tools ste` flagged the generated `FILE_INVENTORY.md`. A test now checks the word limit
|
|
36
|
+
of every warning and that the generated report passes `repo-tools ste`.
|
|
37
|
+
- `repo-tools map` reads a folder named `test/` as a tests folder, as version 1.x did. Before, only `tests/` counted,
|
|
38
|
+
so a file such as `test/setup.ts` got the area `src` in `FILE_INVENTORY.md` and counted as a source file in
|
|
39
|
+
`TEST_COVERAGE.md`. The disposition legend and the `--tests` message name both folders.
|
|
40
|
+
- The `TEST_COVERAGE.md` note on the coverage policy no longer links `COVERAGE_POLICY.md`, a file that the tool does
|
|
41
|
+
not write, and no longer names mathjs or AssemblyScript.
|
|
42
|
+
- The `Expected test` hint in `TEST_COVERAGE.md` names `tests/unit/<module>/`. It appears only when the repository
|
|
43
|
+
keeps tests in `tests/unit/`. Other repositories get the plain list of untested files.
|
|
44
|
+
- The text of `duplicate-symbols.md` and of the `duplicate-symbols.json` note no longer names `Bucket C` or
|
|
45
|
+
`AssemblyScript`.
|
|
46
|
+
|
|
9
47
|
## [2.2.6] - 2026-10-22
|
|
10
48
|
|
|
11
49
|
### Fixed
|
package/README.md
CHANGED
|
@@ -11,13 +11,16 @@
|
|
|
11
11
|
| `chunk` | Splits a large file into chunks, merges the chunks back, and shows which chunks changed. |
|
|
12
12
|
| `compress` | Writes a compact copy of a file for a model context, and restores it. |
|
|
13
13
|
| `query` | Answers structural questions from the graph of `map`, and writes two derived reports. |
|
|
14
|
+
| `clones` | Lists the functions of a TypeScript or JavaScript repository, and finds the functions that copy each other. |
|
|
14
15
|
| `ste` | Checks Markdown or docstring prose against Simplified Technical English. |
|
|
15
16
|
|
|
16
|
-
Status: version 2.
|
|
17
|
+
Status: version 2.3.0.
|
|
17
18
|
|
|
18
19
|
- `map`, `check` and `query` read their settings from `repo-tools.config.json` in the root. The
|
|
19
20
|
section of `map` is `map` (its old name, `depgraph`, is also read). `--config=<file>` makes any
|
|
20
21
|
of the three read another file.
|
|
22
|
+
- `clones` reads its settings from the `clones` section of the same file. A flag overrides the
|
|
23
|
+
file, and the file overrides the default.
|
|
21
24
|
- `repo-tools depgraph` prints one deprecation line and runs `map`. It accepts the 1.x
|
|
22
25
|
scan-scope flags (`--src`, `--tests`, `--exclude`, `--also-exclude`, `--all`,
|
|
23
26
|
`--reachable-only`, `--include-tests`) with a warning, because they have no effect in 2.0.0.
|
|
@@ -41,7 +44,7 @@ repo-tools --version
|
|
|
41
44
|
|
|
42
45
|
The executable holds the tool and the Bun runtime. It needs no Bun, Node or `node_modules`.
|
|
43
46
|
Download the file for your platform and `SHA256SUMS` from the
|
|
44
|
-
[GitHub release](https://github.com/danielsimonjr/repo-tools/releases/tag/v2.
|
|
47
|
+
[GitHub release](https://github.com/danielsimonjr/repo-tools/releases/tag/v2.3.0):
|
|
45
48
|
|
|
46
49
|
| Platform | File |
|
|
47
50
|
|---|---|
|
|
@@ -311,6 +314,38 @@ repo-tools query --emit
|
|
|
311
314
|
the config key `query.out`, else from `map.out`, else from `docs/architecture`. Run
|
|
312
315
|
`repo-tools query --help` for every option.
|
|
313
316
|
|
|
317
|
+
## Find the functions that copy each other
|
|
318
|
+
|
|
319
|
+
`repo-tools clones` lists the functions of a TypeScript or JavaScript repository. It finds the
|
|
320
|
+
functions that copy each other. The command only reads the repository. It writes files only when
|
|
321
|
+
you name an output folder.
|
|
322
|
+
|
|
323
|
+
```sh
|
|
324
|
+
repo-tools clones . # prints the summary and writes no file
|
|
325
|
+
repo-tools clones . --src=src --out=reports/clones
|
|
326
|
+
repo-tools clones . --min-nodes=30 --near=0.7
|
|
327
|
+
```
|
|
328
|
+
|
|
329
|
+
A function is a unit when it has a name and its body has at least 50 labels. A label is one node
|
|
330
|
+
of the body. Names, literals, types and comments give no detail, so a copy with other names still
|
|
331
|
+
matches. The command finds two kinds of result:
|
|
332
|
+
|
|
333
|
+
| Result | Meaning |
|
|
334
|
+
|---|---|
|
|
335
|
+
| Exact group | Units with the same label sequence. |
|
|
336
|
+
| Near pair | Two units that share enough runs of 6 labels. |
|
|
337
|
+
|
|
338
|
+
`--src=<path>...` names the folders to read. The default `auto` reads the files of the `src` area.
|
|
339
|
+
`--out=<dir>` writes `inventory.jsonl` (one line for each unit) and `clones.json` (the groups and the
|
|
340
|
+
pairs) into a folder. Give the folder as a path relative to the root. Run `repo-tools clones --help` for every limit.
|
|
341
|
+
|
|
342
|
+
Exit codes: 0 when the run finished, even if it found no copy. 1 when the root holds no source file
|
|
343
|
+
to read. 2 on a usage error, a bad config file or a root that is not a folder.
|
|
344
|
+
|
|
345
|
+
The result is a list of candidates. Read both bodies of a pair before you change anything.
|
|
346
|
+
`docs/design.md` (section 17) gives the rules. `docs/parity-clones.md` records the comparison with the
|
|
347
|
+
prototype of the memoryjs audit.
|
|
348
|
+
|
|
314
349
|
## Develop
|
|
315
350
|
|
|
316
351
|
| Command | Job |
|
package/dist/cli.js
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
// package.json
|
|
3
3
|
var package_default = {
|
|
4
4
|
name: "@danielsimonjr/repo-tools",
|
|
5
|
-
version: "2.
|
|
5
|
+
version: "2.3.0",
|
|
6
6
|
description: "One CLI for repository analysis: dependency graphs creation and query with per-export API facts, file chunking, and context compression. TypeScript on Bun, compiled to a single executable.",
|
|
7
7
|
type: "module",
|
|
8
8
|
license: "MIT",
|
|
@@ -851,6 +851,16 @@ function checkValue(key, kind, value, fail) {
|
|
|
851
851
|
return text(value, "a non-empty string");
|
|
852
852
|
case "string-or-null":
|
|
853
853
|
return value === null ? null : text(value, "a non-empty string or null");
|
|
854
|
+
case "count":
|
|
855
|
+
if (typeof value !== "number" || !Number.isInteger(value) || value < 1) {
|
|
856
|
+
throw fail(`'${key}' must be a whole number of 1 or more`);
|
|
857
|
+
}
|
|
858
|
+
return value;
|
|
859
|
+
case "fraction":
|
|
860
|
+
if (typeof value !== "number" || !(value >= 0 && value <= 1)) {
|
|
861
|
+
throw fail(`'${key}' must be a number from 0 to 1`);
|
|
862
|
+
}
|
|
863
|
+
return value;
|
|
854
864
|
}
|
|
855
865
|
}
|
|
856
866
|
function checkSection(prefix, object, keys, fail) {
|
|
@@ -867,6 +877,20 @@ var QUERY_KEYS = {
|
|
|
867
877
|
out: "path",
|
|
868
878
|
nodeRuntimes: "names"
|
|
869
879
|
};
|
|
880
|
+
var CLONES_KEYS = {
|
|
881
|
+
src: "src",
|
|
882
|
+
out: "path-or-null",
|
|
883
|
+
minNodes: "count",
|
|
884
|
+
shingleLength: "count",
|
|
885
|
+
nearThreshold: "fraction",
|
|
886
|
+
containmentThreshold: "fraction",
|
|
887
|
+
containmentMinShingles: "count",
|
|
888
|
+
rareShingleLimit: "count",
|
|
889
|
+
minSharedShingles: "count"
|
|
890
|
+
};
|
|
891
|
+
function checkClonesFlag(label, key, value) {
|
|
892
|
+
return checkValue(label, CLONES_KEYS[key], value, (message) => new Error(message));
|
|
893
|
+
}
|
|
870
894
|
function parseConfigSections(parsed, label) {
|
|
871
895
|
const fail = (message) => new Error(`config ${label}: ${message}`);
|
|
872
896
|
if (!isObject(parsed))
|
|
@@ -874,7 +898,7 @@ function parseConfigSections(parsed, label) {
|
|
|
874
898
|
for (const key of Object.keys(parsed)) {
|
|
875
899
|
if (key === "$schema")
|
|
876
900
|
continue;
|
|
877
|
-
if (key !== "depgraph" && key !== "map" && key !== "query") {
|
|
901
|
+
if (key !== "depgraph" && key !== "map" && key !== "query" && key !== "clones") {
|
|
878
902
|
throw fail(`unknown key '${key}'`);
|
|
879
903
|
}
|
|
880
904
|
}
|
|
@@ -882,7 +906,7 @@ function parseConfigSections(parsed, label) {
|
|
|
882
906
|
throw fail("use the section 'map' or its old name 'depgraph', not both");
|
|
883
907
|
}
|
|
884
908
|
const sectionName = parsed.map !== undefined ? "map" : "depgraph";
|
|
885
|
-
const sections = { depgraph: {}, query: {} };
|
|
909
|
+
const sections = { depgraph: {}, query: {}, clones: {} };
|
|
886
910
|
const section = parsed[sectionName];
|
|
887
911
|
if (section !== undefined) {
|
|
888
912
|
if (!isObject(section))
|
|
@@ -901,6 +925,12 @@ function parseConfigSections(parsed, label) {
|
|
|
901
925
|
throw fail("'query' must be an object");
|
|
902
926
|
sections.query = checkSection("query.", query, QUERY_KEYS, fail);
|
|
903
927
|
}
|
|
928
|
+
const clones = parsed.clones;
|
|
929
|
+
if (clones !== undefined) {
|
|
930
|
+
if (!isObject(clones))
|
|
931
|
+
throw fail("'clones' must be an object");
|
|
932
|
+
sections.clones = checkSection("clones.", clones, CLONES_KEYS, fail);
|
|
933
|
+
}
|
|
904
934
|
return sections;
|
|
905
935
|
}
|
|
906
936
|
function loadConfigSections(root, configPath) {
|
|
@@ -911,7 +941,7 @@ function loadConfigSections(root, configPath) {
|
|
|
911
941
|
const label = `<root>/${rel.replace(/\\/g, "/")}`;
|
|
912
942
|
const path = resolveUnderRoot(root, rel);
|
|
913
943
|
if (configPath === undefined && !existsSync2(path))
|
|
914
|
-
return { depgraph: {}, query: {} };
|
|
944
|
+
return { depgraph: {}, query: {}, clones: {} };
|
|
915
945
|
let text;
|
|
916
946
|
try {
|
|
917
947
|
text = readFileSync(path, "utf8");
|
|
@@ -974,7 +1004,7 @@ function mergeDepgraphConfig(cli, file) {
|
|
|
974
1004
|
// src/depgraph/args.ts
|
|
975
1005
|
var SCAN_SCOPE_FLAGS = {
|
|
976
1006
|
"--src": "the census finds the source files itself; the area of a file comes from its path",
|
|
977
|
-
"--tests": "a test file is found by its name or its tests/ folder",
|
|
1007
|
+
"--tests": "a test file is found by its name or by its test/ or tests/ folder",
|
|
978
1008
|
"--exclude": "the census reads the files that git tracks; list a file in .gitignore instead",
|
|
979
1009
|
"--also-exclude": "the census reads the files that git tracks; list a file in .gitignore instead",
|
|
980
1010
|
"--all": "the graph always holds every file; the disposition of each file shows reachability",
|
|
@@ -1372,7 +1402,7 @@ var TYPE_DUP_CATEGORIES = [
|
|
|
1372
1402
|
{ cat: "type", key: "types" },
|
|
1373
1403
|
{ cat: "enum", key: "enums" }
|
|
1374
1404
|
];
|
|
1375
|
-
var DUPLICATE_SYMBOLS_NOTE = "This report groups names by OWN definition, not by call graph, then classifies each " + "flagged name (see DupEntryTag): TRUE_DUPLICATE (the actionable merge targets), " + "DISPATCH_VARIANT (>=2 mathTyped(...) registrations of the same public name — distinct " + "dispatch surfaces,
|
|
1405
|
+
var DUPLICATE_SYMBOLS_NOTE = "This report groups names by OWN definition, not by call graph, then classifies each " + "flagged name (see DupEntryTag): TRUE_DUPLICATE (the actionable merge targets), " + "DISPATCH_VARIANT (>=2 mathTyped(...) registrations of the same public name — distinct " + "dispatch surfaces, not copy-paste bodies), " + "ALIAS_DELEGATION (a const-alias forward to an imported symbol, not an independent body — " + "excluded once fewer than 2 real bodies remain), and ALLOWLISTED (matches " + "duplicate-allowlist.json: for example hot-path is* guards, mirrors in another language, " + "per-package VERSION strings). NOT detected: same-file typed-dispatch overload polymorphism for " + "different argument shapes within one registration — a human still triages TRUE_DUPLICATE " + "entries using the defining files + public flags before merging anything.";
|
|
1376
1406
|
var DUP_ENTRY_TAG_SORT_ORDER = {
|
|
1377
1407
|
TRUE_DUPLICATE: 0,
|
|
1378
1408
|
DISPATCH_VARIANT: 1,
|
|
@@ -6378,7 +6408,7 @@ function classifyArea(relPath) {
|
|
|
6378
6408
|
}
|
|
6379
6409
|
if (CONFIG_PATTERN.test(relPath))
|
|
6380
6410
|
return "config";
|
|
6381
|
-
if (isTestFile || /(^|\/)tests
|
|
6411
|
+
if (isTestFile || /(^|\/)tests?\//.test(relPath))
|
|
6382
6412
|
return "tests";
|
|
6383
6413
|
if (/^(bench|benchmarks)\//.test(relPath) || /\.bench\.\w+$/.test(relPath))
|
|
6384
6414
|
return "benchmarks";
|
|
@@ -12435,7 +12465,7 @@ function packageJsonRoots(root, known) {
|
|
|
12435
12465
|
continue;
|
|
12436
12466
|
const mapped = mapDistToSrc(spec, known) ?? launcherEntry(root, spec, known) ?? declaredSource(spec, known);
|
|
12437
12467
|
if (mapped === null) {
|
|
12438
|
-
warnings.push(`package.json entry ${pyRepr(spec)} did not resolve to a known source file
|
|
12468
|
+
warnings.push(`package.json entry ${pyRepr(spec)} did not resolve to a known source file. The entry is ignored.`);
|
|
12439
12469
|
continue;
|
|
12440
12470
|
}
|
|
12441
12471
|
if (!roots.includes(mapped))
|
|
@@ -13631,7 +13661,7 @@ function generateTestCoverageMarkdown(coverage, options = {}) {
|
|
|
13631
13661
|
lines.push("");
|
|
13632
13662
|
lines.push(...basisNote);
|
|
13633
13663
|
const policyPath = options.policyPath ?? "docs/architecture/coverage-policy.json";
|
|
13634
|
-
lines.push("> The raw figure counts every source file the
|
|
13664
|
+
lines.push("> The raw figure counts every source file the map finds, including files that no test " + "imports directly by design (for example generated code, type-only barrels and code in " + `another language). The **effective** figure excludes the files that \`${policyPath}\` ` + "classifies, so it shows the active code only.");
|
|
13635
13665
|
lines.push("");
|
|
13636
13666
|
lines.push("### Untested-file breakdown by category");
|
|
13637
13667
|
lines.push("");
|
|
@@ -14227,7 +14257,7 @@ function emitTestCoverage(graph, root, outDir, options = {}) {
|
|
|
14227
14257
|
const jsonPath = join16(outDir, "test-coverage.json");
|
|
14228
14258
|
const namespaceImportBasis = !importsNameSymbols(graph.language);
|
|
14229
14259
|
const markdown = generateTestCoverageMarkdown(coverage, {
|
|
14230
|
-
expectedTestHint: graph.language === "typescript",
|
|
14260
|
+
expectedTestHint: graph.language === "typescript" && tests.some((t) => t.path.startsWith("tests/unit/")),
|
|
14231
14261
|
policyPath: options.policyNote ?? "docs/architecture/coverage-policy.json",
|
|
14232
14262
|
namespaceImportBasis
|
|
14233
14263
|
});
|
|
@@ -14287,17 +14317,14 @@ function generateDuplicateSymbolsMarkdown(report) {
|
|
|
14287
14317
|
md += "Each name carries one tag (see `DupEntryTag`), so the actionable subset is clear:\n\n";
|
|
14288
14318
|
md += "- `TRUE_DUPLICATE`: a real merge target.\n";
|
|
14289
14319
|
md += "- `DISPATCH_VARIANT`: two or more `mathTyped(...)` registrations of the same public name. ";
|
|
14290
|
-
md +=
|
|
14291
|
-
md += `not copy-paste bodies.
|
|
14320
|
+
md += `The registrations are distinct dispatch surfaces, not copy-paste bodies.
|
|
14292
14321
|
`;
|
|
14293
14322
|
md += "- `ALIAS_DELEGATION`: a `const X = importedY` forward. ";
|
|
14294
14323
|
md += `The entry is excluded once fewer than two real bodies remain.
|
|
14295
14324
|
`;
|
|
14296
14325
|
md += "- `ALLOWLISTED`: the name matches `duplicate-allowlist.json`. ";
|
|
14297
|
-
md += "Examples are hot-path `is*` guards,
|
|
14298
|
-
md += `strings
|
|
14299
|
-
|
|
14300
|
-
`;
|
|
14326
|
+
md += "Examples are hot-path `is*` guards, mirrors in another language and per-package ";
|
|
14327
|
+
md += "`VERSION` strings.\n\n";
|
|
14301
14328
|
md += `> **Note:** ${report.note}
|
|
14302
14329
|
|
|
14303
14330
|
`;
|
|
@@ -14313,7 +14340,7 @@ function generateDuplicateSymbolsMarkdown(report) {
|
|
|
14313
14340
|
|
|
14314
14341
|
`;
|
|
14315
14342
|
md += renderTaggedSection("TRUE_DUPLICATE — actionable merge targets", report.runtime, "TRUE_DUPLICATE");
|
|
14316
|
-
md += renderTaggedSection("DISPATCH_VARIANT — distinct public typed-dispatch surfaces
|
|
14343
|
+
md += renderTaggedSection("DISPATCH_VARIANT — distinct public typed-dispatch surfaces", report.runtime, "DISPATCH_VARIANT");
|
|
14317
14344
|
md += renderTaggedSection("ALIAS_DELEGATION — const-alias forwards (not independent bodies)", report.runtime, "ALIAS_DELEGATION");
|
|
14318
14345
|
md += renderTaggedSection("ALLOWLISTED — accepted layering (see duplicate-allowlist.json)", report.runtime, "ALLOWLISTED");
|
|
14319
14346
|
md += `## Type duplicates (lower priority)
|
|
@@ -14332,7 +14359,7 @@ var DISPOSITION_LEGEND = [
|
|
|
14332
14359
|
["build-entry", "A `src`-area file that is an entry root."],
|
|
14333
14360
|
["test-only", "A `src`-area file that only a test reaches."],
|
|
14334
14361
|
["orphan", "A `src`-area file that nothing reaches. Delete it, or wire it to a root."],
|
|
14335
|
-
["test", "A file in a `tests/` folder, or a `*.test.ts` or `*.spec.ts` file."],
|
|
14362
|
+
["test", "A file in a `test/` or `tests/` folder, or a `*.test.ts` or `*.spec.ts` file."],
|
|
14336
14363
|
["tool", "A file in a `tools/` or `scripts/` folder outside `src/`."],
|
|
14337
14364
|
["config", "A `*.config.*` file of TypeScript or JavaScript."],
|
|
14338
14365
|
["example", "A file in `examples/` or `docs/`."],
|
|
@@ -16567,9 +16594,655 @@ ${CHUNK_HELP}`);
|
|
|
16567
16594
|
}
|
|
16568
16595
|
}
|
|
16569
16596
|
|
|
16597
|
+
// src/clones/index.ts
|
|
16598
|
+
import { join as join22, resolve as resolve8 } from "node:path";
|
|
16599
|
+
|
|
16600
|
+
// src/clones/detect.ts
|
|
16601
|
+
function fnv1a(text) {
|
|
16602
|
+
let h = 2166136261;
|
|
16603
|
+
for (let i2 = 0;i2 < text.length; i2++) {
|
|
16604
|
+
h ^= text.charCodeAt(i2);
|
|
16605
|
+
h = Math.imul(h, 16777619);
|
|
16606
|
+
}
|
|
16607
|
+
return h >>> 0;
|
|
16608
|
+
}
|
|
16609
|
+
function groupsBy(units, key) {
|
|
16610
|
+
const groups = new Map;
|
|
16611
|
+
units.forEach((unit, id) => {
|
|
16612
|
+
const ids = groups.get(key(unit));
|
|
16613
|
+
if (ids)
|
|
16614
|
+
ids.push(id);
|
|
16615
|
+
else
|
|
16616
|
+
groups.set(key(unit), [id]);
|
|
16617
|
+
});
|
|
16618
|
+
return [...groups.values()].filter((ids) => ids.length > 1);
|
|
16619
|
+
}
|
|
16620
|
+
function shinglesOf(labels, length) {
|
|
16621
|
+
const set = new Set;
|
|
16622
|
+
for (let i2 = 0;i2 + length <= labels.length; i2++) {
|
|
16623
|
+
set.add(fnv1a(labels.slice(i2, i2 + length).join("|")));
|
|
16624
|
+
}
|
|
16625
|
+
return set;
|
|
16626
|
+
}
|
|
16627
|
+
function nested(a, b) {
|
|
16628
|
+
if (a.file !== b.file)
|
|
16629
|
+
return false;
|
|
16630
|
+
const aInB = b.startIndex <= a.startIndex && a.endIndex <= b.endIndex;
|
|
16631
|
+
const bInA = a.startIndex <= b.startIndex && b.endIndex <= a.endIndex;
|
|
16632
|
+
return aInB || bInA;
|
|
16633
|
+
}
|
|
16634
|
+
function overlap(small, big) {
|
|
16635
|
+
let shared = 0;
|
|
16636
|
+
for (const item of small)
|
|
16637
|
+
if (big.has(item))
|
|
16638
|
+
shared++;
|
|
16639
|
+
return shared;
|
|
16640
|
+
}
|
|
16641
|
+
function round3(value) {
|
|
16642
|
+
return Number(value.toFixed(3));
|
|
16643
|
+
}
|
|
16644
|
+
function detect(units, params) {
|
|
16645
|
+
const exactGroups = groupsBy(units, (u) => u.exactHash);
|
|
16646
|
+
const textGroups = groupsBy(units, (u) => u.textHash);
|
|
16647
|
+
const shingles = units.map((u) => shinglesOf(u.labels, params.shingleLength));
|
|
16648
|
+
const index = new Map;
|
|
16649
|
+
shingles.forEach((set, id) => {
|
|
16650
|
+
for (const shingle of set) {
|
|
16651
|
+
const ids = index.get(shingle);
|
|
16652
|
+
if (ids)
|
|
16653
|
+
ids.push(id);
|
|
16654
|
+
else
|
|
16655
|
+
index.set(shingle, [id]);
|
|
16656
|
+
}
|
|
16657
|
+
});
|
|
16658
|
+
const near = [];
|
|
16659
|
+
units.forEach((unit, id) => {
|
|
16660
|
+
const mine = shingles[id];
|
|
16661
|
+
const shared = new Map;
|
|
16662
|
+
for (const shingle of mine) {
|
|
16663
|
+
const ids = index.get(shingle);
|
|
16664
|
+
if (ids.length > params.rareShingleLimit)
|
|
16665
|
+
continue;
|
|
16666
|
+
for (const other of ids)
|
|
16667
|
+
if (other > id)
|
|
16668
|
+
shared.set(other, (shared.get(other) ?? 0) + 1);
|
|
16669
|
+
}
|
|
16670
|
+
for (const [other, count] of shared) {
|
|
16671
|
+
if (count < params.minSharedShingles)
|
|
16672
|
+
continue;
|
|
16673
|
+
const peer = units[other];
|
|
16674
|
+
if (nested(unit, peer))
|
|
16675
|
+
continue;
|
|
16676
|
+
const theirs = shingles[other];
|
|
16677
|
+
const [small, big] = mine.size <= theirs.size ? [mine, theirs] : [theirs, mine];
|
|
16678
|
+
const inter = overlap(small, big);
|
|
16679
|
+
const jaccard = inter / (mine.size + theirs.size - inter);
|
|
16680
|
+
const contain = inter / small.size;
|
|
16681
|
+
const byContainment = contain >= params.containmentThreshold && small.size >= params.containmentMinShingles;
|
|
16682
|
+
if (jaccard < params.nearThreshold && !byContainment)
|
|
16683
|
+
continue;
|
|
16684
|
+
const common = overlap(unit.names, peer.names);
|
|
16685
|
+
const union = unit.names.size + peer.names.size - common;
|
|
16686
|
+
near.push({
|
|
16687
|
+
a: id,
|
|
16688
|
+
b: other,
|
|
16689
|
+
jStruct: round3(jaccard),
|
|
16690
|
+
contain: round3(contain),
|
|
16691
|
+
jNames: round3(common / (union || 1))
|
|
16692
|
+
});
|
|
16693
|
+
}
|
|
16694
|
+
});
|
|
16695
|
+
near.sort((x, y) => y.jStruct - x.jStruct || x.a - y.a || x.b - y.b);
|
|
16696
|
+
return { exactGroups, textGroups, near };
|
|
16697
|
+
}
|
|
16698
|
+
|
|
16699
|
+
// src/clones/report.ts
|
|
16700
|
+
function inventoryRecord(unit, id) {
|
|
16701
|
+
return {
|
|
16702
|
+
id,
|
|
16703
|
+
file: unit.file,
|
|
16704
|
+
name: unit.name,
|
|
16705
|
+
container: unit.container,
|
|
16706
|
+
kind: unit.kind,
|
|
16707
|
+
exported: unit.exported,
|
|
16708
|
+
startLine: unit.startLine,
|
|
16709
|
+
endLine: unit.endLine,
|
|
16710
|
+
loc: unit.loc,
|
|
16711
|
+
params: unit.params,
|
|
16712
|
+
async: unit.async,
|
|
16713
|
+
nodes: unit.nodes,
|
|
16714
|
+
doc: unit.doc,
|
|
16715
|
+
exactHash: unit.exactHash,
|
|
16716
|
+
textHash: unit.textHash
|
|
16717
|
+
};
|
|
16718
|
+
}
|
|
16719
|
+
function inventoryJsonl(units) {
|
|
16720
|
+
return units.map((unit, id) => `${JSON.stringify(inventoryRecord(unit, id))}
|
|
16721
|
+
`).join("");
|
|
16722
|
+
}
|
|
16723
|
+
function listJson(items) {
|
|
16724
|
+
if (items.length === 0)
|
|
16725
|
+
return "[]";
|
|
16726
|
+
return `[
|
|
16727
|
+
${items.map((item) => ` ${JSON.stringify(item)}`).join(`,
|
|
16728
|
+
`)}
|
|
16729
|
+
]`;
|
|
16730
|
+
}
|
|
16731
|
+
function clonesJson(config, detection) {
|
|
16732
|
+
const { out: _out, ...params } = config;
|
|
16733
|
+
return [
|
|
16734
|
+
"{",
|
|
16735
|
+
` "params": ${JSON.stringify(params)},`,
|
|
16736
|
+
` "exactGroups": ${listJson(detection.exactGroups)},`,
|
|
16737
|
+
` "textGroups": ${listJson(detection.textGroups)},`,
|
|
16738
|
+
` "near": ${listJson(detection.near)}`,
|
|
16739
|
+
"}",
|
|
16740
|
+
""
|
|
16741
|
+
].join(`
|
|
16742
|
+
`);
|
|
16743
|
+
}
|
|
16744
|
+
var SUMMARY_BANDS = [
|
|
16745
|
+
["0.9 or more", 0.9, Number.POSITIVE_INFINITY],
|
|
16746
|
+
["0.8 to 0.9", 0.8, 0.9],
|
|
16747
|
+
["0.7 to 0.8", 0.7, 0.8],
|
|
16748
|
+
["0.6 to 0.7", 0.6, 0.7],
|
|
16749
|
+
["below 0.6 (containment)", 0, 0.6]
|
|
16750
|
+
];
|
|
16751
|
+
function folderOf(file) {
|
|
16752
|
+
const parts2 = file.split("/").slice(0, -1);
|
|
16753
|
+
return parts2.length === 0 ? "." : parts2.slice(0, 2).join("/");
|
|
16754
|
+
}
|
|
16755
|
+
function percentile(sorted, p) {
|
|
16756
|
+
return sorted[Math.min(sorted.length - 1, Math.floor(sorted.length * p))] ?? 0;
|
|
16757
|
+
}
|
|
16758
|
+
function summaryText(config, stats, units, detection) {
|
|
16759
|
+
const sizes = units.map((u) => u.nodes).sort((a, b) => a - b);
|
|
16760
|
+
const inGroups = detection.exactGroups.reduce((sum, group) => sum + group.length, 0);
|
|
16761
|
+
const bands = SUMMARY_BANDS.map(([label, lo, hi]) => `${label}: ${detection.near.filter((p) => p.jStruct >= lo && p.jStruct < hi).length}`);
|
|
16762
|
+
const byFolder = new Map;
|
|
16763
|
+
for (const unit of units) {
|
|
16764
|
+
const key = folderOf(unit.file);
|
|
16765
|
+
byFolder.set(key, (byFolder.get(key) ?? 0) + 1);
|
|
16766
|
+
}
|
|
16767
|
+
const folders = [...byFolder.entries()].sort(([a], [b]) => a < b ? -1 : a > b ? 1 : 0).map(([key, count]) => `${key}=${count}`).join(" ");
|
|
16768
|
+
const lines = [
|
|
16769
|
+
`files=${stats.files} parseErrors=${stats.parseErrors} units(minNodes=${config.minNodes})=${units.length}`,
|
|
16770
|
+
`nodes p50=${percentile(sizes, 0.5)} p90=${percentile(sizes, 0.9)} p99=${percentile(sizes, 0.99)} max=${sizes[sizes.length - 1] ?? 0}`,
|
|
16771
|
+
`exact groups=${detection.exactGroups.length} (units in them=${inGroups}) text-identical groups=${detection.textGroups.length}`,
|
|
16772
|
+
`near pairs=${detection.near.length} by jStruct: ${bands.join(", ")}`,
|
|
16773
|
+
`units by folder: ${folders}`
|
|
16774
|
+
];
|
|
16775
|
+
return `${lines.join(`
|
|
16776
|
+
`)}
|
|
16777
|
+
`;
|
|
16778
|
+
}
|
|
16779
|
+
|
|
16780
|
+
// src/clones/settings.ts
|
|
16781
|
+
var CLONES_DEFAULTS = Object.freeze({
|
|
16782
|
+
src: "auto",
|
|
16783
|
+
out: null,
|
|
16784
|
+
minNodes: 50,
|
|
16785
|
+
shingleLength: 6,
|
|
16786
|
+
nearThreshold: 0.6,
|
|
16787
|
+
containmentThreshold: 0.9,
|
|
16788
|
+
containmentMinShingles: 20,
|
|
16789
|
+
rareShingleLimit: 60,
|
|
16790
|
+
minSharedShingles: 3
|
|
16791
|
+
});
|
|
16792
|
+
function mergeClonesConfig(cli, file) {
|
|
16793
|
+
const d = CLONES_DEFAULTS;
|
|
16794
|
+
const src = cli.src ?? file.src ?? d.src;
|
|
16795
|
+
return {
|
|
16796
|
+
src: src === "auto" ? src : [...src],
|
|
16797
|
+
out: cli.out !== undefined ? cli.out : file.out !== undefined ? file.out : d.out,
|
|
16798
|
+
minNodes: cli.minNodes ?? file.minNodes ?? d.minNodes,
|
|
16799
|
+
shingleLength: cli.shingleLength ?? file.shingleLength ?? d.shingleLength,
|
|
16800
|
+
nearThreshold: cli.nearThreshold ?? file.nearThreshold ?? d.nearThreshold,
|
|
16801
|
+
containmentThreshold: cli.containmentThreshold ?? file.containmentThreshold ?? d.containmentThreshold,
|
|
16802
|
+
containmentMinShingles: cli.containmentMinShingles ?? file.containmentMinShingles ?? d.containmentMinShingles,
|
|
16803
|
+
rareShingleLimit: cli.rareShingleLimit ?? file.rareShingleLimit ?? d.rareShingleLimit,
|
|
16804
|
+
minSharedShingles: cli.minSharedShingles ?? file.minSharedShingles ?? d.minSharedShingles
|
|
16805
|
+
};
|
|
16806
|
+
}
|
|
16807
|
+
|
|
16808
|
+
// src/clones/units.ts
|
|
16809
|
+
import { createHash as createHash2 } from "node:crypto";
|
|
16810
|
+
|
|
16811
|
+
// src/clones/normalise.ts
|
|
16812
|
+
var IDENTIFIERS = new Set([
|
|
16813
|
+
"identifier",
|
|
16814
|
+
"property_identifier",
|
|
16815
|
+
"shorthand_property_identifier",
|
|
16816
|
+
"shorthand_property_identifier_pattern",
|
|
16817
|
+
"private_property_identifier",
|
|
16818
|
+
"statement_identifier",
|
|
16819
|
+
"undefined"
|
|
16820
|
+
]);
|
|
16821
|
+
var LITERALS = new Set(["string", "number", "regex", "true", "false", "null"]);
|
|
16822
|
+
var TYPE_SYNTAX = new Set([
|
|
16823
|
+
"type_annotation",
|
|
16824
|
+
"type_arguments",
|
|
16825
|
+
"type_parameters",
|
|
16826
|
+
"type_alias_declaration",
|
|
16827
|
+
"interface_declaration",
|
|
16828
|
+
"enum_declaration",
|
|
16829
|
+
"ambient_declaration",
|
|
16830
|
+
"asserts_annotation",
|
|
16831
|
+
"opting_type_annotation",
|
|
16832
|
+
"omitting_type_annotation",
|
|
16833
|
+
"type_predicate_annotation",
|
|
16834
|
+
"accessibility_modifier",
|
|
16835
|
+
"override_modifier",
|
|
16836
|
+
"decorator",
|
|
16837
|
+
"comment"
|
|
16838
|
+
]);
|
|
16839
|
+
var TRANSPARENT = new Set([
|
|
16840
|
+
"parenthesized_expression",
|
|
16841
|
+
"as_expression",
|
|
16842
|
+
"satisfies_expression",
|
|
16843
|
+
"non_null_expression",
|
|
16844
|
+
"type_assertion",
|
|
16845
|
+
"instantiation_expression"
|
|
16846
|
+
]);
|
|
16847
|
+
var TEMPLATE_TEXT = new Set(["string_fragment", "escape_sequence"]);
|
|
16848
|
+
function wrapped(node) {
|
|
16849
|
+
const children = node.namedChildren;
|
|
16850
|
+
return (node.type === "type_assertion" ? children[children.length - 1] : children[0]) ?? null;
|
|
16851
|
+
}
|
|
16852
|
+
function walk2(node, out2) {
|
|
16853
|
+
if (!node.isNamed)
|
|
16854
|
+
return;
|
|
16855
|
+
const type = node.type;
|
|
16856
|
+
if (TYPE_SYNTAX.has(type))
|
|
16857
|
+
return;
|
|
16858
|
+
if (TRANSPARENT.has(type)) {
|
|
16859
|
+
const inner = wrapped(node);
|
|
16860
|
+
if (inner)
|
|
16861
|
+
walk2(inner, out2);
|
|
16862
|
+
return;
|
|
16863
|
+
}
|
|
16864
|
+
if (IDENTIFIERS.has(type)) {
|
|
16865
|
+
out2.labels.push("ID");
|
|
16866
|
+
out2.names.add(node.text.toLowerCase());
|
|
16867
|
+
return;
|
|
16868
|
+
}
|
|
16869
|
+
if (LITERALS.has(type)) {
|
|
16870
|
+
out2.labels.push("LIT");
|
|
16871
|
+
return;
|
|
16872
|
+
}
|
|
16873
|
+
if (TEMPLATE_TEXT.has(type)) {
|
|
16874
|
+
out2.labels.push("TPL");
|
|
16875
|
+
return;
|
|
16876
|
+
}
|
|
16877
|
+
if (type !== "template_substitution") {
|
|
16878
|
+
const operator = node.childForFieldName("operator");
|
|
16879
|
+
out2.labels.push(operator ? `${type}:${operator.type}` : type);
|
|
16880
|
+
}
|
|
16881
|
+
for (const child of node.children)
|
|
16882
|
+
walk2(child, out2);
|
|
16883
|
+
}
|
|
16884
|
+
function normalise(body2) {
|
|
16885
|
+
const out2 = { labels: [], names: new Set };
|
|
16886
|
+
walk2(body2, out2);
|
|
16887
|
+
return out2;
|
|
16888
|
+
}
|
|
16889
|
+
|
|
16890
|
+
// src/clones/units.ts
|
|
16891
|
+
var DOC_LIMIT = 160;
|
|
16892
|
+
var FUNCTION_VALUES = new Set([
|
|
16893
|
+
"arrow_function",
|
|
16894
|
+
"function_expression",
|
|
16895
|
+
"function",
|
|
16896
|
+
"generator_function"
|
|
16897
|
+
]);
|
|
16898
|
+
var CLASSES = new Set(["class_declaration", "abstract_class_declaration", "class"]);
|
|
16899
|
+
var ANONYMOUS_CLASS = "(class)";
|
|
16900
|
+
function sha1(text) {
|
|
16901
|
+
return createHash2("sha1").update(text).digest("hex").slice(0, 16);
|
|
16902
|
+
}
|
|
16903
|
+
function keyName(key) {
|
|
16904
|
+
if (!key)
|
|
16905
|
+
return null;
|
|
16906
|
+
return key.type === "string" ? key.text.slice(1, -1) : key.text;
|
|
16907
|
+
}
|
|
16908
|
+
function hasToken(node, token) {
|
|
16909
|
+
return node.children.some((c) => !c.isNamed && c.type === token);
|
|
16910
|
+
}
|
|
16911
|
+
function anchorOf(node) {
|
|
16912
|
+
let anchor = node;
|
|
16913
|
+
if (node.type === "variable_declarator" && node.parent)
|
|
16914
|
+
anchor = node.parent;
|
|
16915
|
+
if (anchor.parent?.type === "export_statement")
|
|
16916
|
+
anchor = anchor.parent;
|
|
16917
|
+
return anchor;
|
|
16918
|
+
}
|
|
16919
|
+
function declared(node, container) {
|
|
16920
|
+
switch (node.type) {
|
|
16921
|
+
case "function_declaration":
|
|
16922
|
+
case "generator_function_declaration": {
|
|
16923
|
+
const name2 = keyName(node.childForFieldName("name"));
|
|
16924
|
+
if (!name2)
|
|
16925
|
+
return null;
|
|
16926
|
+
return { fn: node, name: name2, kind: container ? "local" : "function", anchor: anchorOf(node) };
|
|
16927
|
+
}
|
|
16928
|
+
case "method_definition": {
|
|
16929
|
+
const name2 = keyName(node.childForFieldName("name"));
|
|
16930
|
+
if (!name2)
|
|
16931
|
+
return null;
|
|
16932
|
+
const inClass = node.parent?.type === "class_body";
|
|
16933
|
+
let kind = "prop";
|
|
16934
|
+
if (inClass) {
|
|
16935
|
+
if (name2 === "constructor")
|
|
16936
|
+
kind = "ctor";
|
|
16937
|
+
else if (hasToken(node, "get"))
|
|
16938
|
+
kind = "get";
|
|
16939
|
+
else if (hasToken(node, "set"))
|
|
16940
|
+
kind = "set";
|
|
16941
|
+
else
|
|
16942
|
+
kind = "method";
|
|
16943
|
+
}
|
|
16944
|
+
return { fn: node, name: name2, kind, anchor: node };
|
|
16945
|
+
}
|
|
16946
|
+
case "public_field_definition": {
|
|
16947
|
+
const value = node.childForFieldName("value");
|
|
16948
|
+
const name2 = keyName(node.childForFieldName("name"));
|
|
16949
|
+
if (!value || !name2 || !FUNCTION_VALUES.has(value.type))
|
|
16950
|
+
return null;
|
|
16951
|
+
return { fn: value, name: name2, kind: "field", anchor: node };
|
|
16952
|
+
}
|
|
16953
|
+
case "variable_declarator": {
|
|
16954
|
+
const value = node.childForFieldName("value");
|
|
16955
|
+
const id = node.childForFieldName("name");
|
|
16956
|
+
if (!value || !id || id.type !== "identifier" || !FUNCTION_VALUES.has(value.type)) {
|
|
16957
|
+
return null;
|
|
16958
|
+
}
|
|
16959
|
+
return {
|
|
16960
|
+
fn: value,
|
|
16961
|
+
name: id.text,
|
|
16962
|
+
kind: container ? "local" : "function",
|
|
16963
|
+
anchor: anchorOf(node)
|
|
16964
|
+
};
|
|
16965
|
+
}
|
|
16966
|
+
case "pair": {
|
|
16967
|
+
const value = node.childForFieldName("value");
|
|
16968
|
+
const name2 = keyName(node.childForFieldName("key"));
|
|
16969
|
+
if (!value || !name2 || !FUNCTION_VALUES.has(value.type))
|
|
16970
|
+
return null;
|
|
16971
|
+
return { fn: value, name: name2, kind: "prop", anchor: node };
|
|
16972
|
+
}
|
|
16973
|
+
default:
|
|
16974
|
+
return null;
|
|
16975
|
+
}
|
|
16976
|
+
}
|
|
16977
|
+
function paramCount(fn) {
|
|
16978
|
+
const list = fn.childForFieldName("parameters");
|
|
16979
|
+
if (list)
|
|
16980
|
+
return list.namedChildren.filter((c) => c.type !== "comment").length;
|
|
16981
|
+
return fn.childForFieldName("parameter") ? 1 : 0;
|
|
16982
|
+
}
|
|
16983
|
+
function docOf(anchor, source) {
|
|
16984
|
+
const comment = anchor.previousSibling;
|
|
16985
|
+
if (comment?.type !== "comment" || !comment.text.startsWith("/**"))
|
|
16986
|
+
return "";
|
|
16987
|
+
if (source.slice(comment.endIndex, anchor.startIndex).trim() !== "")
|
|
16988
|
+
return "";
|
|
16989
|
+
const lines = comment.text.slice(3, -2).split(`
|
|
16990
|
+
`).map((line) => line.replace(/^\s*\*\s?/, "").trim());
|
|
16991
|
+
const tag = lines.findIndex((line) => line.startsWith("@"));
|
|
16992
|
+
const text = (tag === -1 ? lines : lines.slice(0, tag)).filter((line) => line !== "");
|
|
16993
|
+
return text.join(" ").slice(0, DOC_LIMIT);
|
|
16994
|
+
}
|
|
16995
|
+
function buildUnit(d, container, source, file, minNodes) {
|
|
16996
|
+
const body2 = d.fn.childForFieldName("body");
|
|
16997
|
+
if (!body2)
|
|
16998
|
+
return null;
|
|
16999
|
+
const { labels, names } = normalise(body2);
|
|
17000
|
+
if (labels.length < minNodes)
|
|
17001
|
+
return null;
|
|
17002
|
+
const startLine = d.anchor.startPosition.row + 1;
|
|
17003
|
+
const endLine = d.fn.endPosition.row + 1;
|
|
17004
|
+
return {
|
|
17005
|
+
file,
|
|
17006
|
+
name: d.name,
|
|
17007
|
+
container,
|
|
17008
|
+
kind: d.kind,
|
|
17009
|
+
exported: d.anchor.type === "export_statement",
|
|
17010
|
+
startLine,
|
|
17011
|
+
endLine,
|
|
17012
|
+
loc: endLine - startLine + 1,
|
|
17013
|
+
params: paramCount(d.fn),
|
|
17014
|
+
async: hasToken(d.fn, "async"),
|
|
17015
|
+
nodes: labels.length,
|
|
17016
|
+
doc: docOf(d.anchor, source),
|
|
17017
|
+
exactHash: sha1(labels.join(" ")),
|
|
17018
|
+
textHash: sha1(source.slice(body2.startIndex, body2.endIndex).replace(/\s+/g, "")),
|
|
17019
|
+
startIndex: d.fn.startIndex,
|
|
17020
|
+
endIndex: d.fn.endIndex,
|
|
17021
|
+
labels,
|
|
17022
|
+
names
|
|
17023
|
+
};
|
|
17024
|
+
}
|
|
17025
|
+
function unitsOf(root, source, file, minNodes) {
|
|
17026
|
+
const units = [];
|
|
17027
|
+
const visit = (node, container) => {
|
|
17028
|
+
const found = declared(node, container);
|
|
17029
|
+
if (found) {
|
|
17030
|
+
const unit = buildUnit(found, container, source, file, minNodes);
|
|
17031
|
+
if (unit)
|
|
17032
|
+
units.push(unit);
|
|
17033
|
+
}
|
|
17034
|
+
let inner = container;
|
|
17035
|
+
if (CLASSES.has(node.type)) {
|
|
17036
|
+
inner = keyName(node.childForFieldName("name")) ?? (container || ANONYMOUS_CLASS);
|
|
17037
|
+
}
|
|
17038
|
+
for (const child of node.namedChildren)
|
|
17039
|
+
visit(child, inner);
|
|
17040
|
+
};
|
|
17041
|
+
visit(root, "");
|
|
17042
|
+
return units;
|
|
17043
|
+
}
|
|
17044
|
+
|
|
17045
|
+
// src/clones/index.ts
|
|
17046
|
+
var CLONES_HELP = `Usage: repo-tools clones [root] [--out=<dir>] [--config=<path>] [--src=<path>...]
|
|
17047
|
+
[--min-nodes=<n>] [--shingle=<n>] [--near=<f>]
|
|
17048
|
+
[--containment=<f>] [--containment-min=<n>]
|
|
17049
|
+
[--rare=<n>] [--min-shared=<n>]
|
|
17050
|
+
|
|
17051
|
+
List the functions of a TypeScript or JavaScript repository and find the functions
|
|
17052
|
+
that copy each other. The command only reads the repository. The result is a list
|
|
17053
|
+
of candidates: read both bodies of a pair before you change anything.
|
|
17054
|
+
|
|
17055
|
+
A function is a unit when it has a name and its body has at least --min-nodes
|
|
17056
|
+
labels. A label is one node of the body, with names, literals, types and comments
|
|
17057
|
+
erased. Units with the same label sequence are an exact group. Two units whose
|
|
17058
|
+
shingles (runs of --shingle labels) overlap enough are a near pair.
|
|
17059
|
+
|
|
17060
|
+
--out=<dir> Write inventory.jsonl and clones.json to this folder.
|
|
17061
|
+
Without it the command writes no file.
|
|
17062
|
+
--config=<path> The config file. The default is repo-tools.config.json at
|
|
17063
|
+
the root. The settings are in its "clones" section.
|
|
17064
|
+
--src=<path>... The folders to read. The word "auto" (the default) reads
|
|
17065
|
+
the files of the src area. The flag takes every argument
|
|
17066
|
+
up to the next flag.
|
|
17067
|
+
--min-nodes=<n> The smallest number of labels of a unit. Default 50.
|
|
17068
|
+
--shingle=<n> The labels in one shingle. Default 6.
|
|
17069
|
+
--near=<f> The smallest Jaccard score of a near pair. Default 0.6.
|
|
17070
|
+
--containment=<f> The smallest containment of a near pair. Default 0.9.
|
|
17071
|
+
--containment-min=<n> The smallest shingle count of the smaller unit for the
|
|
17072
|
+
containment rule. Default 20.
|
|
17073
|
+
--rare=<n> A shingle in more units than this does not pick candidate
|
|
17074
|
+
pairs. Default 60.
|
|
17075
|
+
--min-shared=<n> The rare shingles that a candidate pair must share.
|
|
17076
|
+
Default 3.
|
|
17077
|
+
--root=<path> The root. The default is the current folder.
|
|
17078
|
+
--help, -h Show this help.
|
|
17079
|
+
|
|
17080
|
+
A flag overrides the config file, and the config file overrides the default.
|
|
17081
|
+
|
|
17082
|
+
Exit codes: 0 when the run finished. 1 when the root holds no source file to read.
|
|
17083
|
+
2 on a usage error, a bad config file or a root that is not a folder.
|
|
17084
|
+
`;
|
|
17085
|
+
|
|
17086
|
+
class NothingToRead extends Error {
|
|
17087
|
+
}
|
|
17088
|
+
var VALUE_FLAGS2 = {
|
|
17089
|
+
"--out": "out",
|
|
17090
|
+
"--min-nodes": "minNodes",
|
|
17091
|
+
"--shingle": "shingleLength",
|
|
17092
|
+
"--near": "nearThreshold",
|
|
17093
|
+
"--containment": "containmentThreshold",
|
|
17094
|
+
"--containment-min": "containmentMinShingles",
|
|
17095
|
+
"--rare": "rareShingleLimit",
|
|
17096
|
+
"--min-shared": "minSharedShingles"
|
|
17097
|
+
};
|
|
17098
|
+
var NUMERIC = new Set([
|
|
17099
|
+
"minNodes",
|
|
17100
|
+
"shingleLength",
|
|
17101
|
+
"nearThreshold",
|
|
17102
|
+
"containmentThreshold",
|
|
17103
|
+
"containmentMinShingles",
|
|
17104
|
+
"rareShingleLimit",
|
|
17105
|
+
"minSharedShingles"
|
|
17106
|
+
]);
|
|
17107
|
+
function parseArgs2(argv) {
|
|
17108
|
+
const options = { root: ".", config: undefined, settings: {} };
|
|
17109
|
+
const seen = new Set;
|
|
17110
|
+
let rootSet = false;
|
|
17111
|
+
const setRoot = (value) => {
|
|
17112
|
+
if (rootSet)
|
|
17113
|
+
throw new Error("the root is set twice (use --root=<path> or one positional path)");
|
|
17114
|
+
rootSet = true;
|
|
17115
|
+
options.root = value;
|
|
17116
|
+
};
|
|
17117
|
+
const once = (flag) => {
|
|
17118
|
+
if (seen.has(flag))
|
|
17119
|
+
throw new Error(`flag ${flag} is set twice`);
|
|
17120
|
+
seen.add(flag);
|
|
17121
|
+
};
|
|
17122
|
+
for (let i2 = 0;i2 < argv.length; i2++) {
|
|
17123
|
+
const arg = argv[i2] ?? "";
|
|
17124
|
+
if (!arg.startsWith("-")) {
|
|
17125
|
+
setRoot(arg);
|
|
17126
|
+
continue;
|
|
17127
|
+
}
|
|
17128
|
+
const eq = arg.indexOf("=");
|
|
17129
|
+
const name2 = eq === -1 ? arg : arg.slice(0, eq);
|
|
17130
|
+
if (name2 === "--src") {
|
|
17131
|
+
once(name2);
|
|
17132
|
+
const paths = [];
|
|
17133
|
+
if (eq !== -1)
|
|
17134
|
+
paths.push(arg.slice(eq + 1));
|
|
17135
|
+
else
|
|
17136
|
+
while (i2 + 1 < argv.length && !(argv[i2 + 1] ?? "").startsWith("-")) {
|
|
17137
|
+
paths.push(argv[++i2] ?? "");
|
|
17138
|
+
}
|
|
17139
|
+
if (paths.length === 0 || paths.some((p) => p === "")) {
|
|
17140
|
+
throw new Error("flag --src needs a value: --src=<path>");
|
|
17141
|
+
}
|
|
17142
|
+
const only = paths.length === 1 ? paths[0] : undefined;
|
|
17143
|
+
options.settings.src = checkClonesFlag(name2, "src", only === "auto" ? "auto" : paths);
|
|
17144
|
+
continue;
|
|
17145
|
+
}
|
|
17146
|
+
if (name2 !== "--root" && name2 !== "--config" && !Object.hasOwn(VALUE_FLAGS2, name2)) {
|
|
17147
|
+
throw new Error(`unknown flag '${name2}' (see repo-tools clones --help)`);
|
|
17148
|
+
}
|
|
17149
|
+
once(name2);
|
|
17150
|
+
let value = eq === -1 ? undefined : arg.slice(eq + 1);
|
|
17151
|
+
if (eq === -1 && i2 + 1 < argv.length && !(argv[i2 + 1] ?? "").startsWith("-")) {
|
|
17152
|
+
value = argv[++i2];
|
|
17153
|
+
}
|
|
17154
|
+
if (value === undefined || value === "") {
|
|
17155
|
+
throw new Error(`flag ${name2} needs a value: ${name2}=<value>`);
|
|
17156
|
+
}
|
|
17157
|
+
if (name2 === "--root")
|
|
17158
|
+
setRoot(value);
|
|
17159
|
+
else if (name2 === "--config")
|
|
17160
|
+
options.config = value;
|
|
17161
|
+
else {
|
|
17162
|
+
const key = VALUE_FLAGS2[name2];
|
|
17163
|
+
const raw = NUMERIC.has(key) ? /^\d+(\.\d+)?$/.test(value) ? Number(value) : NaN : value;
|
|
17164
|
+
Object.assign(options.settings, { [key]: checkClonesFlag(name2, key, raw) });
|
|
17165
|
+
}
|
|
17166
|
+
}
|
|
17167
|
+
return options;
|
|
17168
|
+
}
|
|
17169
|
+
function folderPrefix(path) {
|
|
17170
|
+
const posix = path.replace(/\\/g, "/").replace(/^(\.\/)+/, "").replace(/\/+$/, "");
|
|
17171
|
+
return posix === "." ? "" : posix;
|
|
17172
|
+
}
|
|
17173
|
+
function selectFiles(root, src) {
|
|
17174
|
+
const found = discover(root, "typescript").filter((f) => !f.path.endsWith(".d.ts"));
|
|
17175
|
+
if (src === "auto")
|
|
17176
|
+
return found.filter((f) => f.area === "src").map((f) => f.path);
|
|
17177
|
+
const folders = src.map(folderPrefix);
|
|
17178
|
+
return found.filter((f) => folders.some((d) => d === "" || f.path === d || f.path.startsWith(`${d}/`))).map((f) => f.path);
|
|
17179
|
+
}
|
|
17180
|
+
async function readUnits(root, files, minNodes) {
|
|
17181
|
+
const units = [];
|
|
17182
|
+
let parseErrors = 0;
|
|
17183
|
+
for (const file of files) {
|
|
17184
|
+
const source = readSource(join22(root, file)).replace(/^\uFEFF/, "");
|
|
17185
|
+
const grammar = grammarFor(file);
|
|
17186
|
+
await loadGrammar(grammar);
|
|
17187
|
+
const tree = parserFor(grammar).parse(source);
|
|
17188
|
+
if (!tree) {
|
|
17189
|
+
parseErrors += 1;
|
|
17190
|
+
continue;
|
|
17191
|
+
}
|
|
17192
|
+
try {
|
|
17193
|
+
if (tree.rootNode.hasError)
|
|
17194
|
+
parseErrors += 1;
|
|
17195
|
+
units.push(...unitsOf(tree.rootNode, source, file, minNodes));
|
|
17196
|
+
} finally {
|
|
17197
|
+
tree.delete();
|
|
17198
|
+
}
|
|
17199
|
+
}
|
|
17200
|
+
return { units, parseErrors };
|
|
17201
|
+
}
|
|
17202
|
+
async function execute(root, config, io) {
|
|
17203
|
+
const files = selectFiles(root, config.src);
|
|
17204
|
+
if (files.length === 0) {
|
|
17205
|
+
throw new NothingToRead(config.src === "auto" ? "no TypeScript or JavaScript file in the src area (set clones.src or pass --src=<path>)" : "no TypeScript or JavaScript file below the listed folders");
|
|
17206
|
+
}
|
|
17207
|
+
const { units, parseErrors } = await readUnits(root, files, config.minNodes);
|
|
17208
|
+
const detection = detect(units, config);
|
|
17209
|
+
io.stdout(summaryText(config, { files: files.length, parseErrors }, units, detection));
|
|
17210
|
+
if (config.out !== null) {
|
|
17211
|
+
const out2 = resolve8(root, config.out);
|
|
17212
|
+
try {
|
|
17213
|
+
writeLf(join22(out2, "inventory.jsonl"), inventoryJsonl(units));
|
|
17214
|
+
writeLf(join22(out2, "clones.json"), clonesJson(config, detection));
|
|
17215
|
+
} catch {
|
|
17216
|
+
throw new Error(`cannot write the output folder '${config.out}'`);
|
|
17217
|
+
}
|
|
17218
|
+
const shown = config.out.replace(/\\/g, "/").replace(/\/+$/, "");
|
|
17219
|
+
io.stdout(`wrote ${shown}/inventory.jsonl
|
|
17220
|
+
wrote ${shown}/clones.json
|
|
17221
|
+
`);
|
|
17222
|
+
}
|
|
17223
|
+
return 0;
|
|
17224
|
+
}
|
|
17225
|
+
async function run4(argv, io) {
|
|
17226
|
+
let root;
|
|
17227
|
+
const stderr = (text) => io.stderr(root ? maskRoot(text, root) : text);
|
|
17228
|
+
try {
|
|
17229
|
+
const options = parseArgs2(argv);
|
|
17230
|
+
root = resolve8(options.root);
|
|
17231
|
+
if (!isDirectory(root))
|
|
17232
|
+
throw new Error("the root <root> is not an existing directory");
|
|
17233
|
+
const file = loadConfigSections(root, options.config).clones;
|
|
17234
|
+
return await execute(root, mergeClonesConfig(options.settings, file), io);
|
|
17235
|
+
} catch (error) {
|
|
17236
|
+
const message = error instanceof Error ? error.message : String(error);
|
|
17237
|
+
stderr(`repo-tools clones: ${message}
|
|
17238
|
+
`);
|
|
17239
|
+
return error instanceof NothingToRead ? 1 : 2;
|
|
17240
|
+
}
|
|
17241
|
+
}
|
|
17242
|
+
|
|
16570
17243
|
// src/compress/index.ts
|
|
16571
17244
|
import { existsSync as existsSync9, readdirSync as readdirSync3, readFileSync as readFileSync20, statSync as statSync8, writeFileSync as writeFileSync4 } from "node:fs";
|
|
16572
|
-
import { basename as basename8, dirname as dirname8, extname as extname4, join as
|
|
17245
|
+
import { basename as basename8, dirname as dirname8, extname as extname4, join as join23 } from "node:path";
|
|
16573
17246
|
|
|
16574
17247
|
// src/compress/formats.ts
|
|
16575
17248
|
import { extname as extname3 } from "node:path";
|
|
@@ -17202,7 +17875,7 @@ a value, a missing input file and two inputs without --batch are errors. No file
|
|
|
17202
17875
|
var defaultDeps = {
|
|
17203
17876
|
readdir: (dir) => readdirSync3(dir, { withFileTypes: true })
|
|
17204
17877
|
};
|
|
17205
|
-
function
|
|
17878
|
+
function parseArgs3(args2) {
|
|
17206
17879
|
const o = {
|
|
17207
17880
|
input: "",
|
|
17208
17881
|
inputs: [],
|
|
@@ -17325,7 +17998,7 @@ function findFiles(dir, pattern, recursive, deps = defaultDeps) {
|
|
|
17325
17998
|
const scan = (current) => {
|
|
17326
17999
|
const entries = [...deps.readdir(current)].sort((a, b) => compareCodeUnits(a.name, b.name));
|
|
17327
18000
|
for (const entry of entries) {
|
|
17328
|
-
const full =
|
|
18001
|
+
const full = join23(current, entry.name);
|
|
17329
18002
|
if (entry.isDirectory()) {
|
|
17330
18003
|
if (recursive && !entry.name.startsWith(".") && entry.name !== "node_modules")
|
|
17331
18004
|
scan(full);
|
|
@@ -17342,16 +18015,16 @@ function isDirectory2(path) {
|
|
|
17342
18015
|
}
|
|
17343
18016
|
function compactName(file) {
|
|
17344
18017
|
const ext = extname4(file);
|
|
17345
|
-
return
|
|
18018
|
+
return join23(dirname8(file), `${basename8(file, ext)}.compact${ext}`);
|
|
17346
18019
|
}
|
|
17347
18020
|
function restoredName(file) {
|
|
17348
18021
|
const base = basename8(file);
|
|
17349
18022
|
const stripped = base.replace(/\.compact(?=\.[^.]*$|$)/i, "");
|
|
17350
|
-
const output = stripped === base ? file :
|
|
18023
|
+
const output = stripped === base ? file : join23(dirname8(file), stripped);
|
|
17351
18024
|
if (output !== file)
|
|
17352
18025
|
return output;
|
|
17353
18026
|
const ext = extname4(file);
|
|
17354
|
-
return
|
|
18027
|
+
return join23(dirname8(file), `${basename8(file, ext)}.restored${ext}`);
|
|
17355
18028
|
}
|
|
17356
18029
|
function assertCanRestore(output, o) {
|
|
17357
18030
|
if (!o.dryRun && !o.yes && existsSync9(output)) {
|
|
@@ -17511,10 +18184,10 @@ Level: ${o.level}
|
|
|
17511
18184
|
}
|
|
17512
18185
|
return 0;
|
|
17513
18186
|
}
|
|
17514
|
-
async function
|
|
18187
|
+
async function run5(argv, io, deps = defaultDeps) {
|
|
17515
18188
|
let o;
|
|
17516
18189
|
try {
|
|
17517
|
-
o =
|
|
18190
|
+
o = parseArgs3(argv);
|
|
17518
18191
|
} catch (error) {
|
|
17519
18192
|
if (!(error instanceof UsageError))
|
|
17520
18193
|
throw error;
|
|
@@ -17565,16 +18238,16 @@ async function run4(argv, io, deps = defaultDeps) {
|
|
|
17565
18238
|
|
|
17566
18239
|
// src/docs/index.ts
|
|
17567
18240
|
import { lstatSync as lstatSync5, readFileSync as readFileSync23, realpathSync as realpathSync3, writeFileSync as writeFileSync5 } from "node:fs";
|
|
17568
|
-
import { basename as basename9, isAbsolute as isAbsolute4, join as
|
|
18241
|
+
import { basename as basename9, isAbsolute as isAbsolute4, join as join27, relative as relative5, resolve as resolve9, sep as sep4 } from "node:path";
|
|
17569
18242
|
|
|
17570
18243
|
// src/docs/analyse.ts
|
|
17571
18244
|
import { readFileSync as readFileSync22 } from "node:fs";
|
|
17572
|
-
import { join as
|
|
18245
|
+
import { join as join25 } from "node:path";
|
|
17573
18246
|
|
|
17574
18247
|
// src/docs/discovery.ts
|
|
17575
18248
|
import { spawnSync as spawnSync2 } from "node:child_process";
|
|
17576
18249
|
import { existsSync as existsSync10, readdirSync as readdirSync4, readFileSync as readFileSync21, statSync as statSync9 } from "node:fs";
|
|
17577
|
-
import { join as
|
|
18250
|
+
import { join as join24 } from "node:path";
|
|
17578
18251
|
var PYTHON_SUFFIXES2 = new Set([".py"]);
|
|
17579
18252
|
var TYPESCRIPT_SUFFIXES = new Set([".ts", ".tsx", ".js", ".jsx", ".mjs", ".cjs"]);
|
|
17580
18253
|
var SKIP_DIRS2 = new Set([
|
|
@@ -17658,8 +18331,8 @@ function walkSource(root) {
|
|
|
17658
18331
|
const rel = `${prefix}${entry.name}`;
|
|
17659
18332
|
if (entry.isDirectory()) {
|
|
17660
18333
|
if (!SKIP_DIRS2.has(entry.name))
|
|
17661
|
-
go(
|
|
17662
|
-
} else if (isSource(rel) && statOk(
|
|
18334
|
+
go(join24(dir, entry.name), `${rel}/`);
|
|
18335
|
+
} else if (isSource(rel) && statOk(join24(dir, entry.name))) {
|
|
17663
18336
|
found.push(rel);
|
|
17664
18337
|
}
|
|
17665
18338
|
}
|
|
@@ -17675,7 +18348,7 @@ function statOk(path) {
|
|
|
17675
18348
|
}
|
|
17676
18349
|
}
|
|
17677
18350
|
function loadExclusions(root) {
|
|
17678
|
-
const file =
|
|
18351
|
+
const file = join24(root, CONFIG_NAME);
|
|
17679
18352
|
if (!existsSync10(file) || !statOk(file))
|
|
17680
18353
|
return [];
|
|
17681
18354
|
let data;
|
|
@@ -18287,9 +18960,9 @@ var EXPORT_TYPE_STAR = new RegExp(`(\\bexport${SPACE_CLASS}+)type(${SPACE_CLASS}
|
|
|
18287
18960
|
function sanitise(source) {
|
|
18288
18961
|
return source.replace(ABSTRACT_PROPERTY, "abstrac_$1").replace(EXPORT_TYPE_STAR, "$1 $2");
|
|
18289
18962
|
}
|
|
18290
|
-
var
|
|
18963
|
+
var IDENTIFIERS2 = new Set(["identifier", "shorthand_property_identifier_pattern"]);
|
|
18291
18964
|
function boundIdentifiers(node, text) {
|
|
18292
|
-
if (
|
|
18965
|
+
if (IDENTIFIERS2.has(node.type))
|
|
18293
18966
|
return [text(node)];
|
|
18294
18967
|
const pattern = node.childForFieldName("pattern");
|
|
18295
18968
|
if (pattern)
|
|
@@ -18307,7 +18980,7 @@ function boundIdentifiers(node, text) {
|
|
|
18307
18980
|
case "rest_pattern":
|
|
18308
18981
|
return into(node.namedChildren[0] ?? null);
|
|
18309
18982
|
default: {
|
|
18310
|
-
const first = node.namedChildren.find((c) =>
|
|
18983
|
+
const first = node.namedChildren.find((c) => IDENTIFIERS2.has(c.type));
|
|
18311
18984
|
return first ? [text(first)] : [];
|
|
18312
18985
|
}
|
|
18313
18986
|
}
|
|
@@ -18316,7 +18989,7 @@ function paramsOf2(node, text) {
|
|
|
18316
18989
|
const list = node.childForFieldName("parameters");
|
|
18317
18990
|
return (list?.namedChildren ?? []).flatMap((param) => boundIdentifiers(param, text));
|
|
18318
18991
|
}
|
|
18319
|
-
function
|
|
18992
|
+
function anchorOf2(node) {
|
|
18320
18993
|
let target = node;
|
|
18321
18994
|
while (target.parent?.type === "export_statement")
|
|
18322
18995
|
target = target.parent;
|
|
@@ -18473,7 +19146,7 @@ async function analyseTypeScript(path, source) {
|
|
|
18473
19146
|
const text = (node) => source.slice(node.startIndex, node.endIndex);
|
|
18474
19147
|
for (const node of tree.rootNode.descendantsOfType(DECLARATION_TYPES)) {
|
|
18475
19148
|
const kind = DECLARATIONS[node.type];
|
|
18476
|
-
const anchor =
|
|
19149
|
+
const anchor = anchorOf2(node);
|
|
18477
19150
|
const lead = stubAnchor(anchor, source, text);
|
|
18478
19151
|
const doc = leadingDoc(anchor, node, text);
|
|
18479
19152
|
const dialect = dialectOf(doc);
|
|
@@ -18530,7 +19203,7 @@ async function analyseFile(root, rel) {
|
|
|
18530
19203
|
const language = languageOf(rel);
|
|
18531
19204
|
let source;
|
|
18532
19205
|
try {
|
|
18533
|
-
source = readSourceText2(
|
|
19206
|
+
source = readSourceText2(join25(root, rel));
|
|
18534
19207
|
} catch (error) {
|
|
18535
19208
|
const report = newReport(rel, language);
|
|
18536
19209
|
report.error = `unreadable: ${readFailure(error)}`;
|
|
@@ -18547,7 +19220,7 @@ async function analyseRepo(root) {
|
|
|
18547
19220
|
}
|
|
18548
19221
|
|
|
18549
19222
|
// src/docs/report.ts
|
|
18550
|
-
import { join as
|
|
19223
|
+
import { join as join26 } from "node:path";
|
|
18551
19224
|
function percent3(part, whole) {
|
|
18552
19225
|
if (whole === 0)
|
|
18553
19226
|
return 100;
|
|
@@ -18659,8 +19332,8 @@ function symbolJson(s) {
|
|
|
18659
19332
|
};
|
|
18660
19333
|
}
|
|
18661
19334
|
function writeOutputs(outDir, reports, stats, provenance, rootName) {
|
|
18662
|
-
const markdown =
|
|
18663
|
-
const json =
|
|
19335
|
+
const markdown = join26(outDir, "COVERAGE.md");
|
|
19336
|
+
const json = join26(outDir, "coverage.json");
|
|
18664
19337
|
writeReport(markdown, renderMarkdown(reports, stats, provenance, rootName));
|
|
18665
19338
|
const payload = {
|
|
18666
19339
|
summary: statsJson(stats),
|
|
@@ -18864,7 +19537,7 @@ Exit codes: 0 when the action succeeded and the gate passed. 1 when the gate
|
|
|
18864
19537
|
failed, or when the root holds no source file to measure. 2 on a usage error,
|
|
18865
19538
|
when the root is not a folder, or when ${CONFIG_NAME} is malformed.
|
|
18866
19539
|
`;
|
|
18867
|
-
var
|
|
19540
|
+
var VALUE_FLAGS3 = {
|
|
18868
19541
|
"--root": "all",
|
|
18869
19542
|
"--out": "scan",
|
|
18870
19543
|
"--path": "stub",
|
|
@@ -18917,8 +19590,8 @@ function parseDocsArgs(argv) {
|
|
|
18917
19590
|
while (i2 + 1 < argv.length && !(argv[i2 + 1] ?? "").startsWith("-"))
|
|
18918
19591
|
options.paths.push(argv[++i2] ?? "");
|
|
18919
19592
|
}
|
|
18920
|
-
} else if (Object.hasOwn(
|
|
18921
|
-
forbid(name2,
|
|
19593
|
+
} else if (Object.hasOwn(VALUE_FLAGS3, name2)) {
|
|
19594
|
+
forbid(name2, VALUE_FLAGS3[name2]);
|
|
18922
19595
|
let value = eq === -1 ? undefined : arg.slice(eq + 1);
|
|
18923
19596
|
if (eq === -1 && i2 + 1 < argv.length && !(argv[i2 + 1] ?? "").startsWith("-"))
|
|
18924
19597
|
value = argv[++i2];
|
|
@@ -19012,7 +19685,7 @@ async function runScan(root, options, io) {
|
|
|
19012
19685
|
if (reports.length === 0)
|
|
19013
19686
|
throw new NothingToMeasure(provenance);
|
|
19014
19687
|
const stats = summarise(reports);
|
|
19015
|
-
const outDir =
|
|
19688
|
+
const outDir = resolve9(options.out ?? join27(root, "docs", "code-docs"));
|
|
19016
19689
|
const written = writeOutputs(outDir, reports, stats, provenance, basename9(root));
|
|
19017
19690
|
io.stdout(`scanned ${stats.filesScanned} file(s) via ${provenance}
|
|
19018
19691
|
`);
|
|
@@ -19052,7 +19725,7 @@ async function runStub(root, options, io) {
|
|
|
19052
19725
|
const targets = report.symbols.filter((s) => s.issues.some((i2) => i2.rule === "M1"));
|
|
19053
19726
|
if (targets.length === 0)
|
|
19054
19727
|
continue;
|
|
19055
|
-
const file =
|
|
19728
|
+
const file = join27(root, report.path);
|
|
19056
19729
|
if (leavesRoot(root, file)) {
|
|
19057
19730
|
io.stdout(` SKIPPED ${report.path}: the path is a link, or it resolves outside the root
|
|
19058
19731
|
`);
|
|
@@ -19178,7 +19851,7 @@ async function runCheck(root, options, io) {
|
|
|
19178
19851
|
`);
|
|
19179
19852
|
return 1;
|
|
19180
19853
|
}
|
|
19181
|
-
async function
|
|
19854
|
+
async function run6(argv, io) {
|
|
19182
19855
|
if (wantsHelp(argv)) {
|
|
19183
19856
|
io.stdout(DOCS_HELP);
|
|
19184
19857
|
return 0;
|
|
@@ -19187,7 +19860,7 @@ async function run5(argv, io) {
|
|
|
19187
19860
|
const stderr = (text) => io.stderr(root ? maskRoot(text, root) : text);
|
|
19188
19861
|
try {
|
|
19189
19862
|
const options = parseDocsArgs(argv);
|
|
19190
|
-
root =
|
|
19863
|
+
root = resolve9(options.root);
|
|
19191
19864
|
if (!isDirectory(root))
|
|
19192
19865
|
throw new Error("the root <root> is not an existing directory");
|
|
19193
19866
|
assertReadableLanguage(root);
|
|
@@ -19206,14 +19879,14 @@ async function run5(argv, io) {
|
|
|
19206
19879
|
|
|
19207
19880
|
// src/query/index.ts
|
|
19208
19881
|
import { existsSync as existsSync12, statSync as statSync10 } from "node:fs";
|
|
19209
|
-
import { resolve as
|
|
19882
|
+
import { resolve as resolve10 } from "node:path";
|
|
19210
19883
|
|
|
19211
19884
|
// src/query/args.ts
|
|
19212
19885
|
var MODE_FLAGS = {
|
|
19213
19886
|
"--emit": { name: "emit" },
|
|
19214
19887
|
"--check-browser-safety": { name: "check-browser-safety" }
|
|
19215
19888
|
};
|
|
19216
|
-
var
|
|
19889
|
+
var VALUE_FLAGS4 = new Set(["--root", "--config", "--out", "--node-runtime"]);
|
|
19217
19890
|
var COMMAND_USAGE = {
|
|
19218
19891
|
dependents: "<file>",
|
|
19219
19892
|
"symbol-users": "<symbol>",
|
|
@@ -19294,7 +19967,7 @@ function parseQueryArgs(argv, cwd) {
|
|
|
19294
19967
|
mode = modeCommand;
|
|
19295
19968
|
continue;
|
|
19296
19969
|
}
|
|
19297
|
-
if (!
|
|
19970
|
+
if (!VALUE_FLAGS4.has(name2)) {
|
|
19298
19971
|
throw new Error(`unknown flag '${name2}' (see repo-tools query --help)`);
|
|
19299
19972
|
}
|
|
19300
19973
|
if (value === undefined || value === "") {
|
|
@@ -19783,12 +20456,12 @@ report, a path that is not a file of the graph, a browser-safety command on a
|
|
|
19783
20456
|
graph that is not TypeScript/JavaScript, or a browser-safety leak with
|
|
19784
20457
|
--check-browser-safety.
|
|
19785
20458
|
`;
|
|
19786
|
-
async function
|
|
20459
|
+
async function run7(argv, io) {
|
|
19787
20460
|
let root;
|
|
19788
20461
|
const stderr = (text) => io.stderr(root ? maskRoot(text, root) : text);
|
|
19789
20462
|
try {
|
|
19790
20463
|
const options = parseQueryArgs(argv, process.cwd());
|
|
19791
|
-
root =
|
|
20464
|
+
root = resolve10(options.root);
|
|
19792
20465
|
if (!(existsSync12(root) && statSync10(root).isDirectory())) {
|
|
19793
20466
|
throw new Error("the root <root> is not an existing directory");
|
|
19794
20467
|
}
|
|
@@ -19824,7 +20497,7 @@ async function run6(argv, io) {
|
|
|
19824
20497
|
|
|
19825
20498
|
// src/ste/index.ts
|
|
19826
20499
|
import { existsSync as existsSync13, lstatSync as lstatSync6, readdirSync as readdirSync5, readFileSync as readFileSync26, statSync as statSync11 } from "node:fs";
|
|
19827
|
-
import { basename as basename10, join as
|
|
20500
|
+
import { basename as basename10, join as join29 } from "node:path";
|
|
19828
20501
|
|
|
19829
20502
|
// src/ste/markdown.ts
|
|
19830
20503
|
function stripNoise(text) {
|
|
@@ -19879,7 +20552,7 @@ function wordCount(sentence) {
|
|
|
19879
20552
|
const s = sentence.replace(CODE_SPAN, "X").replace(LINK, "$1").replace(EMPHASIS_CHARS, "").replace(LIST_MARKER, "");
|
|
19880
20553
|
return pySplit(s).filter((w) => Array.from(w).some(isAlnum)).length;
|
|
19881
20554
|
}
|
|
19882
|
-
function
|
|
20555
|
+
function join28(parts2, start2) {
|
|
19883
20556
|
const chunks = [];
|
|
19884
20557
|
const lineMap = [];
|
|
19885
20558
|
parts2.forEach(([n, body2], i2) => {
|
|
@@ -19909,7 +20582,7 @@ function paragraphs(text) {
|
|
|
19909
20582
|
for (const [n, line] of stripNoise(text)) {
|
|
19910
20583
|
const startsItem = LIST_ITEM.test(line);
|
|
19911
20584
|
if ((n !== prevLine + 1 || startsItem) && para.length > 0) {
|
|
19912
|
-
blocks.push(
|
|
20585
|
+
blocks.push(join28(para, paraStart));
|
|
19913
20586
|
para = [];
|
|
19914
20587
|
}
|
|
19915
20588
|
if (para.length === 0)
|
|
@@ -19918,7 +20591,7 @@ function paragraphs(text) {
|
|
|
19918
20591
|
prevLine = n;
|
|
19919
20592
|
}
|
|
19920
20593
|
if (para.length > 0)
|
|
19921
|
-
blocks.push(
|
|
20594
|
+
blocks.push(join28(para, paraStart));
|
|
19922
20595
|
return blocks;
|
|
19923
20596
|
}
|
|
19924
20597
|
function checkMarkdown(name2, text) {
|
|
@@ -19975,8 +20648,8 @@ identifiers are not checked. A file whose first five lines hold
|
|
|
19975
20648
|
--help, -h Show this help.
|
|
19976
20649
|
|
|
19977
20650
|
Exit codes: 0 when no finding (Markdown mode) or always (--prose mode). 1 when a
|
|
19978
|
-
Markdown finding exists. 2 on a usage error, when
|
|
19979
|
-
or when a file is not UTF-8.
|
|
20651
|
+
Markdown finding exists. 2 on a usage error, when a path does not exist, when no
|
|
20652
|
+
Markdown file was examined, or when a file is not UTF-8.
|
|
19980
20653
|
`;
|
|
19981
20654
|
var HISTORICAL_MARKER = "<!-- ste:historical-record -->";
|
|
19982
20655
|
function readText(path) {
|
|
@@ -20005,7 +20678,7 @@ function targetsFor(path) {
|
|
|
20005
20678
|
return;
|
|
20006
20679
|
}
|
|
20007
20680
|
for (const entry of entries) {
|
|
20008
|
-
const full =
|
|
20681
|
+
const full = join29(dir, entry.name);
|
|
20009
20682
|
if (entry.isDirectory()) {
|
|
20010
20683
|
if (entry.name !== ".pytest_cache")
|
|
20011
20684
|
walk(full, [...parts2, entry.name]);
|
|
@@ -20018,7 +20691,7 @@ function targetsFor(path) {
|
|
|
20018
20691
|
};
|
|
20019
20692
|
if (existsSync13(path) && lstatSync6(path).isDirectory())
|
|
20020
20693
|
walk(path, []);
|
|
20021
|
-
return found.sort(comparePaths).map((parts2) =>
|
|
20694
|
+
return found.sort(comparePaths).map((parts2) => join29(path, ...parts2));
|
|
20022
20695
|
}
|
|
20023
20696
|
function isHistoricalRecord(path) {
|
|
20024
20697
|
let text;
|
|
@@ -20049,7 +20722,7 @@ function runProse(args2, io) {
|
|
|
20049
20722
|
`);
|
|
20050
20723
|
return 0;
|
|
20051
20724
|
}
|
|
20052
|
-
async function
|
|
20725
|
+
async function run8(argv, io) {
|
|
20053
20726
|
if (argv[0] === "--prose")
|
|
20054
20727
|
return runProse(argv.slice(1), io);
|
|
20055
20728
|
if (argv.length === 0) {
|
|
@@ -20060,7 +20733,10 @@ async function run7(argv, io) {
|
|
|
20060
20733
|
let total = 0;
|
|
20061
20734
|
let checked = 0;
|
|
20062
20735
|
let skipped = 0;
|
|
20736
|
+
const missing = [];
|
|
20063
20737
|
for (const arg of argv) {
|
|
20738
|
+
if (!existsSync13(arg))
|
|
20739
|
+
missing.push(arg);
|
|
20064
20740
|
for (const target of targetsFor(arg)) {
|
|
20065
20741
|
const name2 = basename10(target);
|
|
20066
20742
|
if (isHistoricalRecord(target)) {
|
|
@@ -20093,12 +20769,16 @@ async function run7(argv, io) {
|
|
|
20093
20769
|
const tail = skipped > 0 ? `; ${skipped} skipped as historical record(s)` : "";
|
|
20094
20770
|
io.stdout(`
|
|
20095
20771
|
TOTAL: ${total} problem(s) in ${checked} file(s)${tail}
|
|
20772
|
+
`);
|
|
20773
|
+
for (const path of missing)
|
|
20774
|
+
io.stderr(`ERROR: path does not exist: ${path}
|
|
20096
20775
|
`);
|
|
20097
20776
|
if (checked === 0) {
|
|
20098
20777
|
io.stderr(`ERROR: no Markdown file was examined. This is a failure, not a clean result -- check the path(s) given.
|
|
20099
20778
|
`);
|
|
20100
|
-
return 2;
|
|
20101
20779
|
}
|
|
20780
|
+
if (missing.length > 0 || checked === 0)
|
|
20781
|
+
return 2;
|
|
20102
20782
|
return total > 0 ? 1 : 0;
|
|
20103
20783
|
}
|
|
20104
20784
|
|
|
@@ -20124,7 +20804,7 @@ ${MAP_HELP}`,
|
|
|
20124
20804
|
docs: {
|
|
20125
20805
|
summary: "Measure, stub and gate the doc comments of TypeScript, JavaScript and Python source.",
|
|
20126
20806
|
help: DOCS_HELP,
|
|
20127
|
-
run:
|
|
20807
|
+
run: run6
|
|
20128
20808
|
},
|
|
20129
20809
|
chunk: {
|
|
20130
20810
|
summary: "Split a large file into chunks, merge the chunks back, or show changed chunks.",
|
|
@@ -20134,17 +20814,22 @@ ${MAP_HELP}`,
|
|
|
20134
20814
|
compress: {
|
|
20135
20815
|
summary: "Write a compact copy of a file for a model context, or restore it.",
|
|
20136
20816
|
help: HELP,
|
|
20137
|
-
run:
|
|
20817
|
+
run: run5
|
|
20138
20818
|
},
|
|
20139
20819
|
query: {
|
|
20140
20820
|
summary: "Answer structural questions from the depgraph reports, or write derived reports.",
|
|
20141
20821
|
help: QUERY_HELP,
|
|
20142
|
-
run:
|
|
20822
|
+
run: run7
|
|
20143
20823
|
},
|
|
20144
20824
|
ste: {
|
|
20145
20825
|
summary: "Check Markdown or docstring prose against Simplified Technical English.",
|
|
20146
20826
|
help: STE_HELP,
|
|
20147
|
-
run:
|
|
20827
|
+
run: run8
|
|
20828
|
+
},
|
|
20829
|
+
clones: {
|
|
20830
|
+
summary: "List the functions of a repository and find the functions that copy each other.",
|
|
20831
|
+
help: CLONES_HELP,
|
|
20832
|
+
run: run4
|
|
20148
20833
|
}
|
|
20149
20834
|
};
|
|
20150
20835
|
var SUBCOMMANDS = Object.keys(REGISTRY);
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@danielsimonjr/repo-tools",
|
|
3
|
-
"version": "2.
|
|
3
|
+
"version": "2.3.0",
|
|
4
4
|
"description": "One CLI for repository analysis: dependency graphs creation and query with per-export API facts, file chunking, and context compression. TypeScript on Bun, compiled to a single executable.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "MIT",
|