@danielsimonjr/repo-tools 2.2.7 → 2.3.0

package/CHANGELOG.md

@@ -6,6 +6,24 @@ All notable changes to this project are recorded in this file. The format follow
 
 ## [Unreleased]
 
+## [2.3.0] - 2026-10-22
+
+### Added
+
+- `repo-tools clones` lists the functions of a TypeScript or JavaScript repository and finds the functions that copy
+  each other. It reads the source and writes files only when `--out` or the `clones.out` config key names a folder.
+  A unit is a named function with at least 50 labels (`--min-nodes`). The command reports exact groups (the same label
+  sequence) and near pairs (shared runs of labels, scored by Jaccard and containment). It prints a summary, and
+  with an output folder it writes `inventory.jsonl` and `clones.json`. Every limit is a flag and a key of the new
+  `clones` section of `repo-tools.config.json`. A flag overrides the file, and the file overrides the default.
+  Exit code 0 means the run finished, 1 means the root holds no source file to read, and 2 means a usage or config
+  error. `docs/design.md` section 17 gives the rules.
+- `docs/parity-clones.md` compares the command with the prototype of the memoryjs duplicate-function audit on 292
+  files. 1,368 units are the same. The command finds all 20 exact groups of the prototype. Every unit that only one
+  side reports lies at the cutoff of 50 labels.
+- The smoke test runs `clones` on a small repository that holds a planted copy, so the shipped bundle proves the
+  command and its tree-sitter loader.
+
 ## [2.2.7] - 2026-10-22
 
 ### Fixed

package/README.md

@@ -11,13 +11,16 @@
 | `chunk` | Splits a large file into chunks, merges the chunks back, and shows which chunks changed. |
 | `compress` | Writes a compact copy of a file for a model context, and restores it. |
 | `query` | Answers structural questions from the graph of `map`, and writes two derived reports. |
+| `clones` | Lists the functions of a TypeScript or JavaScript repository, and finds the functions that copy each other. |
 | `ste` | Checks Markdown or docstring prose against Simplified Technical English. |
 
-Status: version 2.2.7.
+Status: version 2.3.0.
 
 - `map`, `check` and `query` read their settings from `repo-tools.config.json` in the root. The
   section of `map` is `map` (its old name, `depgraph`, is also read). `--config=<file>` makes any
   of the three read another file.
+- `clones` reads its settings from the `clones` section of the same file. A flag overrides the
+  file, and the file overrides the default.
 - `repo-tools depgraph` prints one deprecation line and runs `map`. It accepts the 1.x
   scan-scope flags (`--src`, `--tests`, `--exclude`, `--also-exclude`, `--all`,
   `--reachable-only`, `--include-tests`) with a warning, because they have no effect in 2.0.0.
@@ -41,7 +44,7 @@ repo-tools --version
 
 The executable holds the tool and the Bun runtime. It needs no Bun, Node or `node_modules`.
 Download the file for your platform and `SHA256SUMS` from the
-[GitHub release](https://github.com/danielsimonjr/repo-tools/releases/tag/v2.2.7):
+[GitHub release](https://github.com/danielsimonjr/repo-tools/releases/tag/v2.3.0):
 
 | Platform | File |
 |---|---|
@@ -311,6 +314,38 @@ repo-tools query --emit
 the config key `query.out`, else from `map.out`, else from `docs/architecture`. Run
 `repo-tools query --help` for every option.
 
+## Find the functions that copy each other
+
+`repo-tools clones` lists the functions of a TypeScript or JavaScript repository. It finds the
+functions that copy each other. The command only reads the repository. It writes files only when
+you name an output folder.
+
+```sh
+repo-tools clones .                              # prints the summary and writes no file
+repo-tools clones . --src=src --out=reports/clones
+repo-tools clones . --min-nodes=30 --near=0.7
+```
+
+A function is a unit when it has a name and its body has at least 50 labels. A label is one node
+of the body. Names, literals, types and comments give no detail, so a copy with other names still
+matches. The command finds two kinds of result:
+
+| Result | Meaning |
+|---|---|
+| Exact group | Units with the same label sequence. |
+| Near pair | Two units that share enough runs of 6 labels. |
+
+`--src=<path>...` names the folders to read. The default `auto` reads the files of the `src` area.
+`--out=<dir>` writes `inventory.jsonl` (one line for each unit) and `clones.json` (the groups and the
+pairs) into a folder. Give the folder as a path relative to the root. Run `repo-tools clones --help` for every limit.
+
+Exit codes: 0 when the run finished, even if it found no copy. 1 when the root holds no source file
+to read. 2 on a usage error, a bad config file or a root that is not a folder.
+
+The result is a list of candidates. Read both bodies of a pair before you change anything.
+`docs/design.md` (section 17) gives the rules. `docs/parity-clones.md` records the comparison with the
+prototype of the memoryjs audit.
+
 ## Develop
 
 | Command | Job |

package/dist/cli.js

@@ -2,7 +2,7 @@
 // package.json
 var package_default = {
   name: "@danielsimonjr/repo-tools",
-  version: "2.2.7",
+  version: "2.3.0",
   description: "One CLI for repository analysis: dependency graphs creation and query with per-export API facts, file chunking, and context compression. TypeScript on Bun, compiled to a single executable.",
   type: "module",
   license: "MIT",
@@ -851,6 +851,16 @@ function checkValue(key, kind, value, fail) {
       return text(value, "a non-empty string");
     case "string-or-null":
       return value === null ? null : text(value, "a non-empty string or null");
+    case "count":
+      if (typeof value !== "number" || !Number.isInteger(value) || value < 1) {
+        throw fail(`'${key}' must be a whole number of 1 or more`);
+      }
+      return value;
+    case "fraction":
+      if (typeof value !== "number" || !(value >= 0 && value <= 1)) {
+        throw fail(`'${key}' must be a number from 0 to 1`);
+      }
+      return value;
   }
 }
 function checkSection(prefix, object, keys, fail) {
@@ -867,6 +877,20 @@ var QUERY_KEYS = {
   out: "path",
   nodeRuntimes: "names"
 };
+var CLONES_KEYS = {
+  src: "src",
+  out: "path-or-null",
+  minNodes: "count",
+  shingleLength: "count",
+  nearThreshold: "fraction",
+  containmentThreshold: "fraction",
+  containmentMinShingles: "count",
+  rareShingleLimit: "count",
+  minSharedShingles: "count"
+};
+function checkClonesFlag(label, key, value) {
+  return checkValue(label, CLONES_KEYS[key], value, (message) => new Error(message));
+}
 function parseConfigSections(parsed, label) {
   const fail = (message) => new Error(`config ${label}: ${message}`);
   if (!isObject(parsed))
@@ -874,7 +898,7 @@ function parseConfigSections(parsed, label) {
   for (const key of Object.keys(parsed)) {
     if (key === "$schema")
       continue;
-    if (key !== "depgraph" && key !== "map" && key !== "query") {
+    if (key !== "depgraph" && key !== "map" && key !== "query" && key !== "clones") {
       throw fail(`unknown key '${key}'`);
     }
   }
@@ -882,7 +906,7 @@ function parseConfigSections(parsed, label) {
     throw fail("use the section 'map' or its old name 'depgraph', not both");
   }
   const sectionName = parsed.map !== undefined ? "map" : "depgraph";
-  const sections = { depgraph: {}, query: {} };
+  const sections = { depgraph: {}, query: {}, clones: {} };
   const section = parsed[sectionName];
   if (section !== undefined) {
     if (!isObject(section))
@@ -901,6 +925,12 @@ function parseConfigSections(parsed, label) {
       throw fail("'query' must be an object");
     sections.query = checkSection("query.", query, QUERY_KEYS, fail);
   }
+  const clones = parsed.clones;
+  if (clones !== undefined) {
+    if (!isObject(clones))
+      throw fail("'clones' must be an object");
+    sections.clones = checkSection("clones.", clones, CLONES_KEYS, fail);
+  }
   return sections;
 }
 function loadConfigSections(root, configPath) {
@@ -911,7 +941,7 @@ function loadConfigSections(root, configPath) {
   const label = `<root>/${rel.replace(/\\/g, "/")}`;
   const path = resolveUnderRoot(root, rel);
   if (configPath === undefined && !existsSync2(path))
-    return { depgraph: {}, query: {} };
+    return { depgraph: {}, query: {}, clones: {} };
   let text;
   try {
     text = readFileSync(path, "utf8");
@@ -16564,9 +16594,655 @@ ${CHUNK_HELP}`);
   }
 }
 
+// src/clones/index.ts
+import { join as join22, resolve as resolve8 } from "node:path";
+
+// src/clones/detect.ts
+function fnv1a(text) {
+  let h = 2166136261;
+  for (let i2 = 0;i2 < text.length; i2++) {
+    h ^= text.charCodeAt(i2);
+    h = Math.imul(h, 16777619);
+  }
+  return h >>> 0;
+}
+function groupsBy(units, key) {
+  const groups = new Map;
+  units.forEach((unit, id) => {
+    const ids = groups.get(key(unit));
+    if (ids)
+      ids.push(id);
+    else
+      groups.set(key(unit), [id]);
+  });
+  return [...groups.values()].filter((ids) => ids.length > 1);
+}
+function shinglesOf(labels, length) {
+  const set = new Set;
+  for (let i2 = 0;i2 + length <= labels.length; i2++) {
+    set.add(fnv1a(labels.slice(i2, i2 + length).join("|")));
+  }
+  return set;
+}
+function nested(a, b) {
+  if (a.file !== b.file)
+    return false;
+  const aInB = b.startIndex <= a.startIndex && a.endIndex <= b.endIndex;
+  const bInA = a.startIndex <= b.startIndex && b.endIndex <= a.endIndex;
+  return aInB || bInA;
+}
+function overlap(small, big) {
+  let shared = 0;
+  for (const item of small)
+    if (big.has(item))
+      shared++;
+  return shared;
+}
+function round3(value) {
+  return Number(value.toFixed(3));
+}
+function detect(units, params) {
+  const exactGroups = groupsBy(units, (u) => u.exactHash);
+  const textGroups = groupsBy(units, (u) => u.textHash);
+  const shingles = units.map((u) => shinglesOf(u.labels, params.shingleLength));
+  const index = new Map;
+  shingles.forEach((set, id) => {
+    for (const shingle of set) {
+      const ids = index.get(shingle);
+      if (ids)
+        ids.push(id);
+      else
+        index.set(shingle, [id]);
+    }
+  });
+  const near = [];
+  units.forEach((unit, id) => {
+    const mine = shingles[id];
+    const shared = new Map;
+    for (const shingle of mine) {
+      const ids = index.get(shingle);
+      if (ids.length > params.rareShingleLimit)
+        continue;
+      for (const other of ids)
+        if (other > id)
+          shared.set(other, (shared.get(other) ?? 0) + 1);
+    }
+    for (const [other, count] of shared) {
+      if (count < params.minSharedShingles)
+        continue;
+      const peer = units[other];
+      if (nested(unit, peer))
+        continue;
+      const theirs = shingles[other];
+      const [small, big] = mine.size <= theirs.size ? [mine, theirs] : [theirs, mine];
+      const inter = overlap(small, big);
+      const jaccard = inter / (mine.size + theirs.size - inter);
+      const contain = inter / small.size;
+      const byContainment = contain >= params.containmentThreshold && small.size >= params.containmentMinShingles;
+      if (jaccard < params.nearThreshold && !byContainment)
+        continue;
+      const common = overlap(unit.names, peer.names);
+      const union = unit.names.size + peer.names.size - common;
+      near.push({
+        a: id,
+        b: other,
+        jStruct: round3(jaccard),
+        contain: round3(contain),
+        jNames: round3(common / (union || 1))
+      });
+    }
+  });
+  near.sort((x, y) => y.jStruct - x.jStruct || x.a - y.a || x.b - y.b);
+  return { exactGroups, textGroups, near };
+}
+
+// src/clones/report.ts
+function inventoryRecord(unit, id) {
+  return {
+    id,
+    file: unit.file,
+    name: unit.name,
+    container: unit.container,
+    kind: unit.kind,
+    exported: unit.exported,
+    startLine: unit.startLine,
+    endLine: unit.endLine,
+    loc: unit.loc,
+    params: unit.params,
+    async: unit.async,
+    nodes: unit.nodes,
+    doc: unit.doc,
+    exactHash: unit.exactHash,
+    textHash: unit.textHash
+  };
+}
+function inventoryJsonl(units) {
+  return units.map((unit, id) => `${JSON.stringify(inventoryRecord(unit, id))}
+`).join("");
+}
+function listJson(items) {
+  if (items.length === 0)
+    return "[]";
+  return `[
+${items.map((item) => `    ${JSON.stringify(item)}`).join(`,
+`)}
+  ]`;
+}
+function clonesJson(config, detection) {
+  const { out: _out, ...params } = config;
+  return [
+    "{",
+    `  "params": ${JSON.stringify(params)},`,
+    `  "exactGroups": ${listJson(detection.exactGroups)},`,
+    `  "textGroups": ${listJson(detection.textGroups)},`,
+    `  "near": ${listJson(detection.near)}`,
+    "}",
+    ""
+  ].join(`
+`);
+}
+var SUMMARY_BANDS = [
+  ["0.9 or more", 0.9, Number.POSITIVE_INFINITY],
+  ["0.8 to 0.9", 0.8, 0.9],
+  ["0.7 to 0.8", 0.7, 0.8],
+  ["0.6 to 0.7", 0.6, 0.7],
+  ["below 0.6 (containment)", 0, 0.6]
+];
+function folderOf(file) {
+  const parts2 = file.split("/").slice(0, -1);
+  return parts2.length === 0 ? "." : parts2.slice(0, 2).join("/");
+}
+function percentile(sorted, p) {
+  return sorted[Math.min(sorted.length - 1, Math.floor(sorted.length * p))] ?? 0;
+}
+function summaryText(config, stats, units, detection) {
+  const sizes = units.map((u) => u.nodes).sort((a, b) => a - b);
+  const inGroups = detection.exactGroups.reduce((sum, group) => sum + group.length, 0);
+  const bands = SUMMARY_BANDS.map(([label, lo, hi]) => `${label}: ${detection.near.filter((p) => p.jStruct >= lo && p.jStruct < hi).length}`);
+  const byFolder = new Map;
+  for (const unit of units) {
+    const key = folderOf(unit.file);
+    byFolder.set(key, (byFolder.get(key) ?? 0) + 1);
+  }
+  const folders = [...byFolder.entries()].sort(([a], [b]) => a < b ? -1 : a > b ? 1 : 0).map(([key, count]) => `${key}=${count}`).join(" ");
+  const lines = [
+    `files=${stats.files} parseErrors=${stats.parseErrors} units(minNodes=${config.minNodes})=${units.length}`,
+    `nodes p50=${percentile(sizes, 0.5)} p90=${percentile(sizes, 0.9)} p99=${percentile(sizes, 0.99)} max=${sizes[sizes.length - 1] ?? 0}`,
+    `exact groups=${detection.exactGroups.length} (units in them=${inGroups}) text-identical groups=${detection.textGroups.length}`,
+    `near pairs=${detection.near.length} by jStruct: ${bands.join(", ")}`,
+    `units by folder: ${folders}`
+  ];
+  return `${lines.join(`
+`)}
+`;
+}
+
+// src/clones/settings.ts
+var CLONES_DEFAULTS = Object.freeze({
+  src: "auto",
+  out: null,
+  minNodes: 50,
+  shingleLength: 6,
+  nearThreshold: 0.6,
+  containmentThreshold: 0.9,
+  containmentMinShingles: 20,
+  rareShingleLimit: 60,
+  minSharedShingles: 3
+});
+function mergeClonesConfig(cli, file) {
+  const d = CLONES_DEFAULTS;
+  const src = cli.src ?? file.src ?? d.src;
+  return {
+    src: src === "auto" ? src : [...src],
+    out: cli.out !== undefined ? cli.out : file.out !== undefined ? file.out : d.out,
+    minNodes: cli.minNodes ?? file.minNodes ?? d.minNodes,
+    shingleLength: cli.shingleLength ?? file.shingleLength ?? d.shingleLength,
+    nearThreshold: cli.nearThreshold ?? file.nearThreshold ?? d.nearThreshold,
+    containmentThreshold: cli.containmentThreshold ?? file.containmentThreshold ?? d.containmentThreshold,
+    containmentMinShingles: cli.containmentMinShingles ?? file.containmentMinShingles ?? d.containmentMinShingles,
+    rareShingleLimit: cli.rareShingleLimit ?? file.rareShingleLimit ?? d.rareShingleLimit,
+    minSharedShingles: cli.minSharedShingles ?? file.minSharedShingles ?? d.minSharedShingles
+  };
+}
+
+// src/clones/units.ts
+import { createHash as createHash2 } from "node:crypto";
+
+// src/clones/normalise.ts
+var IDENTIFIERS = new Set([
+  "identifier",
+  "property_identifier",
+  "shorthand_property_identifier",
+  "shorthand_property_identifier_pattern",
+  "private_property_identifier",
+  "statement_identifier",
+  "undefined"
+]);
+var LITERALS = new Set(["string", "number", "regex", "true", "false", "null"]);
+var TYPE_SYNTAX = new Set([
+  "type_annotation",
+  "type_arguments",
+  "type_parameters",
+  "type_alias_declaration",
+  "interface_declaration",
+  "enum_declaration",
+  "ambient_declaration",
+  "asserts_annotation",
+  "opting_type_annotation",
+  "omitting_type_annotation",
+  "type_predicate_annotation",
+  "accessibility_modifier",
+  "override_modifier",
+  "decorator",
+  "comment"
+]);
+var TRANSPARENT = new Set([
+  "parenthesized_expression",
+  "as_expression",
+  "satisfies_expression",
+  "non_null_expression",
+  "type_assertion",
+  "instantiation_expression"
+]);
+var TEMPLATE_TEXT = new Set(["string_fragment", "escape_sequence"]);
+function wrapped(node) {
+  const children = node.namedChildren;
+  return (node.type === "type_assertion" ? children[children.length - 1] : children[0]) ?? null;
+}
+function walk2(node, out2) {
+  if (!node.isNamed)
+    return;
+  const type = node.type;
+  if (TYPE_SYNTAX.has(type))
+    return;
+  if (TRANSPARENT.has(type)) {
+    const inner = wrapped(node);
+    if (inner)
+      walk2(inner, out2);
+    return;
+  }
+  if (IDENTIFIERS.has(type)) {
+    out2.labels.push("ID");
+    out2.names.add(node.text.toLowerCase());
+    return;
+  }
+  if (LITERALS.has(type)) {
+    out2.labels.push("LIT");
+    return;
+  }
+  if (TEMPLATE_TEXT.has(type)) {
+    out2.labels.push("TPL");
+    return;
+  }
+  if (type !== "template_substitution") {
+    const operator = node.childForFieldName("operator");
+    out2.labels.push(operator ? `${type}:${operator.type}` : type);
+  }
+  for (const child of node.children)
+    walk2(child, out2);
+}
+function normalise(body2) {
+  const out2 = { labels: [], names: new Set };
+  walk2(body2, out2);
+  return out2;
+}
+
+// src/clones/units.ts
+var DOC_LIMIT = 160;
+var FUNCTION_VALUES = new Set([
+  "arrow_function",
+  "function_expression",
+  "function",
+  "generator_function"
+]);
+var CLASSES = new Set(["class_declaration", "abstract_class_declaration", "class"]);
+var ANONYMOUS_CLASS = "(class)";
+function sha1(text) {
+  return createHash2("sha1").update(text).digest("hex").slice(0, 16);
+}
+function keyName(key) {
+  if (!key)
+    return null;
+  return key.type === "string" ? key.text.slice(1, -1) : key.text;
+}
+function hasToken(node, token) {
+  return node.children.some((c) => !c.isNamed && c.type === token);
+}
+function anchorOf(node) {
+  let anchor = node;
+  if (node.type === "variable_declarator" && node.parent)
+    anchor = node.parent;
+  if (anchor.parent?.type === "export_statement")
+    anchor = anchor.parent;
+  return anchor;
+}
+function declared(node, container) {
+  switch (node.type) {
+    case "function_declaration":
+    case "generator_function_declaration": {
+      const name2 = keyName(node.childForFieldName("name"));
+      if (!name2)
+        return null;
+      return { fn: node, name: name2, kind: container ? "local" : "function", anchor: anchorOf(node) };
+    }
+    case "method_definition": {
+      const name2 = keyName(node.childForFieldName("name"));
+      if (!name2)
+        return null;
+      const inClass = node.parent?.type === "class_body";
+      let kind = "prop";
+      if (inClass) {
+        if (name2 === "constructor")
+          kind = "ctor";
+        else if (hasToken(node, "get"))
+          kind = "get";
+        else if (hasToken(node, "set"))
+          kind = "set";
+        else
+          kind = "method";
+      }
+      return { fn: node, name: name2, kind, anchor: node };
+    }
+    case "public_field_definition": {
+      const value = node.childForFieldName("value");
+      const name2 = keyName(node.childForFieldName("name"));
+      if (!value || !name2 || !FUNCTION_VALUES.has(value.type))
+        return null;
+      return { fn: value, name: name2, kind: "field", anchor: node };
+    }
+    case "variable_declarator": {
+      const value = node.childForFieldName("value");
+      const id = node.childForFieldName("name");
+      if (!value || !id || id.type !== "identifier" || !FUNCTION_VALUES.has(value.type)) {
+        return null;
+      }
+      return {
+        fn: value,
+        name: id.text,
+        kind: container ? "local" : "function",
+        anchor: anchorOf(node)
+      };
+    }
+    case "pair": {
+      const value = node.childForFieldName("value");
+      const name2 = keyName(node.childForFieldName("key"));
+      if (!value || !name2 || !FUNCTION_VALUES.has(value.type))
+        return null;
+      return { fn: value, name: name2, kind: "prop", anchor: node };
+    }
+    default:
+      return null;
+  }
+}
+function paramCount(fn) {
+  const list = fn.childForFieldName("parameters");
+  if (list)
+    return list.namedChildren.filter((c) => c.type !== "comment").length;
+  return fn.childForFieldName("parameter") ? 1 : 0;
+}
+function docOf(anchor, source) {
+  const comment = anchor.previousSibling;
+  if (comment?.type !== "comment" || !comment.text.startsWith("/**"))
+    return "";
+  if (source.slice(comment.endIndex, anchor.startIndex).trim() !== "")
+    return "";
+  const lines = comment.text.slice(3, -2).split(`
+`).map((line) => line.replace(/^\s*\*\s?/, "").trim());
+  const tag = lines.findIndex((line) => line.startsWith("@"));
+  const text = (tag === -1 ? lines : lines.slice(0, tag)).filter((line) => line !== "");
+  return text.join(" ").slice(0, DOC_LIMIT);
+}
+function buildUnit(d, container, source, file, minNodes) {
+  const body2 = d.fn.childForFieldName("body");
+  if (!body2)
+    return null;
+  const { labels, names } = normalise(body2);
+  if (labels.length < minNodes)
+    return null;
+  const startLine = d.anchor.startPosition.row + 1;
+  const endLine = d.fn.endPosition.row + 1;
+  return {
+    file,
+    name: d.name,
+    container,
+    kind: d.kind,
+    exported: d.anchor.type === "export_statement",
+    startLine,
+    endLine,
+    loc: endLine - startLine + 1,
+    params: paramCount(d.fn),
+    async: hasToken(d.fn, "async"),
+    nodes: labels.length,
+    doc: docOf(d.anchor, source),
+    exactHash: sha1(labels.join(" ")),
+    textHash: sha1(source.slice(body2.startIndex, body2.endIndex).replace(/\s+/g, "")),
+    startIndex: d.fn.startIndex,
+    endIndex: d.fn.endIndex,
+    labels,
+    names
+  };
+}
+function unitsOf(root, source, file, minNodes) {
+  const units = [];
+  const visit = (node, container) => {
+    const found = declared(node, container);
+    if (found) {
+      const unit = buildUnit(found, container, source, file, minNodes);
+      if (unit)
+        units.push(unit);
+    }
+    let inner = container;
+    if (CLASSES.has(node.type)) {
+      inner = keyName(node.childForFieldName("name")) ?? (container || ANONYMOUS_CLASS);
+    }
+    for (const child of node.namedChildren)
+      visit(child, inner);
+  };
+  visit(root, "");
+  return units;
+}
+
+// src/clones/index.ts
+var CLONES_HELP = `Usage: repo-tools clones [root] [--out=<dir>] [--config=<path>] [--src=<path>...]
+                         [--min-nodes=<n>] [--shingle=<n>] [--near=<f>]
+                         [--containment=<f>] [--containment-min=<n>]
+                         [--rare=<n>] [--min-shared=<n>]
+
+List the functions of a TypeScript or JavaScript repository and find the functions
+that copy each other. The command only reads the repository. The result is a list
+of candidates: read both bodies of a pair before you change anything.
+
+A function is a unit when it has a name and its body has at least --min-nodes
+labels. A label is one node of the body, with names, literals, types and comments
+erased. Units with the same label sequence are an exact group. Two units whose
+shingles (runs of --shingle labels) overlap enough are a near pair.
+
+  --out=<dir>           Write inventory.jsonl and clones.json to this folder.
+                        Without it the command writes no file.
+  --config=<path>       The config file. The default is repo-tools.config.json at
+                        the root. The settings are in its "clones" section.
+  --src=<path>...       The folders to read. The word "auto" (the default) reads
+                        the files of the src area. The flag takes every argument
+                        up to the next flag.
+  --min-nodes=<n>       The smallest number of labels of a unit. Default 50.
+  --shingle=<n>         The labels in one shingle. Default 6.
+  --near=<f>            The smallest Jaccard score of a near pair. Default 0.6.
+  --containment=<f>     The smallest containment of a near pair. Default 0.9.
+  --containment-min=<n> The smallest shingle count of the smaller unit for the
+                        containment rule. Default 20.
+  --rare=<n>            A shingle in more units than this does not pick candidate
+                        pairs. Default 60.
+  --min-shared=<n>      The rare shingles that a candidate pair must share.
+                        Default 3.
+  --root=<path>         The root. The default is the current folder.
+  --help, -h            Show this help.
+
+A flag overrides the config file, and the config file overrides the default.
+
+Exit codes: 0 when the run finished. 1 when the root holds no source file to read.
+2 on a usage error, a bad config file or a root that is not a folder.
+`;
+
+class NothingToRead extends Error {
+}
+var VALUE_FLAGS2 = {
+  "--out": "out",
+  "--min-nodes": "minNodes",
+  "--shingle": "shingleLength",
+  "--near": "nearThreshold",
+  "--containment": "containmentThreshold",
+  "--containment-min": "containmentMinShingles",
+  "--rare": "rareShingleLimit",
+  "--min-shared": "minSharedShingles"
+};
+var NUMERIC = new Set([
+  "minNodes",
+  "shingleLength",
+  "nearThreshold",
+  "containmentThreshold",
+  "containmentMinShingles",
+  "rareShingleLimit",
+  "minSharedShingles"
+]);
+function parseArgs2(argv) {
+  const options = { root: ".", config: undefined, settings: {} };
+  const seen = new Set;
+  let rootSet = false;
+  const setRoot = (value) => {
+    if (rootSet)
+      throw new Error("the root is set twice (use --root=<path> or one positional path)");
+    rootSet = true;
+    options.root = value;
+  };
+  const once = (flag) => {
+    if (seen.has(flag))
+      throw new Error(`flag ${flag} is set twice`);
+    seen.add(flag);
+  };
+  for (let i2 = 0;i2 < argv.length; i2++) {
+    const arg = argv[i2] ?? "";
+    if (!arg.startsWith("-")) {
+      setRoot(arg);
+      continue;
+    }
+    const eq = arg.indexOf("=");
+    const name2 = eq === -1 ? arg : arg.slice(0, eq);
+    if (name2 === "--src") {
+      once(name2);
+      const paths = [];
+      if (eq !== -1)
+        paths.push(arg.slice(eq + 1));
+      else
+        while (i2 + 1 < argv.length && !(argv[i2 + 1] ?? "").startsWith("-")) {
+          paths.push(argv[++i2] ?? "");
+        }
+      if (paths.length === 0 || paths.some((p) => p === "")) {
+        throw new Error("flag --src needs a value: --src=<path>");
+      }
+      const only = paths.length === 1 ? paths[0] : undefined;
+      options.settings.src = checkClonesFlag(name2, "src", only === "auto" ? "auto" : paths);
+      continue;
+    }
+    if (name2 !== "--root" && name2 !== "--config" && !Object.hasOwn(VALUE_FLAGS2, name2)) {
+      throw new Error(`unknown flag '${name2}' (see repo-tools clones --help)`);
+    }
+    once(name2);
+    let value = eq === -1 ? undefined : arg.slice(eq + 1);
+    if (eq === -1 && i2 + 1 < argv.length && !(argv[i2 + 1] ?? "").startsWith("-")) {
+      value = argv[++i2];
+    }
+    if (value === undefined || value === "") {
+      throw new Error(`flag ${name2} needs a value: ${name2}=<value>`);
+    }
+    if (name2 === "--root")
+      setRoot(value);
+    else if (name2 === "--config")
+      options.config = value;
+    else {
+      const key = VALUE_FLAGS2[name2];
+      const raw = NUMERIC.has(key) ? /^\d+(\.\d+)?$/.test(value) ? Number(value) : NaN : value;
+      Object.assign(options.settings, { [key]: checkClonesFlag(name2, key, raw) });
+    }
+  }
+  return options;
+}
+function folderPrefix(path) {
+  const posix = path.replace(/\\/g, "/").replace(/^(\.\/)+/, "").replace(/\/+$/, "");
+  return posix === "." ? "" : posix;
+}
+function selectFiles(root, src) {
+  const found = discover(root, "typescript").filter((f) => !f.path.endsWith(".d.ts"));
+  if (src === "auto")
+    return found.filter((f) => f.area === "src").map((f) => f.path);
+  const folders = src.map(folderPrefix);
+  return found.filter((f) => folders.some((d) => d === "" || f.path === d || f.path.startsWith(`${d}/`))).map((f) => f.path);
+}
+async function readUnits(root, files, minNodes) {
+  const units = [];
+  let parseErrors = 0;
+  for (const file of files) {
+    const source = readSource(join22(root, file)).replace(/^\uFEFF/, "");
+    const grammar = grammarFor(file);
+    await loadGrammar(grammar);
+    const tree = parserFor(grammar).parse(source);
+    if (!tree) {
+      parseErrors += 1;
+      continue;
+    }
+    try {
+      if (tree.rootNode.hasError)
+        parseErrors += 1;
+      units.push(...unitsOf(tree.rootNode, source, file, minNodes));
+    } finally {
+      tree.delete();
+    }
+  }
+  return { units, parseErrors };
+}
+async function execute(root, config, io) {
+  const files = selectFiles(root, config.src);
+  if (files.length === 0) {
+    throw new NothingToRead(config.src === "auto" ? "no TypeScript or JavaScript file in the src area (set clones.src or pass --src=<path>)" : "no TypeScript or JavaScript file below the listed folders");
+  }
+  const { units, parseErrors } = await readUnits(root, files, config.minNodes);
+  const detection = detect(units, config);
+  io.stdout(summaryText(config, { files: files.length, parseErrors }, units, detection));
+  if (config.out !== null) {
+    const out2 = resolve8(root, config.out);
+    try {
+      writeLf(join22(out2, "inventory.jsonl"), inventoryJsonl(units));
+      writeLf(join22(out2, "clones.json"), clonesJson(config, detection));
+    } catch {
+      throw new Error(`cannot write the output folder '${config.out}'`);
+    }
+    const shown = config.out.replace(/\\/g, "/").replace(/\/+$/, "");
+    io.stdout(`wrote ${shown}/inventory.jsonl
+wrote ${shown}/clones.json
+`);
+  }
+  return 0;
+}
+async function run4(argv, io) {
+  let root;
+  const stderr = (text) => io.stderr(root ? maskRoot(text, root) : text);
+  try {
+    const options = parseArgs2(argv);
+    root = resolve8(options.root);
+    if (!isDirectory(root))
+      throw new Error("the root <root> is not an existing directory");
+    const file = loadConfigSections(root, options.config).clones;
+    return await execute(root, mergeClonesConfig(options.settings, file), io);
+  } catch (error) {
+    const message = error instanceof Error ? error.message : String(error);
+    stderr(`repo-tools clones: ${message}
+`);
+    return error instanceof NothingToRead ? 1 : 2;
+  }
+}
+
 // src/compress/index.ts
 import { existsSync as existsSync9, readdirSync as readdirSync3, readFileSync as readFileSync20, statSync as statSync8, writeFileSync as writeFileSync4 } from "node:fs";
-import { basename as basename8, dirname as dirname8, extname as extname4, join as join22 } from "node:path";
+import { basename as basename8, dirname as dirname8, extname as extname4, join as join23 } from "node:path";
 
 // src/compress/formats.ts
 import { extname as extname3 } from "node:path";
@@ -17199,7 +17875,7 @@ a value, a missing input file and two inputs without --batch are errors. No file
 var defaultDeps = {
   readdir: (dir) => readdirSync3(dir, { withFileTypes: true })
 };
-function parseArgs2(args2) {
+function parseArgs3(args2) {
   const o = {
     input: "",
     inputs: [],
@@ -17322,7 +17998,7 @@ function findFiles(dir, pattern, recursive, deps = defaultDeps) {
   const scan = (current) => {
     const entries = [...deps.readdir(current)].sort((a, b) => compareCodeUnits(a.name, b.name));
     for (const entry of entries) {
-      const full = join22(current, entry.name);
+      const full = join23(current, entry.name);
       if (entry.isDirectory()) {
         if (recursive && !entry.name.startsWith(".") && entry.name !== "node_modules")
           scan(full);
@@ -17339,16 +18015,16 @@ function isDirectory2(path) {
 }
 function compactName(file) {
   const ext = extname4(file);
-  return join22(dirname8(file), `${basename8(file, ext)}.compact${ext}`);
+  return join23(dirname8(file), `${basename8(file, ext)}.compact${ext}`);
 }
 function restoredName(file) {
   const base = basename8(file);
   const stripped = base.replace(/\.compact(?=\.[^.]*$|$)/i, "");
-  const output = stripped === base ? file : join22(dirname8(file), stripped);
+  const output = stripped === base ? file : join23(dirname8(file), stripped);
   if (output !== file)
     return output;
   const ext = extname4(file);
-  return join22(dirname8(file), `${basename8(file, ext)}.restored${ext}`);
+  return join23(dirname8(file), `${basename8(file, ext)}.restored${ext}`);
 }
 function assertCanRestore(output, o) {
   if (!o.dryRun && !o.yes && existsSync9(output)) {
@@ -17508,10 +18184,10 @@ Level: ${o.level}
   }
   return 0;
 }
-async function run4(argv, io, deps = defaultDeps) {
+async function run5(argv, io, deps = defaultDeps) {
   let o;
   try {
-    o = parseArgs2(argv);
+    o = parseArgs3(argv);
   } catch (error) {
     if (!(error instanceof UsageError))
       throw error;
@@ -17562,16 +18238,16 @@ async function run4(argv, io, deps = defaultDeps) {
 
 // src/docs/index.ts
 import { lstatSync as lstatSync5, readFileSync as readFileSync23, realpathSync as realpathSync3, writeFileSync as writeFileSync5 } from "node:fs";
-import { basename as basename9, isAbsolute as isAbsolute4, join as join26, relative as relative5, resolve as resolve8, sep as sep4 } from "node:path";
+import { basename as basename9, isAbsolute as isAbsolute4, join as join27, relative as relative5, resolve as resolve9, sep as sep4 } from "node:path";
 
 // src/docs/analyse.ts
 import { readFileSync as readFileSync22 } from "node:fs";
-import { join as join24 } from "node:path";
+import { join as join25 } from "node:path";
 
 // src/docs/discovery.ts
 import { spawnSync as spawnSync2 } from "node:child_process";
 import { existsSync as existsSync10, readdirSync as readdirSync4, readFileSync as readFileSync21, statSync as statSync9 } from "node:fs";
-import { join as join23 } from "node:path";
+import { join as join24 } from "node:path";
 var PYTHON_SUFFIXES2 = new Set([".py"]);
 var TYPESCRIPT_SUFFIXES = new Set([".ts", ".tsx", ".js", ".jsx", ".mjs", ".cjs"]);
 var SKIP_DIRS2 = new Set([
@@ -17655,8 +18331,8 @@ function walkSource(root) {
       const rel = `${prefix}${entry.name}`;
       if (entry.isDirectory()) {
         if (!SKIP_DIRS2.has(entry.name))
-          go(join23(dir, entry.name), `${rel}/`);
-      } else if (isSource(rel) && statOk(join23(dir, entry.name))) {
+          go(join24(dir, entry.name), `${rel}/`);
+      } else if (isSource(rel) && statOk(join24(dir, entry.name))) {
         found.push(rel);
       }
     }
@@ -17672,7 +18348,7 @@ function statOk(path) {
   }
 }
 function loadExclusions(root) {
-  const file = join23(root, CONFIG_NAME);
+  const file = join24(root, CONFIG_NAME);
   if (!existsSync10(file) || !statOk(file))
     return [];
   let data;
@@ -18284,9 +18960,9 @@ var EXPORT_TYPE_STAR = new RegExp(`(\\bexport${SPACE_CLASS}+)type(${SPACE_CLASS}
 function sanitise(source) {
   return source.replace(ABSTRACT_PROPERTY, "abstrac_$1").replace(EXPORT_TYPE_STAR, "$1    $2");
 }
-var IDENTIFIERS = new Set(["identifier", "shorthand_property_identifier_pattern"]);
+var IDENTIFIERS2 = new Set(["identifier", "shorthand_property_identifier_pattern"]);
 function boundIdentifiers(node, text) {
-  if (IDENTIFIERS.has(node.type))
+  if (IDENTIFIERS2.has(node.type))
     return [text(node)];
   const pattern = node.childForFieldName("pattern");
   if (pattern)
@@ -18304,7 +18980,7 @@ function boundIdentifiers(node, text) {
     case "rest_pattern":
       return into(node.namedChildren[0] ?? null);
     default: {
-      const first = node.namedChildren.find((c) => IDENTIFIERS.has(c.type));
+      const first = node.namedChildren.find((c) => IDENTIFIERS2.has(c.type));
       return first ? [text(first)] : [];
     }
   }
@@ -18313,7 +18989,7 @@ function paramsOf2(node, text) {
   const list = node.childForFieldName("parameters");
   return (list?.namedChildren ?? []).flatMap((param) => boundIdentifiers(param, text));
 }
-function anchorOf(node) {
+function anchorOf2(node) {
   let target = node;
   while (target.parent?.type === "export_statement")
     target = target.parent;
@@ -18470,7 +19146,7 @@ async function analyseTypeScript(path, source) {
     const text = (node) => source.slice(node.startIndex, node.endIndex);
     for (const node of tree.rootNode.descendantsOfType(DECLARATION_TYPES)) {
       const kind = DECLARATIONS[node.type];
-      const anchor = anchorOf(node);
+      const anchor = anchorOf2(node);
       const lead = stubAnchor(anchor, source, text);
       const doc = leadingDoc(anchor, node, text);
       const dialect = dialectOf(doc);
@@ -18527,7 +19203,7 @@ async function analyseFile(root, rel) {
   const language = languageOf(rel);
   let source;
   try {
-    source = readSourceText2(join24(root, rel));
+    source = readSourceText2(join25(root, rel));
   } catch (error) {
     const report = newReport(rel, language);
     report.error = `unreadable: ${readFailure(error)}`;
@@ -18544,7 +19220,7 @@ async function analyseRepo(root) {
 }
 
 // src/docs/report.ts
-import { join as join25 } from "node:path";
+import { join as join26 } from "node:path";
 function percent3(part, whole) {
   if (whole === 0)
     return 100;
@@ -18656,8 +19332,8 @@ function symbolJson(s) {
   };
 }
 function writeOutputs(outDir, reports, stats, provenance, rootName) {
-  const markdown = join25(outDir, "COVERAGE.md");
-  const json = join25(outDir, "coverage.json");
+  const markdown = join26(outDir, "COVERAGE.md");
+  const json = join26(outDir, "coverage.json");
   writeReport(markdown, renderMarkdown(reports, stats, provenance, rootName));
   const payload = {
     summary: statsJson(stats),
@@ -18861,7 +19537,7 @@ Exit codes: 0 when the action succeeded and the gate passed. 1 when the gate
 failed, or when the root holds no source file to measure. 2 on a usage error,
 when the root is not a folder, or when ${CONFIG_NAME} is malformed.
 `;
-var VALUE_FLAGS2 = {
+var VALUE_FLAGS3 = {
   "--root": "all",
   "--out": "scan",
   "--path": "stub",
@@ -18914,8 +19590,8 @@ function parseDocsArgs(argv) {
         while (i2 + 1 < argv.length && !(argv[i2 + 1] ?? "").startsWith("-"))
           options.paths.push(argv[++i2] ?? "");
       }
-    } else if (Object.hasOwn(VALUE_FLAGS2, name2)) {
-      forbid(name2, VALUE_FLAGS2[name2]);
+    } else if (Object.hasOwn(VALUE_FLAGS3, name2)) {
+      forbid(name2, VALUE_FLAGS3[name2]);
       let value = eq === -1 ? undefined : arg.slice(eq + 1);
       if (eq === -1 && i2 + 1 < argv.length && !(argv[i2 + 1] ?? "").startsWith("-"))
         value = argv[++i2];
@@ -19009,7 +19685,7 @@ async function runScan(root, options, io) {
   if (reports.length === 0)
     throw new NothingToMeasure(provenance);
   const stats = summarise(reports);
-  const outDir = resolve8(options.out ?? join26(root, "docs", "code-docs"));
+  const outDir = resolve9(options.out ?? join27(root, "docs", "code-docs"));
   const written = writeOutputs(outDir, reports, stats, provenance, basename9(root));
   io.stdout(`scanned ${stats.filesScanned} file(s) via ${provenance}
 `);
@@ -19049,7 +19725,7 @@ async function runStub(root, options, io) {
     const targets = report.symbols.filter((s) => s.issues.some((i2) => i2.rule === "M1"));
     if (targets.length === 0)
       continue;
-    const file = join26(root, report.path);
+    const file = join27(root, report.path);
     if (leavesRoot(root, file)) {
       io.stdout(`  SKIPPED ${report.path}: the path is a link, or it resolves outside the root
 `);
@@ -19175,7 +19851,7 @@ async function runCheck(root, options, io) {
 `);
   return 1;
 }
-async function run5(argv, io) {
+async function run6(argv, io) {
   if (wantsHelp(argv)) {
     io.stdout(DOCS_HELP);
     return 0;
@@ -19184,7 +19860,7 @@ async function run5(argv, io) {
   const stderr = (text) => io.stderr(root ? maskRoot(text, root) : text);
   try {
     const options = parseDocsArgs(argv);
-    root = resolve8(options.root);
+    root = resolve9(options.root);
     if (!isDirectory(root))
       throw new Error("the root <root> is not an existing directory");
     assertReadableLanguage(root);
@@ -19203,14 +19879,14 @@ async function run5(argv, io) {
 
 // src/query/index.ts
 import { existsSync as existsSync12, statSync as statSync10 } from "node:fs";
-import { resolve as resolve9 } from "node:path";
+import { resolve as resolve10 } from "node:path";
 
 // src/query/args.ts
 var MODE_FLAGS = {
   "--emit": { name: "emit" },
   "--check-browser-safety": { name: "check-browser-safety" }
 };
-var VALUE_FLAGS3 = new Set(["--root", "--config", "--out", "--node-runtime"]);
+var VALUE_FLAGS4 = new Set(["--root", "--config", "--out", "--node-runtime"]);
 var COMMAND_USAGE = {
   dependents: "<file>",
   "symbol-users": "<symbol>",
@@ -19291,7 +19967,7 @@ function parseQueryArgs(argv, cwd) {
       mode = modeCommand;
       continue;
     }
-    if (!VALUE_FLAGS3.has(name2)) {
+    if (!VALUE_FLAGS4.has(name2)) {
       throw new Error(`unknown flag '${name2}' (see repo-tools query --help)`);
     }
     if (value === undefined || value === "") {
@@ -19780,12 +20456,12 @@ report, a path that is not a file of the graph, a browser-safety command on a
 graph that is not TypeScript/JavaScript, or a browser-safety leak with
 --check-browser-safety.
 `;
-async function run6(argv, io) {
+async function run7(argv, io) {
   let root;
   const stderr = (text) => io.stderr(root ? maskRoot(text, root) : text);
   try {
     const options = parseQueryArgs(argv, process.cwd());
-    root = resolve9(options.root);
+    root = resolve10(options.root);
     if (!(existsSync12(root) && statSync10(root).isDirectory())) {
       throw new Error("the root <root> is not an existing directory");
     }
@@ -19821,7 +20497,7 @@ async function run6(argv, io) {
 
 // src/ste/index.ts
 import { existsSync as existsSync13, lstatSync as lstatSync6, readdirSync as readdirSync5, readFileSync as readFileSync26, statSync as statSync11 } from "node:fs";
-import { basename as basename10, join as join28 } from "node:path";
+import { basename as basename10, join as join29 } from "node:path";
 
 // src/ste/markdown.ts
 function stripNoise(text) {
@@ -19876,7 +20552,7 @@ function wordCount(sentence) {
   const s = sentence.replace(CODE_SPAN, "X").replace(LINK, "$1").replace(EMPHASIS_CHARS, "").replace(LIST_MARKER, "");
   return pySplit(s).filter((w) => Array.from(w).some(isAlnum)).length;
 }
-function join27(parts2, start2) {
+function join28(parts2, start2) {
   const chunks = [];
   const lineMap = [];
   parts2.forEach(([n, body2], i2) => {
@@ -19906,7 +20582,7 @@ function paragraphs(text) {
   for (const [n, line] of stripNoise(text)) {
     const startsItem = LIST_ITEM.test(line);
     if ((n !== prevLine + 1 || startsItem) && para.length > 0) {
-      blocks.push(join27(para, paraStart));
+      blocks.push(join28(para, paraStart));
       para = [];
     }
     if (para.length === 0)
@@ -19915,7 +20591,7 @@ function paragraphs(text) {
     prevLine = n;
   }
   if (para.length > 0)
-    blocks.push(join27(para, paraStart));
+    blocks.push(join28(para, paraStart));
   return blocks;
 }
 function checkMarkdown(name2, text) {
@@ -20002,7 +20678,7 @@ function targetsFor(path) {
       return;
     }
     for (const entry of entries) {
-      const full = join28(dir, entry.name);
+      const full = join29(dir, entry.name);
       if (entry.isDirectory()) {
         if (entry.name !== ".pytest_cache")
           walk(full, [...parts2, entry.name]);
@@ -20015,7 +20691,7 @@ function targetsFor(path) {
   };
   if (existsSync13(path) && lstatSync6(path).isDirectory())
     walk(path, []);
-  return found.sort(comparePaths).map((parts2) => join28(path, ...parts2));
+  return found.sort(comparePaths).map((parts2) => join29(path, ...parts2));
 }
 function isHistoricalRecord(path) {
   let text;
@@ -20046,7 +20722,7 @@ function runProse(args2, io) {
 `);
   return 0;
 }
-async function run7(argv, io) {
+async function run8(argv, io) {
   if (argv[0] === "--prose")
     return runProse(argv.slice(1), io);
   if (argv.length === 0) {
@@ -20128,7 +20804,7 @@ ${MAP_HELP}`,
   docs: {
     summary: "Measure, stub and gate the doc comments of TypeScript, JavaScript and Python source.",
     help: DOCS_HELP,
-    run: run5
+    run: run6
   },
   chunk: {
     summary: "Split a large file into chunks, merge the chunks back, or show changed chunks.",
@@ -20138,17 +20814,22 @@ ${MAP_HELP}`,
   compress: {
     summary: "Write a compact copy of a file for a model context, or restore it.",
     help: HELP,
-    run: run4
+    run: run5
   },
   query: {
     summary: "Answer structural questions from the depgraph reports, or write derived reports.",
     help: QUERY_HELP,
-    run: run6
+    run: run7
   },
   ste: {
     summary: "Check Markdown or docstring prose against Simplified Technical English.",
     help: STE_HELP,
-    run: run7
+    run: run8
+  },
+  clones: {
+    summary: "List the functions of a repository and find the functions that copy each other.",
+    help: CLONES_HELP,
+    run: run4
   }
 };
 var SUBCOMMANDS = Object.keys(REGISTRY);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@danielsimonjr/repo-tools",
-  "version": "2.2.7",
+  "version": "2.3.0",
   "description": "One CLI for repository analysis: dependency graphs creation and query with per-export API facts, file chunking, and context compression. TypeScript on Bun, compiled to a single executable.",
   "type": "module",
   "license": "MIT",