jscpd-rs 0.1.0

package/docs/upstream-issue-drafts.md

@@ -0,0 +1,393 @@
+# Upstream Issue Drafts
+
+These drafts are prepared from `docs/upstream-bugs.md` for filing issues in
+`kucherenko/jscpd`. Re-verify each issue against the current upstream default
+branch before posting. Drafts that use public repositories include pinned
+commits; the other drafts use upstream fixtures or minimal inline
+reproductions.
+
+Verification snapshot: on 2026-05-31, upstream remote `HEAD` resolved to
+`refs/heads/master` at `50290cf`; no `refs/heads/main` was advertised. The
+local upstream submodule is clean at that SHA and was used as the current
+upstream checkout for quick verification. Drafts 1, 2, 3, 5, 6, and 7
+reproduced. Draft 4 is covered by the public benchmark compatibility gate
+recorded in `docs/compat-baseline.md`; the full GitHub Actions release-candidate
+gate also passed on 2026-05-31.
+
+Quick verification notes:
+
+- Draft 1 reproduced the tokenizer output
+  `comment "//${b}${c?\":\"+c:\"\"}`}function k(){return 1}"`.
+- Draft 2 reproduced `git blame` exit 128 from the parent repository on
+  `jscpd/fixtures/javascript/file_4.js`.
+- Draft 3 reproduced the reported fixture ranges
+  `pug file1.pug:1-274 <> file2.pug:1-266`,
+  `haml file1.haml:1-26 <> file2.haml:1-26`, and
+  `aspnet file1.aspx:18-36 <> file2.aspx:18-43`.
+- Draft 5 reproduced
+  `TypeError: Cannot read properties of undefined (reading 'range')`.
+- Draft 6 was rechecked by source search: `cache` is defaulted/copied but has
+  no CLI flag or runtime read, `listeners` is only normalized, and
+  `tokensToSkip` appears only in the options interface.
+- Draft 7 reproduced the documented bare-flag behavior for `--threshold`,
+  `--exitCode`, `--formats-exts javascript`, and `--ignore`.
+
+## Draft 1: JavaScript tokenizer treats `//` inside a template literal as a comment
+
+Suggested title:
+
+```text
+JavaScript tokenizer treats `//` inside a template literal as a line comment
+```
+
+Suggested labels: `bug`, `javascript`, `tokenizer`.
+
+Summary:
+
+The Prism-backed JavaScript tokenizer can treat `//` inside a JavaScript
+template literal as a line comment. The resulting comment token consumes the
+rest of the physical line, including ordinary JavaScript after the template
+literal. This can hide valid duplicated code behind one large comment token in
+minified or bundled JavaScript.
+
+Minimal tokenizer repro:
+
+Run this from the upstream `jscpd` repository after building packages:
+
+```bash
+node - <<'NODE'
+const { Tokenizer } = require('./packages/tokenizer/dist/index.js');
+const { mild } = require('./packages/core/dist/index.js');
+
+const code =
+  'function h(a){let j="//";return`${j}`}function j(){let{protocol:a,hostname:b,port:c}=window.location;return`${a}//${b}${c?":"+c:""}`}function k(){return 1}\n';
+
+const tokens = new Tokenizer()
+  .generateMaps('repro.js', code, 'javascript', { minTokens: 1, mode: mild })[0]
+  .tokens;
+
+for (const token of tokens.filter((t) => t.type === 'comment' || t.value.includes('//'))) {
+  console.log(`${token.type} ${JSON.stringify(token.value)} line=${token.loc.start.line} col=${token.loc.start.column}`);
+}
+NODE
+```
+
+Observed symptoms:
+
+- The first `//` string literal is emitted correctly as a `string` token.
+- The `//` inside the template literal is emitted as a `comment` token:
+
+```text
+string "\"//\"" line=1 col=21
+comment "//${b}${c?\":\"+c:\"\"}`}function k(){return 1}" line=1 col=113
+```
+
+Expected behavior:
+
+`//` inside template literal text should remain a template string segment and
+should not comment out the rest of the generated line.
+
+Impact:
+
+Generated/minified SSR bundles can lose duplicate coverage because ordinary
+module code after the template literal is hidden inside one incorrect comment
+token.
+
+## Draft 2: `--blame` fails for paths inside a nested Git repository
+
+Suggested title:
+
+```text
+--blame fails when scanned files are inside a nested Git repository or submodule
+```
+
+Suggested labels: `bug`, `blame`, `git`.
+
+Summary:
+
+When `jscpd` is launched from a parent repository and the scan target is inside a
+nested Git repository or submodule, `--blame` invokes `git blame` from the
+parent working directory with the nested file path. The parent repository does
+not track that nested file as a normal file, so `git blame` exits with 128 and
+the whole detection run fails.
+
+Repro:
+
+```bash
+node jscpd/apps/jscpd/bin/jscpd jscpd/fixtures/javascript \
+  --format javascript \
+  --reporters json \
+  --output /tmp/jscpd-upstream-blame \
+  --silent \
+  --noTips \
+  --blame \
+  --min-tokens 20 \
+  --min-lines 3 \
+  --max-size 1mb \
+  --exitCode 0
+```
+
+Observed first error on the current submodule:
+
+```text
+Error: Command failed with exit code 128: /usr/bin/git blame -w jscpd/fixtures/javascript/file_2.mjs
+fatal: no such path 'jscpd/fixtures/javascript/file_2.mjs' in HEAD
+```
+
+Expected behavior:
+
+Blame should run from the scanned file's own repository/worktree, or blame
+should fail per file without aborting the whole detection run.
+
+## Draft 3: Clone ranges can extend through non-matching embedded/template blocks
+
+Suggested title:
+
+```text
+Clone ranges can extend through non-matching embedded or template blocks
+```
+
+Suggested labels: `bug`, `detector`, `reporting`.
+
+Summary:
+
+Some reported clone ranges include neighboring source that does not match the
+paired fragment. The issue is visible in fixture formats that contain large
+block tokens or embedded markup.
+
+Fixture repros:
+
+```bash
+FORMAT=pug MIN_TOKENS=20 MIN_LINES=3 MAX_SIZE=1mb KEEP=1 \
+  scripts/compat.sh jscpd/fixtures/pug
+
+FORMAT=haml MIN_TOKENS=20 MIN_LINES=3 MAX_SIZE=1mb KEEP=1 \
+  scripts/compat.sh jscpd/fixtures/haml
+
+FORMAT=aspnet MIN_TOKENS=20 MIN_LINES=3 MAX_SIZE=1mb KEEP=1 \
+  scripts/compat.sh jscpd/fixtures/htmlembedded
+```
+
+Observed examples:
+
+- Pug reports `file1.pug:1-274` against `file2.pug:1-266`, including a
+  `style.` plain-text block whose CSS values differ.
+- HAML reports `file1.haml:1-26` against `file2.haml:1-26`, including
+  different silent-comment blocks.
+- ASP.NET reports `file1.aspx:18-36` against `file2.aspx:18-43`, including an
+  inserted email form group that is not present in the paired file.
+
+Expected behavior:
+
+Clone ranges should stop at the last matching token run, split around inserted
+content, or document that specific ignored block types may intentionally extend
+reported source ranges through non-matching text.
+
+## Draft 4: Public benchmark clone ranges are sometimes overextended or reversed
+
+Suggested title:
+
+```text
+Some clone ranges are overextended or reversed on large public repositories
+```
+
+Suggested labels: `bug`, `detector`, `reporting`.
+
+Summary:
+
+Large-repository runs show reported fragments whose line ranges extend across
+neighboring non-matching tests, table entries, or declarations. Several ranges
+also have reversed start/end ordering.
+
+Repro shapes:
+
+```bash
+git clone https://github.com/facebook/react.git /tmp/jscpd-react
+git -C /tmp/jscpd-react checkout f0dfee3
+node apps/jscpd/bin/jscpd /tmp/jscpd-react \
+  --format javascript \
+  --reporters json \
+  --output /tmp/jscpd-react-report \
+  --silent \
+  --noTips \
+  --min-tokens 50 \
+  --min-lines 5 \
+  --max-size 1mb \
+  --exitCode 0
+
+git clone https://github.com/vercel/next.js.git /tmp/jscpd-next
+git -C /tmp/jscpd-next checkout 2bbb67b9
+node apps/jscpd/bin/jscpd /tmp/jscpd-next \
+  --format typescript \
+  --reporters json \
+  --output /tmp/jscpd-next-report \
+  --silent \
+  --noTips \
+  --min-tokens 50 \
+  --min-lines 5 \
+  --max-size 1mb \
+  --exitCode 0
+
+git clone https://github.com/prometheus/prometheus.git /tmp/jscpd-prometheus
+git -C /tmp/jscpd-prometheus checkout a0524ee
+node apps/jscpd/bin/jscpd /tmp/jscpd-prometheus \
+  --format go \
+  --reporters json \
+  --output /tmp/jscpd-prometheus-report \
+  --silent \
+  --noTips \
+  --min-tokens 50 \
+  --min-lines 5 \
+  --max-size 1mb \
+  --exitCode 0
+```
+
+Observed examples:
+
+- React reports ranges such as `ReactDOMFizzServerNode.js:229-179` and clone
+  fragments that continue into neighboring test bodies.
+- Next.js reports broad ranges around inline snapshots and multi-test files,
+  including reversed endpoints like `459-314` and `892-745`.
+- Prometheus reports broad table-test ranges where one early case is stretched
+  through unrelated later cases.
+
+Expected behavior:
+
+Reported fragments should keep start/end ordering stable and should stop at the
+actual matching token run instead of stretching one match across unrelated
+neighboring source.
+
+## Draft 5: Config string `minTokens` can corrupt token-window indexing
+
+Suggested title:
+
+```text
+String minTokens values from config can corrupt detector token windows
+```
+
+Suggested labels: `bug`, `config`.
+
+Summary:
+
+Runtime config loaded from `.jscpd.json` or `package.json#jscpd` is merged
+without the CLI numeric parser. Numeric-looking strings for some fields continue
+through JavaScript coercion, but `minTokens` is later used in token-window
+indexing with `+` before numeric subtraction. A value such as `"5"` can produce
+string-concatenated indices and eventually an undefined token frame.
+
+Minimal repro:
+
+Run this from the upstream `jscpd` repository root:
+
+```bash
+tmp=$(mktemp -d)
+JSCPD_REPO=$(pwd)
+mkdir -p "$tmp/src"
+printf 'function alpha(){\n  return [1,2,3,4,5,6,7,8,9,10].map((x)=>x+1).join(",");\n}\nfunction beta(){\n  return alpha();\n}\n' > "$tmp/src/a.js"
+cp "$tmp/src/a.js" "$tmp/src/b.js"
+printf '{"path":["src"],"format":["javascript"],"reporters":["json"],"silent":true,"minTokens":"5","minLines":1,"maxSize":"1mb","exitCode":0}\n' > "$tmp/.jscpd.json"
+
+cd "$tmp"
+node "$JSCPD_REPO/apps/jscpd/bin/jscpd" --config .jscpd.json --noTips
+```
+
+Observed first error:
+
+```text
+TypeError: Cannot read properties of undefined (reading 'range')
+    at _RabinKarp.enlargeClone (.../packages/core/dist/index.js:100:49)
+```
+
+Expected behavior:
+
+Config numeric fields should be parsed and validated before detector execution,
+or invalid string values should fail with a clear configuration error.
+
+## Draft 6: Public option fields are exposed but unused at runtime
+
+Suggested title:
+
+```text
+Some public option fields are exposed but unused at runtime
+```
+
+Suggested labels: `bug`, `options`, `documentation`.
+
+Summary:
+
+The option surface exposes fields that look user-facing but are not consumed by
+the runtime:
+
+- `cache` is defined and defaulted, but there is no `--cache` CLI option and no
+  detector/tokenizer read of `options.cache`.
+- `listeners` is normalized to an array, but runtime subscriptions come only
+  from built-in verbose/progress handling.
+- `tokensToSkip` appears in the options interface but is not consumed by
+  tokenizer or detector code.
+
+Expected behavior:
+
+These fields should either be documented as reserved/no-op, removed from the
+public option surface, or wired to actual runtime behavior.
+
+## Draft 7: Optional CLI values can produce TypeErrors or accidental behavior
+
+Suggested title:
+
+```text
+Bare optional CLI flags can produce TypeErrors or accidental behavior
+```
+
+Suggested labels: `bug`, `cli`.
+
+Summary:
+
+Several Commander options accept optional values. When passed without a value,
+Commander supplies boolean `true`, and later runtime code either crashes with a
+TypeError or continues with surprising semantics.
+
+Repro examples:
+
+```bash
+node jscpd/apps/jscpd/bin/jscpd jscpd/fixtures/javascript \
+  --threshold \
+  --silent \
+  --noTips \
+  --min-tokens 20 \
+  --min-lines 3 \
+  --max-size 1mb
+
+node jscpd/apps/jscpd/bin/jscpd jscpd/fixtures/javascript \
+  --exitCode \
+  --silent \
+  --noTips \
+  --min-tokens 20 \
+  --min-lines 3 \
+  --max-size 1mb
+
+node jscpd/apps/jscpd/bin/jscpd jscpd/fixtures/custom \
+  --formats-exts javascript \
+  --silent \
+  --noTips \
+  --min-tokens 20 \
+  --min-lines 3 \
+  --max-size 1mb
+```
+
+Observed behavior:
+
+- Bare `--threshold` becomes `Number(true)`, so the threshold is treated as
+  `1%`.
+- Bare `--exitCode` stores boolean `true`, which Node rejects as
+  `process.exitCode` when clones are found.
+- Bare string flags such as `--ignore`, `--reporters`, `--mode`, `--format`,
+  `--formats-exts`, and `--formats-names` later crash because boolean `true`
+  is used as a string.
+- Malformed mapping values like `--formats-exts javascript` crash during option
+  conversion because the parser assumes each entry contains `:`.
+
+Expected behavior:
+
+Flags that require values should declare required values, validate the bare flag
+case explicitly, or normalize bare flags to documented defaults before option
+conversion.

package/docs/user-guide.md

@@ -0,0 +1,309 @@
+# jscpd-rs User Guide
+
+`jscpd-rs` is a native Rust implementation of the common `jscpd` workflows:
+scan source trees, detect copy-paste fragments, write reports, fail CI on
+thresholds, and serve snippet checks over HTTP/MCP.
+
+This guide is intentionally user-facing. Release policy, benchmark evidence,
+and compatibility details live in the release docs linked from the README.
+
+## Installation
+
+Install the Rust binaries from crates.io after publication:
+
+```bash
+cargo install jscpd-rs --locked
+```
+
+Install through npm/npx when Node is already part of the workflow:
+
+```bash
+npm install -g jscpd-rs
+npx jscpd-rs --version
+```
+
+The first npm package builds the native binaries from source during
+`postinstall`, so a Rust toolchain must be available. Prebuilt npm platform
+packages are planned after the first release.
+
+Install from a checkout:
+
+```bash
+git clone https://github.com/vv-bogdanov/jscpd-rs.git
+cd jscpd-rs
+cargo install --path . --bins --locked
+```
+
+## CLI Usage
+
+Run a scan:
+
+```bash
+jscpd .
+```
+
+Scan selected paths and formats:
+
+```bash
+jscpd --format javascript,typescript apps packages
+```
+
+Tune clone size:
+
+```bash
+jscpd --min-lines 5 --min-tokens 50 src
+```
+
+Write machine-readable reports:
+
+```bash
+jscpd --reporters console,json,sarif --output report src
+```
+
+Fail CI when the duplicated-line percentage is too high:
+
+```bash
+jscpd --threshold 5 --exitCode 1 .
+```
+
+Inspect the upstream-compatible help and format registry:
+
+```bash
+jscpd --help
+jscpd --list
+```
+
+## Common Options
+
+| Option | Purpose | Default |
+| --- | --- | --- |
+| `--min-lines`, `-l` | minimum clone size in source lines | `5` |
+| `--min-tokens`, `-k` | minimum clone size in tokens | `50` |
+| `--max-lines`, `-x` | skip sources with more lines | `1000` |
+| `--max-size`, `-z` | skip sources above a byte size such as `100kb` or `1mb` | `100kb` |
+| `--threshold`, `-t` | fail when duplicated percentage is above the threshold | unset |
+| `--exitCode` | exit code to use when duplication is detected | upstream default |
+| `--format`, `-f` | comma-separated format allowlist | all formats |
+| `--pattern`, `-p` | glob pattern used during discovery | `**/*` |
+| `--ignore`, `-i` | comma-separated ignore globs | unset |
+| `--ignore-pattern` | skip code blocks matching regular expressions | unset |
+| `--mode`, `-m` | detection mode: `strict`, `mild`, or `weak` | `mild` |
+| `--skipComments` | shorthand for `--mode weak` | `false` |
+| `--reporters`, `-r` | comma-separated reporter list | `console` |
+| `--output`, `-o` | report output directory | `./report` |
+| `--blame`, `-b` | add Git author/date information to duplicate fragments | `false` |
+| `--absolute`, `-a` | write absolute source paths in reports | `false` |
+| `--noSymlinks`, `-n` | do not follow symlinks during discovery | `false` |
+| `--gitignore` / `--no-gitignore` | enable or disable Git ignore handling | enabled |
+| `--skipLocal` | report only cross-folder duplication | `false` |
+| `--noTips` | suppress post-run tips and promotional output | `false` |
+
+The CLI accepts multiple `--ignore` flags and comma-separated ignore lists:
+
+```bash
+jscpd --ignore "target/**" --ignore "node_modules/**,dist/**" .
+```
+
+## Configuration
+
+`jscpd-rs` reads `.jscpd.json` by default when present. It also reads a `jscpd`
+section from `package.json`, matching the common upstream workflow.
+
+Example `.jscpd.json`:
+
+```json
+{
+  "path": ["src", "packages"],
+  "format": ["javascript", "typescript", "rust"],
+  "minLines": 5,
+  "minTokens": 50,
+  "threshold": 5,
+  "reporters": ["console", "json", "html", "sarif"],
+  "output": "report",
+  "ignore": ["target/**", "node_modules/**", "dist/**"],
+  "gitignore": true,
+  "noTips": true
+}
+```
+
+Example `package.json` section:
+
+```json
+{
+  "jscpd": {
+    "path": ["src"],
+    "reporters": ["console", "json"],
+    "threshold": 5,
+    "ignore": ["coverage/**", "**/*.snap"]
+  }
+}
+```
+
+CLI arguments are applied after config loading, so command-line values can
+override project defaults.
+
+## Reporters
+
+Built-in native reporters:
+
+| Reporter | Output |
+| --- | --- |
+| `console` | compact terminal summary and clone list |
+| `consoleFull` | terminal output with duplicated fragments |
+| `ai` | compact, token-efficient clone list for agent prompts |
+| `json` | `jscpd-report.json` |
+| `xml` | `jscpd-report.xml` |
+| `csv` | `jscpd-report.csv` |
+| `markdown` | `jscpd-report.md` |
+| `html` | self-contained HTML report under `html/` |
+| `sarif` | `jscpd-sarif.json` for code-scanning pipelines |
+| `xcode` | diagnostics formatted for Xcode-style tooling |
+| `threshold` | threshold-only error output |
+| `badge` | `jscpd-badge.svg` |
+| `silent` | suppress report output |
+
+Dynamic npm reporters are intentionally not loaded in the first release. Unknown
+external reporter names keep upstream-style warnings and continue where upstream
+continues.
+
+## Server And MCP
+
+Start the native server:
+
+```bash
+jscpd-server . --host 127.0.0.1 --port 3000
+```
+
+Health and statistics:
+
+```bash
+curl http://127.0.0.1:3000/api/health
+curl http://127.0.0.1:3000/api/stats
+```
+
+Check a snippet against the scanned project:
+
+```bash
+curl -X POST http://127.0.0.1:3000/api/check \
+  -H "Content-Type: application/json" \
+  -d '{"format":"javascript","code":"function sum(a,b){return a+b;}"}'
+```
+
+Refresh the project token maps after changing files:
+
+```bash
+curl -X POST http://127.0.0.1:3000/api/recheck
+```
+
+MCP endpoint:
+
+```text
+http://127.0.0.1:3000/mcp
+```
+
+Example MCP client entry:
+
+```json
+{
+  "mcpServers": {
+    "jscpd": {
+      "type": "streamable-http",
+      "url": "http://127.0.0.1:3000/mcp"
+    }
+  }
+}
+```
+
+The server exposes core duplication tools and a statistics resource over native
+JSON-RPC HTTP.
+
+## Rust API
+
+Path-based detection:
+
+```rust
+use std::path::PathBuf;
+
+fn main() -> anyhow::Result<()> {
+    let mut options = jscpd_rs::get_default_options();
+    options.paths = vec![PathBuf::from("src")];
+    options.reporters.clear();
+    options.silent = true;
+
+    let result = jscpd_rs::detect_clones_and_statistics(&options)?;
+    println!("{} clones", result.clones.len());
+    Ok(())
+}
+```
+
+In-memory detection:
+
+```rust
+let mut options = jscpd_rs::get_default_options();
+options.reporters.clear();
+options.min_lines = 2;
+options.min_tokens = 5;
+
+let files = vec![
+    jscpd_rs::SourceFile {
+        source_id: "a.js".to_string(),
+        format: "javascript".to_string(),
+        content: "const a = 1;\nconst b = 2;\nconst c = a + b;\n".to_string(),
+    },
+    jscpd_rs::SourceFile {
+        source_id: "b.js".to_string(),
+        format: "javascript".to_string(),
+        content: "const a = 1;\nconst b = 2;\nconst c = a + b;\n".to_string(),
+    },
+];
+
+let result = jscpd_rs::detect_source_files(files, &options);
+assert!(!result.clones.is_empty());
+```
+
+Useful entry points:
+
+- `get_default_options`
+- `get_options_from_args`
+- `detect_clones`
+- `detect_clones_and_statistics`
+- `detect_source_files`
+- `get_supported_formats`
+- `get_format_by_file`
+- `Tokenizer`
+- `Detector`
+- `MemoryStore`
+
+## AI Skills
+
+Install the tool-reference and dry-refactoring skills:
+
+```bash
+npx skills add vv-bogdanov/jscpd-rs --skill jscpd
+npx skills add vv-bogdanov/jscpd-rs --skill dry-refactoring
+```
+
+The `ai` reporter is designed for short clone summaries that can be passed to
+coding agents without including duplicated source fragments by default:
+
+```bash
+jscpd --reporters ai src
+```
+
+## Compatibility Notes
+
+The first release target is practical, coverage-first upstream compatibility.
+For the same inputs and options, `jscpd-rs` must not miss duplicated source
+lines reported by upstream `jscpd`. Extra Rust findings are allowed while
+compatibility converges and remain visible in compatibility reports.
+
+Intentional first-release limits:
+
+- dynamic npm reporters, stores, listeners, and plugins are not loaded;
+- exact token totals and pair ordering may differ from upstream while
+  duplicated upstream lines remain covered;
+- HTML output is self-contained and practically compatible, not pixel-perfect;
+- the Rust crate exposes a native Rust API, not the upstream JavaScript API.
+
+See `docs/release-decisions.md` and `docs/compat-baseline.md` for the current
+release policy and compatibility evidence.