jscpd-rs 0.1.0

package/Cargo.toml

@@ -0,0 +1,54 @@
+[package]
+name = "jscpd-rs"
+version = "0.1.0"
+edition = "2024"
+rust-version = "1.93"
+license = "MIT"
+description = "Fast Rust clone of jscpd"
+readme = "README.md"
+repository = "https://github.com/vv-bogdanov/jscpd-rs"
+documentation = "https://docs.rs/jscpd-rs"
+keywords = ["jscpd", "copy-paste", "duplication", "clone-detection", "cli"]
+categories = ["command-line-utilities", "development-tools"]
+include = [
+    "/CHANGELOG.md",
+    "/Cargo.lock",
+    "/Cargo.toml",
+    "/LICENSE",
+    "/README.md",
+    "/docs/**",
+    "/examples/**",
+    "/skills/**",
+    "/src/**",
+]
+
+[[bin]]
+name = "jscpd"
+path = "src/main.rs"
+
+[[bin]]
+name = "jscpd-server"
+path = "src/bin/jscpd-server.rs"
+
+[dependencies]
+anyhow = "1.0"
+axum = "0.8"
+clap = { version = "4.5", features = ["derive"] }
+form_urlencoded = "1.2"
+getrandom = "0.2"
+globset = "0.4"
+ignore = "0.4"
+oxc_allocator = "0.133.0"
+oxc_parser = { version = "0.133.0", features = ["benchmarking"] }
+oxc_span = "0.133.0"
+rayon = "1.11"
+regex = "1.11"
+rustc-hash = "2.1"
+serde = { version = "1.0", features = ["derive"] }
+serde_json = "1.0"
+time = { version = "0.3", features = ["formatting"] }
+tokio = { version = "1.48", features = ["macros", "net", "rt-multi-thread"] }
+xxhash-rust = { version = "0.8", features = ["xxh3"] }
+
+[dev-dependencies]
+tower = { version = "0.5", features = ["util"] }

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026  Viacheslav V Bogdanov
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,372 @@
+# jscpd-rs
+
+[![release-gate](https://github.com/vv-bogdanov/jscpd-rs/actions/workflows/release-gate.yml/badge.svg)](https://github.com/vv-bogdanov/jscpd-rs/actions/workflows/release-gate.yml)
+[![crates.io](https://img.shields.io/crates/v/jscpd-rs.svg?style=flat-square)](https://crates.io/crates/jscpd-rs)
+[![docs.rs](https://img.shields.io/docsrs/jscpd-rs?style=flat-square)](https://docs.rs/jscpd-rs)
+[![npm](https://img.shields.io/npm/v/jscpd-rs.svg?style=flat-square)](https://www.npmjs.com/package/jscpd-rs)
+[![license](https://img.shields.io/github/license/vv-bogdanov/jscpd-rs.svg?style=flat-square)](LICENSE)
+[![rust](https://img.shields.io/badge/rust-1.93%2B-dea584?style=flat-square)](https://www.rust-lang.org/)
+
+Fast native Rust clone of [`jscpd`](https://github.com/kucherenko/jscpd)
+for copy-paste and duplicate-code detection in local development and CI/CD.
+It scans a codebase, finds duplicated source fragments across files, writes
+reports for humans and automation, and can fail a build when duplication
+crosses a configured threshold.
+
+`jscpd-rs` keeps the upstream command shape, configuration formats, reports,
+exit-code workflows, and server workflow, while moving the hot path to native
+Rust. The practical goal is simple: keep duplication checks always-on without
+spending unnecessary CI minutes, developer waiting time, cloud compute budget,
+or electricity on repeated quality gates.
+
+Recorded public release-candidate benchmarks are currently 50x+ faster than
+upstream on the covered repositories. The compatibility gate is coverage-first:
+on the same inputs and options, `jscpd-rs` must not miss duplicated source lines
+reported by upstream `jscpd`.
+
+## Install
+
+Cargo:
+
+```bash
+cargo install jscpd-rs --locked
+jscpd --version
+```
+
+npm/npx:
+
+```bash
+npm install -g jscpd-rs
+jscpd --version
+
+npx jscpd-rs --version
+npx jscpd-rs .
+```
+
+The first npm package is a source-build package: install/postinstall compiles
+the native Rust binaries with Cargo. A Rust toolchain must be available on the
+installing machine. Prebuilt platform packages are a planned publication
+improvement.
+
+From this repository:
+
+```bash
+git clone https://github.com/vv-bogdanov/jscpd-rs.git
+cd jscpd-rs
+cargo install --path . --bins --locked
+```
+
+Build without installing:
+
+```bash
+cargo build --release --bin jscpd
+cargo build --release --bin jscpd-server
+```
+
+## Quick Start
+
+Scan a project:
+
+```bash
+jscpd .
+```
+
+Tune the detection threshold:
+
+```bash
+jscpd --min-lines 5 --min-tokens 50 src
+```
+
+Generate reports:
+
+```bash
+jscpd --reporters console,json,html --output report src
+```
+
+Fail CI when duplication is above a threshold:
+
+```bash
+jscpd --threshold 5 --exitCode 1 src
+```
+
+Use the upstream-compatible command help and format list:
+
+```bash
+jscpd --help
+jscpd --list
+```
+
+Start the native REST/MCP server:
+
+```bash
+jscpd-server . --host 127.0.0.1 --port 3000
+curl http://127.0.0.1:3000/api/health
+```
+
+The current server exposes `/`, `/api/health`, `/api/stats`, `/api/check`,
+`/api/recheck`, and `/mcp`. Snippet checks reuse project token maps refreshed
+by `/api/recheck`.
+
+For full CLI, configuration, reporter, server, MCP, and Rust API examples, see
+[docs/user-guide.md](docs/user-guide.md).
+If you already use upstream `jscpd`, see
+[docs/migrating-from-jscpd.md](docs/migrating-from-jscpd.md).
+
+## GitHub Actions
+
+Install from crates.io after publication:
+
+```yaml
+jobs:
+  duplication:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v5
+      - uses: dtolnay/rust-toolchain@stable
+      - run: cargo install jscpd-rs --locked
+      - run: jscpd src --reporters console,json --threshold 5 --exitCode 1
+```
+
+Use npm/npx when a Node-based CI environment is already available:
+
+```yaml
+jobs:
+  duplication:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v5
+      - uses: dtolnay/rust-toolchain@stable
+      - uses: actions/setup-node@v5
+        with:
+          node-version: 22
+      - run: npx jscpd-rs src --reporters console,json --threshold 5 --exitCode 1
+```
+
+Install from a checked-out source tree:
+
+```yaml
+jobs:
+  duplication:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v5
+      - uses: dtolnay/rust-toolchain@stable
+      - run: cargo install --path . --bins --locked
+      - run: jscpd src --reporters console,json --threshold 5 --exitCode 1
+```
+
+## AI Refactoring Skills
+
+Install the project skills for duplication detection and guided dry
+refactoring:
+
+```bash
+npx skills add vv-bogdanov/jscpd-rs --skill jscpd
+npx skills add vv-bogdanov/jscpd-rs --skill dry-refactoring
+```
+
+## Why jscpd-rs
+
+- **Fast CI/CD gates:** duplicate detection should be cheap enough to run on
+  every pull request.
+- **Lower operating cost:** shorter scans reduce paid compute minutes and
+  repeated developer wait time.
+- **Native detector path:** the detector does not embed or spawn JavaScript for
+  core behavior.
+- **Practical compatibility:** the CLI, config, reporter, server, and exit-code
+  workflows are designed to map onto common upstream `jscpd` usage.
+- **Small project-specific core:** use battle-tested Rust crates for CLI
+  parsing, config formats, ignore handling, serialization, token processing,
+  concurrency, and reporting wherever that keeps the implementation simpler.
+
+## What Works Today
+
+This is pre-release software. The first release target is a coverage-first
+compatible CLI replacement for common `jscpd` workflows:
+
+- `jscpd` and `jscpd-server` binaries with upstream-compatible command names;
+- CLI and config option surface covered by compatibility scripts;
+- native built-in reporters: `ai`, `console`, `consoleFull`, `csv`, `html`,
+  `json`, `markdown`, `silent`, `sarif`, `threshold`, `xcode`, `xml`, and
+  `badge`;
+- upstream-synchronized format registry with native JS/TS/JSX/TSX tokenization
+  and generic native tokenization for long-tail formats;
+- native Git blame support;
+- native REST/MCP server workflow;
+- Rust library API for running detection from paths or prepared in-memory
+  sources.
+
+Dynamic npm reporters, stores, listeners, and plugins are intentionally out of
+scope for the first release. Unknown external reporters/stores keep
+upstream-style warnings and continue where upstream continues.
+
+## Compatibility Contract
+
+The release gate is coverage-first. For the same inputs and options,
+`jscpd-rs` must not miss duplicated lines reported by upstream `jscpd`. Extra
+Rust duplicates are allowed while compatibility converges, but compatibility
+reports keep them visible as `extra` findings.
+
+Exact pair ordering and token totals are quality metrics rather than the
+default blocking gate. This matters for multi-way clones: different pair
+selection can still cover the same duplicated source lines.
+
+The upstream repository is checked out as `jscpd/` and treated as the
+executable specification. Compatibility scripts run both implementations and
+compare their reports.
+
+## Performance
+
+Latest recorded public benchmark baseline:
+
+| Repo | Format | Rust avg | Upstream avg | Speedup |
+| --- | --- | ---: | ---: | ---: |
+| React | JavaScript | 0.199097s | 10.079214s | 50.62x |
+| Next.js | TypeScript | 0.262433s | 14.715736s | 56.07x |
+| Prometheus | Go | 0.085239s | 4.642435s | 54.46x |
+
+Reproduce the public benchmark and coverage suite:
+
+```bash
+PUBLIC=1 PUBLIC_RUNS=3 scripts/release-gate.sh
+```
+
+The release-candidate workflow reruns the public suite before publication so
+README numbers stay tied to a concrete commit and gate output.
+
+## Library API
+
+The crate exposes the detector core for native integrations:
+
+```rust
+let options = jscpd_rs::get_default_options();
+let result = jscpd_rs::detect_clones_and_statistic(&options)?;
+let clones = result.clones;
+
+let clones = jscpd_rs::jscpd(["jscpd", "src", "--silent", "--noTips"])?;
+```
+
+`detect_clones_and_statistics` is also available as the idiomatic Rust spelling.
+`jscpd` and `jscpd_with_exit_callback` provide a native embeddable argv runner
+similar to upstream `jscpd(argv, exitCallback?)`. `get_options_from_args` parses
+upstream-style argv into normalized `Options` for native integrations.
+
+`Tokenizer` provides a native generate-maps entrypoint over the same tokenizer
+used by detection. `Detector`, `Statistic`, and `MemoryStore` expose native
+counterparts for the main upstream core classes without loading JavaScript.
+`detect_source_files` accepts in-memory `SourceFile` values, which is the
+foundation for the upstream-style snippet/server workflow. Format helpers are
+available through `get_supported_formats`, `get_format_by_file`, and
+`get_format_by_file_with_mappings`.
+
+## Architecture
+
+The implementation keeps the hot path small and native:
+
+```text
+paths/config
+  -> ignore-aware discovery
+  -> native token maps
+  -> parallel duplicate detection
+  -> reporters / exit codes / server API
+```
+
+Core crates and libraries:
+
+- `clap`, `serde`, and config parsers for the CLI/config surface;
+- `ignore`, `globset`, and Git ignore handling for file discovery;
+- Oxc-backed JS/TS/JSX/TSX token processing for the highest-volume languages;
+- native generic tokenizers for long-tail formats and embedded code blocks;
+- `rayon` and Rust data structures for parallel discovery/detection work;
+- native reporters for JSON, SARIF, XML, CSV, Markdown, HTML, console, badge,
+  Xcode, threshold, silent, and AI refactoring output.
+
+## Known First-Release Deviations
+
+The first release is native-only and coverage-first. These differences from the
+JavaScript package are intentional unless a real workflow proves otherwise:
+
+- dynamic npm reporters, stores, listeners, and plugins are not loaded;
+- token totals and exact clone pair ordering may differ from Prism-based
+  upstream reports while duplicated upstream lines remain covered;
+- HTML reports are self-contained and practically compatible, not pixel-perfect;
+- the Rust crate exposes a native Rust API, not the upstream JavaScript package
+  API.
+
+## Development
+
+The upstream repository is checked out as the `jscpd/` git submodule and is the
+executable specification for behavior.
+
+```bash
+git submodule update --init --recursive
+cargo test
+```
+
+Useful focused checks:
+
+```bash
+scripts/compat-cli.sh
+scripts/compat-config.sh
+scripts/compat-reporters.sh
+STRICT=coverage scripts/compat-matrix.sh
+```
+
+Known upstream bug candidates and intentional compatibility exceptions are
+tracked in [docs/upstream-bugs.md](docs/upstream-bugs.md). GitHub-ready issue
+drafts are prepared in
+[docs/upstream-issue-drafts.md](docs/upstream-issue-drafts.md).
+
+## Release Gates
+
+Fast local gate:
+
+```bash
+scripts/release-gate.sh
+```
+
+Package/install gate:
+
+```bash
+scripts/package-check.sh
+```
+
+npm package/npx gate:
+
+```bash
+scripts/npm-package-check.sh
+```
+
+Full compatibility matrix:
+
+```bash
+FULL=1 scripts/release-gate.sh
+```
+
+Public benchmark and coverage gate:
+
+```bash
+PUBLIC=1 PUBLIC_RUNS=3 scripts/release-gate.sh
+```
+
+Release candidate gate:
+
+```bash
+scripts/release-candidate.sh
+```
+
+The GitHub Actions workflow runs the fast gate on pushes and pull requests.
+Manual workflow runs can enable the full compatibility matrix and public
+benchmark suite before a release, or set `release_candidate=true` to run the
+full release-candidate gate in CI.
+
+See [docs/compat-baseline.md](docs/compat-baseline.md) for the current gate
+baseline, [docs/release-readiness.md](docs/release-readiness.md) for component
+status, [docs/release-checklist.md](docs/release-checklist.md) for the
+publication checklist, [CHANGELOG.md](CHANGELOG.md) for release notes, and
+[docs/release-decisions.md](docs/release-decisions.md) for approved
+first-release compatibility decisions.
+
+## License
+
+MIT

package/docs/api-parity.md

@@ -0,0 +1,49 @@
+# API Parity
+
+Last updated: 2026-05-31.
+
+This document tracks the upstream JavaScript API surface against the native Rust
+API. The first release remains native-only: the crate does not embed Node.js,
+does not call JavaScript at runtime, and does not ship a JavaScript package
+wrapper unless that is chosen as a separate release target.
+
+## App-Level API
+
+| Upstream API | Rust status | Notes |
+| --- | --- | --- |
+| `detectClones(opts, store?, statisticProvider?)` | covered | Use `detect_clones(&Options)` for path-based detection. Native `MemoryStore` and `Statistic` helpers are exposed, but custom provider injection is not part of the first-release API. |
+| `detectClonesAndStatistic(opts, store?)` | covered | Use `detect_clones_and_statistic(&Options)`. `detect_clones_and_statistics(&Options)` remains the idiomatic Rust spelling. |
+| `jscpd(argv, exitCallback?)` | covered natively | Use the `jscpd` binary for process behavior, `jscpd(args)` for embeddable argv execution, or `jscpd_with_exit_callback(args, callback)` for upstream-style duplicate exit callback semantics. Exact JavaScript package export shape is not implemented. |
+
+## Core And Tokenizer Helpers
+
+| Upstream API | Rust status | Notes |
+| --- | --- | --- |
+| `getDefaultOptions()` | covered | Use `get_default_options()`. Defaults are also available through `Options::default()`. |
+| `getSupportedFormats()` | covered | Use `get_supported_formats()`. The registry is generated from upstream and currently has 223 formats. |
+| `getFormatByFile(path, formatsExts?, formatsNames?)` | covered | Use `get_format_by_file(path)` for default mappings or `get_format_by_file_with_mappings(path, formats_exts, formats_names)` for explicit mappings. |
+| `Tokenizer` class | covered natively | Use `Tokenizer::new()` or `Tokenizer::with_options(options)` and `generate_maps(source_id, content, format)`. Exact JavaScript package export shape is not implemented. |
+| `Detector` class | covered natively | Use `Detector::new(options)` for stateful in-memory source detection or `detect_source_files(files, options)` for batch detection. Exact JavaScript constructor shape is not implemented. |
+| `Statistic` class | covered natively | Use `Statistic::new()`, `match_source(...)`, `clone_found(...)`, and `get_statistic()` for upstream-style statistics accumulation. |
+| `MemoryStore` class | covered natively | Use `MemoryStore<T>::new()`, `namespace(...)`, `get(...)`, `set(...)`, and `close()` for the native in-memory store concept. |
+| Validators, subscribers, custom stores, custom reporters | option-surface only | CLI/config options are preserved where practical; dynamic npm loading is intentionally out of scope for the first release. |
+
+## Server API
+
+| Upstream API | Rust status | Notes |
+| --- | --- | --- |
+| `jscpd-server` binary | partial | Native binary covers help, startup, core HTTP routes, and MCP smoke contracts through `scripts/compat-server.sh`. Exact Streamable HTTP SDK edge cases remain follow-up. |
+| `startServer`, `JscpdServer`, `JscpdServerService` exports | partial | Native server modules exist, but JavaScript export shape parity is not implemented. |
+
+## Remaining API Gaps
+
+- Decide later whether a JavaScript package wrapper is worth shipping. The
+  current recommendation is to keep the first release native-only.
+- Keep `jscpd(args)` / `jscpd_with_exit_callback(args, callback)` as the native
+  embeddable path; add a JavaScript wrapper only if publishing an npm package
+  becomes an explicit target.
+- Keep `Tokenizer`, `Detector`, `Statistic`, and `MemoryStore` native and
+  detection-oriented for now; add exact JavaScript iterator/object-shape
+  wrappers only if an npm/API compatibility release is chosen.
+- Keep custom store/reporter/provider APIs out of the release path until a real
+  integration requires native hooks.