jscpd-rs 0.1.0

package/docs/release-checklist.md

@@ -0,0 +1,200 @@
+# Release Checklist
+
+This checklist is the publication runbook for the first Rust release candidate.
+Policy decisions live in `docs/release-decisions.md`; current evidence lives in
+`docs/compat-baseline.md` and `docs/release-readiness.md`.
+
+## Current Release Candidate Evidence
+
+Latest full local prepublish gate:
+
+```bash
+scripts/prepublish-check.sh
+```
+
+Passed on 2026-05-31 at code commit `8c3da0e`. This includes
+`scripts/release-candidate.sh`, package/install verification, crate/tag
+availability checks, npm package/name/npx verification, and
+`cargo publish --dry-run --locked`. Later documentation-only commits may reuse
+the release-candidate evidence if they do not change code, scripts, package
+metadata, or benchmark configuration; rerun
+`RUN_RELEASE_CANDIDATE=0 scripts/prepublish-check.sh` after documentation edits
+so package/dry-run evidence matches the exact package contents being tagged.
+
+GitHub Actions default `release-gate` must pass on the exact pushed commit
+being published. Check the current run in GitHub Actions after the final push;
+the publish blocker below is the authoritative gate.
+
+Latest GitHub Actions default release-gate:
+
+```text
+push
+```
+
+Passed on 2026-05-31 at code commit `8c3da0e`:
+https://github.com/vv-bogdanov/jscpd-rs/actions/runs/26710762680
+
+CI timing snapshot after the first cache/timing pass, from cold GitHub Actions
+run `26710415211` on commit `7bdf12f`:
+
+| Step | Time |
+| --- | ---: |
+| `package/install check` | 192s |
+| `CLI compatibility` | 56s |
+| `cargo test` | 35s |
+| `server compatibility` | 28s |
+| `config compatibility` | 4s |
+
+Optimized default CI run `26710686373` on commit `71088f1` completed in
+2m18s. Current warm-cache bottlenecks:
+
+| Step | Time |
+| --- | ---: |
+| `CLI compatibility` | 36s |
+| `server compatibility` | 29s |
+| `package/install check` | 23s |
+| `cargo test` | 6s |
+| `config compatibility` | 6s |
+
+The workflow now restores Cargo, pnpm, and upstream build caches, resolves the
+pnpm store with upstream's `pnpm@10.28.0`, validates restored pnpm
+`node_modules` symlinks before using upstream builds, skips `clippy`
+installation outside release-candidate runs, and uses a target-reuse npm
+package smoke. Keep the full cold npm source-build in
+`scripts/prepublish-check.sh` and release-candidate runs.
+
+Recorded public benchmark baseline for this release candidate:
+
+| Case | Commit | Format | Rust avg | Upstream avg | Speedup | Compat |
+| --- | --- | --- | ---: | ---: | ---: | --- |
+| `react` | `f0dfee3` | `javascript` | 0.199097s | 10.079214s | 50.62x | pass |
+| `next` | `2bbb67b9` | `typescript` | 0.262433s | 14.715736s | 56.07x | pass |
+| `prometheus` | `a0524ee` | `go` | 0.085239s | 4.642435s | 54.46x | pass |
+
+## Publish Blockers
+
+Before publishing, all of these must be true:
+
+- `git status --short` is clean.
+- `git submodule status jscpd` points at the reviewed upstream reference.
+- `scripts/release-candidate.sh` passes on the exact code commit being tagged.
+- GitHub Actions `release-gate` passes on the pushed commit.
+- `scripts/package-check.sh` passes and the package file list excludes
+  `jscpd/`, `target/`, `node_modules/`, and `scripts/`.
+- `scripts/npm-package-check.sh` passes, including `npm pack`,
+  `npm publish --dry-run --json`, local install smoke checks for `jscpd-rs`,
+  `jscpd`, and `jscpd-server`, and an `npx --package <tarball>` smoke run.
+- `cargo publish --dry-run --locked` passes for the exact package manifest and
+  include list being published.
+- `README.md`, `docs/compat-baseline.md`, and
+  `docs/public-benchmark-suite.md` contain the same recorded public benchmark
+  numbers.
+- For the first publication, the `jscpd-rs` crate and npm package names are
+  still available or already owned by this project, and `v0.1.0` does not
+  already exist locally or on the remote.
+- `docs/upstream-bugs.md` contains concrete repro commands for upstream issues
+  we plan to file.
+- `docs/upstream-issue-drafts.md` contains reviewed issue drafts ready to
+  verify against current upstream and post.
+- `CHANGELOG.md` contains the exact release notes for the version being tagged.
+
+## First-Release Scope
+
+Treat these as release scope:
+
+- `jscpd` and `jscpd-server` binaries with upstream-compatible command names.
+- CLI/config option surface covered by `scripts/compat-cli.sh` and
+  `scripts/compat-config.sh`.
+- Coverage-first duplicate parity: Rust must not miss duplicated upstream lines
+  for the same inputs and options.
+- Built-in native reporters: `ai`, `console`, `consoleFull`, `csv`, `html`,
+  `json`, `markdown`, `silent`, `sarif`, `threshold`, `xcode`, `xml`, and
+  `badge`.
+- Native file discovery, format registry, JS/TS/JSX/TSX tokenization, generic
+  long-tail tokenization, blame, native API, and native server endpoints listed
+  in `docs/release-readiness.md`.
+
+## Intentional First-Release Deviations
+
+These are not publication blockers for the first release:
+
+- Dynamic npm reporters, stores, listeners, and plugins are not loaded. The
+  compatible option surface and upstream-style missing-package warnings are the
+  release contract.
+- Exact token totals, pair ordering, and boundaries are diagnostic while
+  upstream duplicated lines remain covered.
+- HTML output is self-contained and practically compatible, not pixel-perfect.
+- The Rust crate exposes a native Rust API, not the upstream JavaScript package
+  API.
+- Persistent store/cache backends remain demand-driven until benchmark data
+  shows the in-memory path is insufficient.
+- Full Prism grammar parity for every long-tail format is not attempted; promote
+  formats only when coverage gates show missed upstream lines.
+
+## Pre-Tag Commands
+
+Run from the repository root:
+
+```bash
+scripts/prepublish-check.sh
+```
+
+The script checks clean git state, the reviewed `jscpd` submodule reference,
+local and remote tag availability, exact crate-name availability through
+`cargo search`, exact npm package-name availability through `npm view`,
+benchmark-number consistency across release docs, the full release-candidate
+gate, package/install validation, npm pack/npx validation, and
+`cargo publish --dry-run --locked`. Set `RUN_RELEASE_CANDIDATE=0` only when the
+same code commit already has fresh local and CI release-candidate evidence.
+
+Then push the exact release commit and verify the GitHub Actions
+`release-gate` result. Use the workflow dispatch `release_candidate` input for a
+full CI-side release-candidate run when needed.
+
+For the first publication candidate checked on 2026-05-31, local and remote
+`v0.1.0` tag lookups returned no entries. `cargo search jscpd-rs --limit 5`
+returned no exact crate, `npm view jscpd-rs version` returned `E404`, and the
+sparse crates.io index path
+`https://index.crates.io/js/cp/jscpd-rs` returned 404.
+
+## Post-Tag Smoke
+
+After tagging or publishing, install the package into a temporary Cargo root and
+check the binaries:
+
+```bash
+cargo install --path . --bin jscpd --root /tmp/jscpd-rs-install --force --locked
+cargo install --path . --bin jscpd-server --root /tmp/jscpd-rs-install --force --locked
+/tmp/jscpd-rs-install/bin/jscpd --version
+/tmp/jscpd-rs-install/bin/jscpd --help
+/tmp/jscpd-rs-install/bin/jscpd-server --version
+```
+
+The package gate itself installs both binaries with `cargo install --bins`; the
+two explicit commands above are a manual smoke equivalent for post-tag checks.
+
+## Next Release Themes
+
+Track these after the first release candidate:
+
+- Reduce noisy extra Rust findings where they are user-visible false positives.
+- Add native persistent store/cache only if release-scale benchmark data needs
+  it.
+- Tighten MCP Streamable HTTP SDK edge cases if real MCP clients require them.
+- Promote long-tail tokenizers only from concrete missed-coverage evidence.
+- File upstream bug reports from `docs/upstream-issue-drafts.md`.
+
+## CI Speed Plan
+
+Keep this as part of the release plan:
+
+- Treat `package/install check`, `CLI compatibility`, `cargo test`, and
+  `server compatibility` as the CI bottleneck watchlist.
+- Compare future warm-cache GitHub Actions runs with optimized run
+  `26710686373` and cold run `26710415211`.
+- Keep default push/PR CI as a fast confidence gate; keep cold npm source
+  install, `clippy`, full matrix, and public benchmark coverage in
+  `scripts/prepublish-check.sh` and manual release-candidate workflow runs.
+- If default CI remains above roughly three minutes after warm caches, split the
+  package surface smoke from the compatibility gates into separate jobs or make
+  package source-build a release-candidate-only job.

package/docs/release-decisions.md

@@ -0,0 +1,103 @@
+# Release Decisions
+
+Approved on 2026-05-30.
+
+These decisions define the first-release direction. They can change only when a
+compatibility gate, public benchmark, or real user workflow proves the current
+choice insufficient.
+
+## Compatibility Gate
+
+- The blocking rule is coverage-first parity: Rust must not miss duplicate
+  lines reported by upstream `jscpd` for the same input and options.
+- Rust may report additional duplicates while compatibility is converging.
+  Extra findings remain visible in comparison output and should be reduced when
+  they are false positives or noisy user-visible ranges.
+- Exact 1:1 clone boundaries, pair ordering, and token totals are quality
+  metrics, not the default release blocker.
+
+## Runtime Strategy
+
+- Do not embed a JavaScript runtime or spawn upstream JavaScript from Rust to
+  implement tokenizers, reporters, stores, or plugins.
+- Prefer native Rust implementations backed by maintained crates.
+- If a format cannot be supported well without a large custom port, keep it on
+  generic tokenization until a fixture or public-repo gate shows missed
+  upstream coverage.
+
+## Reporters
+
+- Built-in reporters are first-release scope and should be implemented natively:
+  `ai`, `console`, `consoleFull`, `csv`, `html`, `json`, `markdown`, `silent`,
+  `sarif`, `threshold`, `xcode`, `xml`, and the commonly used badge reporter.
+- Machine-readable reporter contracts are strict release scope: JSON, XML,
+  SARIF, CSV, and Markdown should stay structurally compatible with upstream.
+- HTML must remain practically compatible and self-contained. Pixel-perfect
+  upstream parity is not a first-release blocker.
+- Dynamic npm reporter loading is post-MVP. Unknown reporter names should keep
+  the upstream-style warning and continue where upstream continues.
+
+## Stores And Cache
+
+- The default in-memory store is the first-release store path.
+- `--store <name>` should keep the upstream missing-store fallback shape unless
+  a native store backend is deliberately added.
+- Dynamic npm store loading is post-MVP.
+- Native persistent/cache stores are demand-driven. Add one only if public
+  benchmark data shows the in-memory path is not enough for release viability.
+- `cache`, config `listeners`, and `tokensToSkip` remain option-surface
+  compatibility fields while upstream exposes but does not consume them in the
+  current CLI runtime.
+
+## Formats And Tokenization
+
+- JS/TS/JSX/TSX stay on the Oxc-backed native path.
+- Markdown, Vue, Svelte, Astro, Apex, and markup keep small native block
+  splitters where that unlocks upstream coverage without a full grammar port.
+- Long-tail formats use the upstream-synchronized registry plus generic
+  tokenization by default.
+- Promote a long-tail format only with a focused fixture, `STRICT=coverage`
+  comparison, and a clear reason that generic tokenization is insufficient.
+
+## CLI And Upstream Quirks
+
+- Mirror upstream behavior when it is visible, common, and covered by tests.
+- Crash-only Commander edge cases are documented first and mirrored only if a
+  release gate or user workflow makes exact behavior necessary.
+- Keep upstream bug candidates in `docs/upstream-bugs.md` so we can file them
+  later with concrete repro commands.
+
+## Blame
+
+- Blame stays native through Git commands or a Git library.
+- Prefer per-file failure isolation over inheriting upstream nested-repository
+  failure modes.
+
+## Performance
+
+- Performance remains a product requirement. Public benchmark runs should be
+  repeated before publication with pinned commits and recorded speedups.
+- The aspirational target is 50x on representative cases, but release gating
+  should use measured thresholds from the selected public benchmark suite.
+
+## Approved Complex Feature Choices
+
+These are the current choices for features that are expensive to clone exactly:
+the user explicitly approved these tradeoffs on 2026-05-30.
+
+- Dynamic npm reporters, stores, listeners, and plugins: do not implement for
+  the first release. Keep option-surface compatibility, native built-ins, and
+  upstream-style missing-package warnings.
+- Long-tail Prism tokenizer parity: do not port all grammars eagerly. Keep the
+  upstream-synchronized format registry plus generic tokenization, then promote
+  formats only when fixtures or public-repo gates show missed upstream coverage.
+- Extra Rust findings: acceptable while compatibility converges. The release
+  blocker is missing upstream duplicated line coverage, not 1:1 pair identity.
+- Node/Commander quirks: mirror visible behavior only when covered by gates.
+  Document crash-only or incidental quirks in `docs/upstream-bugs.md`.
+- HTML reporter: keep self-contained and practically compatible. Pixel-perfect
+  parity is not a first-release blocker.
+- Persistent cache/store backends: postpone until benchmark data shows the
+  in-memory detector is insufficient on release-scale repositories.
+- Blame failures: prefer native per-file isolation instead of inheriting
+  upstream failure modes around nested or unusual Git repositories.

package/docs/release-readiness.md

@@ -0,0 +1,51 @@
+# Release Readiness
+
+Last updated: 2026-05-31.
+
+This is the working component checklist for the first release. The authoritative
+policy decisions are still in `docs/release-decisions.md`; this file tracks the
+current implementation status.
+
+## Ready For First Release
+
+| Component | Status | Notes |
+| --- | --- | --- |
+| Binary/package surface | ready | `jscpd` binary name, Cargo package include list, publish dry-run, install check, version/help smoke checks, npm package metadata, and local `npx` smoke checks. |
+| CLI option surface | ready | Main upstream flags are parsed, including visible Commander quirks gated by `scripts/compat-cli.sh`. |
+| Config loading | ready | `.jscpd.json` and `package.json#jscpd`, config-relative paths/ignore, malformed JSON behavior, symlinked explicit config paths. |
+| File discovery | ready | Format filters, custom extensions/names, `.gitignore`, global Git excludes, symlink policy, shebang detection, max size/line filtering. |
+| Format registry | ready | Generated from upstream tokenizer build; current registry has 223 formats and 206 extension mappings. |
+| Detector core | ready | Numeric hashes, compact token streams, per-format sharding, parallel preparation/detection, coverage-first comparator. |
+| Hot JS/TS tokenization | ready | Native Oxc-backed paths for JavaScript, TypeScript, JSX, and TSX with coverage gates. |
+| Embedded/block formats | ready | Markdown, markup, Vue, Svelte, Astro, Apex, and TAP have native block handling where needed for upstream coverage. |
+| Built-in reporters | ready | `ai`, `console`, `consoleFull`, `csv`, `html`, `json`, `markdown`, `silent`, `sarif`, `threshold`, `xcode`, `xml`, and `badge`; file reporters are gated for clone and no-duplicate reports. |
+| Blame | ready | Native `git blame -w` data is populated and gated by `scripts/compat-blame.sh`. |
+| Native Rust API | ready | `jscpd`, `jscpd_with_exit_callback`, `Tokenizer`, `Detector`, `Statistic`, `MemoryStore`, `detect_clones`, `detect_clones_and_statistic`, `detect_clones_and_statistics`, `detect_source_files`, default options, argv option parsing, supported formats, and format lookup helpers expose the app, tokenizer, detector, statistics, and store core for path-based and in-memory integrations. See `docs/api-parity.md`. |
+| Native server | partial | `jscpd-server` exposes `/`, `/api/health`, `/api/stats`, `/api/check`, `/api/recheck`, and `/mcp`; exact help text, stable CLI, HTTP success/error, and MCP contracts are gated; `/api/check` reuses prepared project token maps; exact upstream Streamable HTTP SDK behavior remains follow-up. |
+| Performance harness | ready | Local benchmark script and public benchmark suite with pinned output recording and speedup gates. |
+| Release gates | ready | Default CI gate, full compatibility matrix, Cargo/npm package checks, reporter/config/CLI/blame gates. The default gate now prints per-step timings, caches Cargo/pnpm/upstream build artifacts, and uses target-reuse npm package smoke in push/PR CI while keeping cold npm source-build in release-candidate/prepublish gates. |
+
+## Partial Or Follow-Up
+
+| Component | Status | Recommended action |
+| --- | --- | --- |
+| Long-tail tokenization | coverage-first | Keep generic tokenization by default. Promote formats only when fixtures or public repos show missed upstream coverage. |
+| Exact pair parity | diagnostic | Do not block release while every upstream duplicated line is covered. Reduce noisy extras after user-facing reports become annoying. |
+| Token totals | diagnostic | Native token streams may differ from Prism. Keep report-visible clone coverage as the gate. |
+| HTML reporter polish | practical parity | Keep self-contained HTML stable. Do not chase pixel-perfect upstream parity for the first release. |
+| Terminal cosmetics | practical parity | Important messages are gated; exact wrapping/order remains lower priority. |
+| Upstream JavaScript API parity | follow-up | Native Rust helpers cover the practical app/tokenizer/detector/statistics/store concepts, including an embeddable argv runner and tokenizer map generation; exact JS package export shape is not implemented in the Rust crate. See `docs/api-parity.md`. |
+| Server snippet matching | optimized baseline | Native `/api/check` and MCP `check_duplication` are functional and reuse project token maps from the last scan; add a dedicated window index only if real server benchmarks require it. |
+| Npm prebuilt binaries | follow-up | The first npm package is source-build: install/postinstall compiles with Cargo. Add platform-specific prebuilt packages before broad npm promotion if install speed or Rust toolchain requirements become a blocker. |
+| Latest full publication gate | ready | `scripts/prepublish-check.sh` passed locally on code commit `8c3da0e`, including `scripts/release-candidate.sh`, package/install verification, crate/tag availability checks, npm package/name/npx verification, and `cargo publish --dry-run --locked`. GitHub Actions default `release-gate` passed on code commit `8c3da0e` in run `26710762680`. After benchmark documentation updates, `RUN_RELEASE_CANDIDATE=0 scripts/prepublish-check.sh` is the package/dry-run refresh gate for the exact package contents being tagged. |
+
+## Post-MVP
+
+| Component | Decision |
+| --- | --- |
+| Dynamic npm reporters | Do not implement for the first release; keep upstream-style missing-package warnings. |
+| Dynamic npm stores | Do not implement for the first release; default in-memory store is the release path. |
+| Listeners/plugins runtime | Option-surface compatibility only unless a real workflow requires native support. |
+| MCP endpoint polish | Core native endpoint exists; tighten exact SDK edge cases only when MCP client compatibility demands it. |
+| Persistent cache/store backends | Add only if public benchmark data proves the in-memory path is insufficient. |
+| Full Prism grammar port | Do not rewrite all grammars eagerly; use native crates or small scanners only for proven gaps. |