clonehunter 1.0.0__tar.gz

clonehunter-1.0.0/.gitignore

@@ -0,0 +1,266 @@
+# Created by https://www.toptal.com/developers/gitignore/api/python,visualstudiocode,macos,windows,venv
+# Edit at https://www.toptal.com/developers/gitignore?templates=python,visualstudiocode,macos,windows,venv
+
+### macOS ###
+# General
+.DS_Store
+.AppleDouble
+.LSOverride
+
+# Icon must end with two \r
+Icon
+
+
+# Thumbnails
+._*
+
+# Files that might appear in the root of a volume
+.DocumentRevisions-V100
+.fseventsd
+.Spotlight-V100
+.TemporaryItems
+.Trashes
+.VolumeIcon.icns
+.com.apple.timemachine.donotpresent
+
+# Directories potentially created on remote AFP share
+.AppleDB
+.AppleDesktop
+Network Trash Folder
+Temporary Items
+.apdisk
+
+### macOS Patch ###
+# iCloud generated files
+*.icloud
+
+### Python ###
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#pdm.lock
+#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
+#   in version control.
+#   https://pdm.fming.dev/#use-with-ide
+.pdm.toml
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/
+
+### Python Patch ###
+# Poetry local configuration file - https://python-poetry.org/docs/configuration/#local-configuration
+poetry.toml
+
+# ruff
+.ruff_cache/
+
+# LSP config files
+pyrightconfig.json
+
+### venv ###
+# Virtualenv
+# http://iamzed.com/2009/05/07/a-primer-on-virtualenv/
+[Bb]in
+[Ii]nclude
+[Ll]ib
+[Ll]ib64
+[Ll]ocal
+[Ss]cripts
+pyvenv.cfg
+pip-selfcheck.json
+
+### VisualStudioCode ###
+.vscode/*
+!.vscode/settings.json
+!.vscode/tasks.json
+!.vscode/launch.json
+!.vscode/extensions.json
+!.vscode/*.code-snippets
+
+# Local History for Visual Studio Code
+.history/
+
+# Built Visual Studio Code Extensions
+*.vsix
+
+### VisualStudioCode Patch ###
+# Ignore all local history of files
+.history
+.ionide
+
+### Windows ###
+# Windows thumbnail cache files
+Thumbs.db
+Thumbs.db:encryptable
+ehthumbs.db
+ehthumbs_vista.db
+
+# Dump file
+*.stackdump
+
+# Folder config file
+[Dd]esktop.ini
+
+# Recycle Bin used on file shares
+$RECYCLE.BIN/
+
+# Windows Installer files
+*.cab
+*.msi
+*.msix
+*.msm
+*.msp
+
+# Windows shortcuts
+*.lnk
+
+# End of https://www.toptal.com/developers/gitignore/api/python,visualstudiocode,macos,windows,venv

clonehunter-1.0.0/.python-version

@@ -0,0 +1 @@
+3.13

clonehunter-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 David Rogers
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

clonehunter-1.0.0/PKG-INFO

@@ -0,0 +1,308 @@
+Metadata-Version: 2.4
+Name: clonehunter
+Version: 1.0.0
+Summary: Find duplicate code in mixed-language repositories using semantic and lexical similarity
+Project-URL: Homepage, https://github.com/drogers0/clonehunter
+Project-URL: Source, https://github.com/drogers0/clonehunter
+Project-URL: Issues, https://github.com/drogers0/clonehunter/issues
+License-Expression: MIT
+License-File: LICENSE
+Keywords: code-clone,developer-tools,duplicate-code,refactoring,static-analysis
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: Software Development :: Quality Assurance
+Requires-Python: >=3.10
+Requires-Dist: numpy>=2.0.0
+Requires-Dist: tomli>=2.0.1; python_version < '3.11'
+Requires-Dist: torch>=2.2.0
+Requires-Dist: tqdm>=4.66.0
+Requires-Dist: transformers>=4.40.0
+Provides-Extra: dev
+Requires-Dist: pyright>=1.1.380; extra == 'dev'
+Requires-Dist: pytest>=8.4.0; extra == 'dev'
+Requires-Dist: ruff>=0.15.0; extra == 'dev'
+Provides-Extra: faiss
+Requires-Dist: faiss-cpu>=1.7.4; extra == 'faiss'
+Description-Content-Type: text/markdown
+
+# CloneHunter
+
+CloneHunter finds duplicate code across mixed-language repositories. It uses function-aware analysis for Python and windows-based analysis for other code files, with evidence so you can decide what to refactor.
+
+Under the hood, CloneHunter combines snippet generation (function/window/call-expansion), transformer-based code embeddings (CodeBERT), vector similarity search (brute-force or FAISS), and lexical scoring before rolling matches up into findings and HTML/JSON/SARIF reports. This is intentionally not a lightweight grep-style checker: it runs a semantic retrieval pipeline with model inference and indexing to catch harder, non-trivial duplicate patterns.
+
+![CloneHunter HTML report screenshot](assets/clonehunter-report-demo.png)
+
+## Quickstart
+
+Requires Python 3.10+.
+
+### Install with uv
+
+```bash
+uv python install 3.10
+uv pip install git+https://github.com/drogers0/clonehunter
+```
+
+### Install from a release tag
+
+```bash
+uv pip install git+https://github.com/drogers0/clonehunter@v1.0.0
+```
+
+### Install with venv + pip
+
+```bash
+python3.10 -m venv .venv
+source .venv/bin/activate
+pip install --upgrade pip
+pip install git+https://github.com/drogers0/clonehunter
+```
+
+### From source (dev)
+
+```bash
+git clone https://github.com/drogers0/clonehunter
+cd clonehunter
+uv python install 3.10
+uv sync
+uv pip install -e .
+```
+
+### Run
+
+```bash
+clonehunter scan . --format html --out clonehunter_report.html
+```
+
+If the CLI entrypoint is not on your PATH, use:
+
+```bash
+uv run clonehunter scan . --format html --out clonehunter_report.html
+```
+
+### Notes on dependencies
+
+* `codebert` embedder requires `torch` and `transformers`.
+* `faiss` index is optional; install `faiss-cpu` for faster search.
+* Use `--embedder stub` for quick local runs without ML dependencies.
+
+---
+
+## Basic Usage
+
+Scan a repository (defaults to HTML and `clonehunter_report.html`; output extension follows `--format`):
+
+```bash
+uv run clonehunter scan .
+```
+
+Generate a JSON report:
+
+```bash
+uv run clonehunter scan . --format json --out report.json
+```
+
+Generate an HTML report:
+
+```bash
+uv run clonehunter scan . --format html --out report.html
+```
+
+Generate a SARIF report:
+
+```bash
+uv run clonehunter scan . --format sarif --out report.sarif
+```
+
+Diff scan (compare changed files against the full repo):
+
+```bash
+clonehunter diff --base HEAD --format json --out diff.json
+```
+
+### Language Support
+
+* **Python files**: parsed with AST extraction and analyzed with function/window snippets.
+* **Other code files**: analyzed in implicit windows-only mode by file content.
+* **Cross-file-type comparisons** are allowed.
+* Results can vary by language and repository shape; tune thresholds/windows for best quality.
+
+---
+
+## How Scoring Works
+
+Scores are **composite**: embedding similarity blended with lexical similarity.
+
+```
+composite = (1 - lexical_weight) * embedding + lexical_weight * lexical
+```
+
+Matches are filtered by `lexical_min_ratio`, and then the composite score is compared against the relevant threshold (`func`, `win`, or `exp`).
+
+---
+
+## Configuration (pyproject.toml)
+
+```toml
+[tool.clonehunter]
+engine = "semantic"
+cluster_findings = false
+cluster_min_size = 2
+
+[tool.clonehunter.thresholds]
+func = 0.92
+win = 0.90
+exp = 0.90
+min_window_hits = 2
+lexical_min_ratio = 0.5
+lexical_weight = 0.3
+
+[tool.clonehunter.windows]
+window_lines = 12
+stride_lines = 6
+min_nonempty = 4
+
+[tool.clonehunter.expansion]
+enabled = false
+depth = 1
+max_chars = 4000
+
+[tool.clonehunter.index]
+name = "brute"
+top_k = 25
+
+[tool.clonehunter.cache]
+path = "~/.cache/clonehunter"
+
+[tool.clonehunter.embedder]
+name = "codebert"
+model_name = "microsoft/codebert-base"
+revision = "main"
+max_length = 256
+batch_size = 16
+device = "cpu"
+```
+
+By default, CLI scans apply the `monorepo` repotype preset unless overridden with `--repotype` or `--repotype none`.
+
+---
+
+## CLI Options (selected)
+
+```
+clonehunter scan [PATHS...] [--format json|html|sarif] [--out FILE]
+  --engine semantic|sonarqube
+  --embedder codebert|stub
+  --index brute|faiss
+  --threshold-func FLOAT
+  --threshold-win FLOAT
+  --threshold-exp FLOAT
+  --min-window-hits INT
+  --lexical-min-ratio FLOAT
+  --lexical-weight FLOAT
+  --window-lines INT
+  --stride-lines INT
+  --min-nonempty INT
+  --expand-calls
+  --expand-depth INT
+  --expand-max-chars INT
+  --cache-path PATH
+  --cluster
+  --cluster-min-size INT
+  --repotype dotnet|go|java|kotlin|monorepo|node|none|php|python|react|ruby|rust|swift|cpp
+                                  # repeatable preset globs
+  --include-globs GLOB   # repeatable; merged with config includes
+  --exclude-globs GLOB   # repeatable; merged with config excludes
+
+clonehunter diff --base REF [--format json|html|sarif] [--out FILE]
+```
+
+`--repotype` is additive and can be mixed (for example `--repotype python --repotype react`).
+If `--repotype` is omitted, CloneHunter defaults to `monorepo`.
+Use `--repotype none` to disable repotype presets.
+Merge order is: `pyproject.toml` globs, then `--repotype`, then explicit `--include-globs/--exclude-globs`.
+When the same glob appears in both include/exclude sets, the most recent CLI layer wins.
+
+Example mixed-language scan with custom overrides:
+
+```bash
+uv run clonehunter scan . \
+  --repotype python \
+  --repotype react \
+  --repotype cpp \
+  --exclude-globs "**/__generated__/**" \
+  --include-globs "**/*.txt" \
+  --format html --out report.html
+```
+
+---
+
+## Example Outputs
+
+Example reports live in the `examples/` folder:
+
+* [`examples/clonehunter_report.html`](examples/clonehunter_report.html)
+* [`examples/clonehunter_report.json`](examples/clonehunter_report.json)
+* [`examples/clonehunter_report.sarif`](examples/clonehunter_report.sarif)
+* [`examples/clonehunter_diff.json`](examples/clonehunter_diff.json)
+* [`examples/clonehunter_diff_report.html`](examples/clonehunter_diff_report.html)
+
+Generate the example reports:
+
+```bash
+uv run clonehunter scan . --format html --out examples/clonehunter_report.html
+uv run clonehunter scan . --format json --out examples/clonehunter_report.json
+uv run clonehunter scan . --format sarif --out examples/clonehunter_report.sarif
+uv run clonehunter diff --base HEAD --format json --out examples/clonehunter_diff.json
+uv run clonehunter diff --base HEAD --format html --out examples/clonehunter_diff_report.html
+```
+
+---
+
+## Tuning Tips
+
+* If you see too many false positives, increase `lexical_min_ratio` and/or `lexical_weight`.
+* Increase `min_window_hits` to require stronger evidence.
+* Exclude tests or generated code via `exclude_globs`.
+
+---
+
+## Limitations
+
+* Semantic similarity is approximate, not guaranteed equivalence.
+* Python findings are generally richer due to AST/function context.
+* Non-Python findings use windows-only analysis and may require threshold/window tuning.
+* Very small functions are harder to compare meaningfully.
+* Domain-specific logic may require threshold tuning.
+
+---
+
+## Development
+
+Run the full test suite:
+
+```bash
+python -m pytest
+```
+
+Run formatting and type checks:
+
+```bash
+ruff format .
+ruff check .
+pyright
+```
+
+Install dev dependencies:
+
+```bash
+uv pip install -e ".[dev,faiss]"
+```