csim-ai 0.0.1__tar.gz

csim_ai-0.0.1/.gitignore

@@ -0,0 +1,27 @@
+.venv/
+__pycache__/
+*.pyc
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+dist/
+build/
+*.egg-info/
+
+# model checkpoints -- large binaries, don't belong in git history.
+# train_log_v1.jsonl / *_log.jsonl (metric history) files are tracked.
+training/artifacts/best_checkpoint/
+training/artifacts/last_checkpoint/
+training/artifacts/onnx_model/
+training/artifacts/temp_sweep/*/best_checkpoint/
+training/artifacts/temp_sweep/*/last_checkpoint/
+
+# Dolos CLI (Fase 4 baseline) -- npm install output, not source.
+training/eval/dolos_tool/node_modules/
+training/eval/dolos_tool/package-lock.json
+dolos-report-*/
+
+# Fase 4 scorer -- raw per-pair feature dumps (tens of MB, regenerable in
+# ~15-20min via build_features.py). fusion_model_v1.joblib (~300KB) and
+# eval_fusion_v1.json (the actual result) are tracked.
+training/scorer/artifacts/features_*.jsonl

csim_ai-0.0.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Edson Eddy
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

csim_ai-0.0.1/PKG-INFO

@@ -0,0 +1,196 @@
+Metadata-Version: 2.5
+Name: csim-ai
+Version: 0.0.1
+Summary: Neural-augmented Python code plagiarism detection for programming judges.
+Project-URL: Homepage, https://github.com/edsoneddy/csim-ai
+Project-URL: Source Code, https://github.com/edsoneddy/csim-ai
+Project-URL: Bug Tracker, https://github.com/edsoneddy/csim-ai/issues
+Project-URL: Documentation, https://github.com/edsoneddy/csim-ai#readme
+Author-email: Edsone Eddy <crew0eddy@gmail.com>
+License: MIT
+License-File: LICENSE
+Keywords: bi-encoder,code similarity,onnx,plagiarism detection,programming judges
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Software Development :: Quality Assurance
+Requires-Python: >=3.10
+Requires-Dist: huggingface-hub>=0.25
+Requires-Dist: numpy>=2.0
+Requires-Dist: onnxruntime>=1.20
+Requires-Dist: tokenizers>=0.20
+Provides-Extra: ast
+Requires-Dist: csim>=3.3.0; extra == 'ast'
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Provides-Extra: export
+Requires-Dist: onnx>=1.15; extra == 'export'
+Requires-Dist: torch>=2.9; extra == 'export'
+Requires-Dist: transformers>=5.0; extra == 'export'
+Provides-Extra: scorer
+Requires-Dist: joblib>=1.3; extra == 'scorer'
+Requires-Dist: scikit-learn>=1.5; extra == 'scorer'
+Provides-Extra: torch
+Requires-Dist: sentence-transformers>=3.0; extra == 'torch'
+Requires-Dist: torch>=2.9; extra == 'torch'
+Requires-Dist: transformers>=5.0; extra == 'torch'
+Provides-Extra: train
+Requires-Dist: csim>=3.3.0; extra == 'train'
+Requires-Dist: libcst>=1.0; extra == 'train'
+Requires-Dist: pyyaml>=6.0; extra == 'train'
+Description-Content-Type: text/markdown
+
+# csim-ai
+
+Neural-augmented Python code plagiarism detection for programming judges.
+Successor to [csim](https://github.com/edsoneddy/csim) (ANTLR4 parse-tree
+normalization + Tree Edit Distance), adding a contrastively fine-tuned
+bi-encoder for the structural/semantic plagiarism cases where pure TED
+similarity degrades. Scores are a fusion of both signals via a small
+GBDT, verified to beat [Dolos](https://dolos.ugent.be/) on this
+project's own test data -- see [docs/REPORT.md](docs/REPORT.md) for the
+full methodology and results, [docs/DEVELOPMENT.md](docs/DEVELOPMENT.md)
+for the phase-by-phase build log.
+
+**Task**: plagiarism detection (did B derive from A?), not semantic clone
+detection (does B solve the same problem as A?). Two independent correct
+solutions to the same problem are a negative, not a positive.
+
+## Install
+
+```bash
+pip install csim-ai              # bi-encoder cosine similarity only (onnxruntime, no torch)
+pip install csim-ai[ast,scorer]  # + csim TED signal + GBDT fusion -- the full hybrid score
+```
+
+Model weights aren't bundled in the package (the ONNX export is
+~500MB) -- run `csim-ai setup` once after installing to download and
+cache them from Hugging Face Hub
+([edson-eddy/csim-ai](https://huggingface.co/edson-eddy/csim-ai)).
+After that, both the CLI and the `Scorer` class auto-detect the cache
+and give the full hybrid score with no further flags or arguments.
+
+```bash
+pip install csim-ai[ast,scorer]
+csim-ai setup
+# bi-encoder cached at: ~/.cache/huggingface/hub/models--edson-eddy--csim-ai/...
+# fusion model cached at: .../fusion_model.joblib
+```
+
+## CLI
+
+**CLI shape follows csim's**, not a from-scratch design: a single
+`csim-ai` command, an action positional, `--path` pointing at a
+directory compared exhaustively -- same pattern as `csim
+{report,group,tree,view,info} --path DIR --lang ... --talg ...`, since
+this tool has the same predecessor and audience.
+
+```bash
+csim-ai report --path submissions/
+# b.py is similar to a.py with similarity index: 0.9998 (biencoder_cosine=1.0000, csim_ted=1.0000, fusion=0.9998)
+
+csim-ai group --path submissions/ --threshold 0.9
+# Group 1 (Average Similarity: 1.00):
+# a.py
+# c.py
+# Unique Files (similarity below threshold):
+# b.py
+
+csim-ai info
+# which optional backends (onnxruntime, tokenizers, huggingface_hub, csim, scikit-learn, torch) are available
+```
+
+### `report`
+
+Pairwise similarity report over every `.py` file in `--path`, all
+combinations.
+
+| Flag | Default | Meaning |
+|---|---|---|
+| `--path`, `-p` | required | Directory of `.py` files to compare exhaustively. |
+| `--model-path` | Hugging Face Hub | Directory with `model.onnx`/`tokenizer.json`. Skips the Hub entirely if given. |
+| `--fusion-model` | none | Path to a `fusion_model.joblib`. Skips the Hub entirely if given. |
+| `--use-fusion` | off | Force-download the fusion model from HF Hub if it isn't already cached and no `--fusion-model` is given. |
+
+### `group`
+
+Same comparison as `report`, but groups files into connected components
+by a similarity threshold instead of listing every pair.
+
+Same flags as `report`, plus:
+
+| Flag | Default | Meaning |
+|---|---|---|
+| `--threshold`, `-t` | required | Similarity threshold (0.0-1.0) for grouping. |
+
+### `info`
+
+No comparison -- just reports which optional backends are importable
+(`onnxruntime`/`tokenizers`/`huggingface_hub` from the base install;
+`csim`/`scikit-learn` from `[ast,scorer]`; `torch` from `[export]`).
+Takes an optional `--model-path` to also check a directory for
+`model.onnx`/`tokenizer.json`.
+
+### `setup`
+
+Not part of `pip install .` -- a separate step because the weights
+aren't bundled in the package.
+
+| Flag | Default | Meaning |
+|---|---|---|
+| (none) | -- | Downloads and caches the bi-encoder + fusion model from Hugging Face Hub. |
+| `--export-from CHECKPOINT` | none | Export a local torch checkpoint to ONNX instead of downloading (requires `pip install csim-ai[export]`) -- entirely offline, for your own fine-tuned weights rather than this project's. |
+| `--out` | `./onnx_model` | Output directory for `--export-from`. |
+| `--opset` | `17` | ONNX opset version for `--export-from`. |
+| `--no-verify` | off | Skip the PyTorch-vs-ONNX parity check after `--export-from`. |
+
+For **every** `report`/`group` result, `"similarity index"` is the
+fusion score when available, else `biencoder_cosine`. `csim_ted`/
+`fusion` come back as `None` (and are dropped from the report line) when
+`csim`/`scikit-learn` aren't installed, so a bare `pip install csim-ai`
+(no extras, no `setup`) still gives a usable bi-encoder-only score.
+
+## Python API
+
+```python
+from csim_ai import Scorer
+
+scorer = Scorer()                     # after `csim-ai setup`: full hybrid, cache auto-detected
+scorer = Scorer(use_fusion=True)      # force-downloads the fusion model too if `setup` wasn't run yet
+scorer = Scorer(
+    "path/to/onnx_model",
+    fusion_model_path="path/to/fusion_model.joblib",
+)                                      # fully local, no network
+
+scorer.score(code_a, code_b)
+# {"biencoder_cosine": 0.987, "csim_ted": 0.83, "fusion": 0.978}
+```
+
+`Scorer(model_path=None, fusion_model_path=None, use_fusion=False)`:
+
+- `model_path`: directory with `model.onnx`/`tokenizer.json`. `None`
+  (default) downloads from Hugging Face Hub, cached after first call.
+- `fusion_model_path`: path to a `fusion_model.joblib`. `None` (default)
+  auto-uses a fusion model already cached by a prior `csim-ai setup` or
+  `use_fusion=True` call, without triggering a network request to check.
+- `use_fusion`: if `True` and no `fusion_model_path` is given,
+  force-downloads the fusion model from HF Hub instead of just checking
+  the cache.
+
+`scorer.score(code_a: str, code_b: str) -> dict` returns
+`{"biencoder_cosine": float, "csim_ted": float | None, "fusion": float | None}`
+-- `csim_ted`/`fusion` are `None` when `csim`/`scikit-learn` aren't
+installed or no fusion model is available.
+
+## Layout
+
+```
+src/csim_ai/       inference package -- ONNX bi-encoder + csim TED + GBDT fusion
+tests/             pytest smoke tests for src/csim_ai
+training/          dataset prep, synthetic plagiarism generation, training, eval, export tooling
+docs/
+  REPORT.md        project narrative: problem, methodology, results, limitations
+  DEVELOPMENT.md   phase-by-phase build log: commands, exact numbers, bugs hit and fixed
+```

csim_ai-0.0.1/README.md

@@ -0,0 +1,152 @@
+# csim-ai
+
+Neural-augmented Python code plagiarism detection for programming judges.
+Successor to [csim](https://github.com/edsoneddy/csim) (ANTLR4 parse-tree
+normalization + Tree Edit Distance), adding a contrastively fine-tuned
+bi-encoder for the structural/semantic plagiarism cases where pure TED
+similarity degrades. Scores are a fusion of both signals via a small
+GBDT, verified to beat [Dolos](https://dolos.ugent.be/) on this
+project's own test data -- see [docs/REPORT.md](docs/REPORT.md) for the
+full methodology and results, [docs/DEVELOPMENT.md](docs/DEVELOPMENT.md)
+for the phase-by-phase build log.
+
+**Task**: plagiarism detection (did B derive from A?), not semantic clone
+detection (does B solve the same problem as A?). Two independent correct
+solutions to the same problem are a negative, not a positive.
+
+## Install
+
+```bash
+pip install csim-ai              # bi-encoder cosine similarity only (onnxruntime, no torch)
+pip install csim-ai[ast,scorer]  # + csim TED signal + GBDT fusion -- the full hybrid score
+```
+
+Model weights aren't bundled in the package (the ONNX export is
+~500MB) -- run `csim-ai setup` once after installing to download and
+cache them from Hugging Face Hub
+([edson-eddy/csim-ai](https://huggingface.co/edson-eddy/csim-ai)).
+After that, both the CLI and the `Scorer` class auto-detect the cache
+and give the full hybrid score with no further flags or arguments.
+
+```bash
+pip install csim-ai[ast,scorer]
+csim-ai setup
+# bi-encoder cached at: ~/.cache/huggingface/hub/models--edson-eddy--csim-ai/...
+# fusion model cached at: .../fusion_model.joblib
+```
+
+## CLI
+
+**CLI shape follows csim's**, not a from-scratch design: a single
+`csim-ai` command, an action positional, `--path` pointing at a
+directory compared exhaustively -- same pattern as `csim
+{report,group,tree,view,info} --path DIR --lang ... --talg ...`, since
+this tool has the same predecessor and audience.
+
+```bash
+csim-ai report --path submissions/
+# b.py is similar to a.py with similarity index: 0.9998 (biencoder_cosine=1.0000, csim_ted=1.0000, fusion=0.9998)
+
+csim-ai group --path submissions/ --threshold 0.9
+# Group 1 (Average Similarity: 1.00):
+# a.py
+# c.py
+# Unique Files (similarity below threshold):
+# b.py
+
+csim-ai info
+# which optional backends (onnxruntime, tokenizers, huggingface_hub, csim, scikit-learn, torch) are available
+```
+
+### `report`
+
+Pairwise similarity report over every `.py` file in `--path`, all
+combinations.
+
+| Flag | Default | Meaning |
+|---|---|---|
+| `--path`, `-p` | required | Directory of `.py` files to compare exhaustively. |
+| `--model-path` | Hugging Face Hub | Directory with `model.onnx`/`tokenizer.json`. Skips the Hub entirely if given. |
+| `--fusion-model` | none | Path to a `fusion_model.joblib`. Skips the Hub entirely if given. |
+| `--use-fusion` | off | Force-download the fusion model from HF Hub if it isn't already cached and no `--fusion-model` is given. |
+
+### `group`
+
+Same comparison as `report`, but groups files into connected components
+by a similarity threshold instead of listing every pair.
+
+Same flags as `report`, plus:
+
+| Flag | Default | Meaning |
+|---|---|---|
+| `--threshold`, `-t` | required | Similarity threshold (0.0-1.0) for grouping. |
+
+### `info`
+
+No comparison -- just reports which optional backends are importable
+(`onnxruntime`/`tokenizers`/`huggingface_hub` from the base install;
+`csim`/`scikit-learn` from `[ast,scorer]`; `torch` from `[export]`).
+Takes an optional `--model-path` to also check a directory for
+`model.onnx`/`tokenizer.json`.
+
+### `setup`
+
+Not part of `pip install .` -- a separate step because the weights
+aren't bundled in the package.
+
+| Flag | Default | Meaning |
+|---|---|---|
+| (none) | -- | Downloads and caches the bi-encoder + fusion model from Hugging Face Hub. |
+| `--export-from CHECKPOINT` | none | Export a local torch checkpoint to ONNX instead of downloading (requires `pip install csim-ai[export]`) -- entirely offline, for your own fine-tuned weights rather than this project's. |
+| `--out` | `./onnx_model` | Output directory for `--export-from`. |
+| `--opset` | `17` | ONNX opset version for `--export-from`. |
+| `--no-verify` | off | Skip the PyTorch-vs-ONNX parity check after `--export-from`. |
+
+For **every** `report`/`group` result, `"similarity index"` is the
+fusion score when available, else `biencoder_cosine`. `csim_ted`/
+`fusion` come back as `None` (and are dropped from the report line) when
+`csim`/`scikit-learn` aren't installed, so a bare `pip install csim-ai`
+(no extras, no `setup`) still gives a usable bi-encoder-only score.
+
+## Python API
+
+```python
+from csim_ai import Scorer
+
+scorer = Scorer()                     # after `csim-ai setup`: full hybrid, cache auto-detected
+scorer = Scorer(use_fusion=True)      # force-downloads the fusion model too if `setup` wasn't run yet
+scorer = Scorer(
+    "path/to/onnx_model",
+    fusion_model_path="path/to/fusion_model.joblib",
+)                                      # fully local, no network
+
+scorer.score(code_a, code_b)
+# {"biencoder_cosine": 0.987, "csim_ted": 0.83, "fusion": 0.978}
+```
+
+`Scorer(model_path=None, fusion_model_path=None, use_fusion=False)`:
+
+- `model_path`: directory with `model.onnx`/`tokenizer.json`. `None`
+  (default) downloads from Hugging Face Hub, cached after first call.
+- `fusion_model_path`: path to a `fusion_model.joblib`. `None` (default)
+  auto-uses a fusion model already cached by a prior `csim-ai setup` or
+  `use_fusion=True` call, without triggering a network request to check.
+- `use_fusion`: if `True` and no `fusion_model_path` is given,
+  force-downloads the fusion model from HF Hub instead of just checking
+  the cache.
+
+`scorer.score(code_a: str, code_b: str) -> dict` returns
+`{"biencoder_cosine": float, "csim_ted": float | None, "fusion": float | None}`
+-- `csim_ted`/`fusion` are `None` when `csim`/`scikit-learn` aren't
+installed or no fusion model is available.
+
+## Layout
+
+```
+src/csim_ai/       inference package -- ONNX bi-encoder + csim TED + GBDT fusion
+tests/             pytest smoke tests for src/csim_ai
+training/          dataset prep, synthetic plagiarism generation, training, eval, export tooling
+docs/
+  REPORT.md        project narrative: problem, methodology, results, limitations
+  DEVELOPMENT.md   phase-by-phase build log: commands, exact numbers, bugs hit and fixed
+```

csim_ai-0.0.1/pyproject.toml

@@ -0,0 +1,65 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "csim-ai"
+version = "0.0.1"
+description = "Neural-augmented Python code plagiarism detection for programming judges."
+readme = "README.md"
+requires-python = ">=3.10"
+license = { text = "MIT" }
+authors = [{ name = "Edsone Eddy", email = "crew0eddy@gmail.com" }]
+keywords = ["plagiarism detection", "code similarity", "bi-encoder", "onnx", "programming judges"]
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "Topic :: Software Development :: Quality Assurance",
+]
+dependencies = ["numpy>=2.0", "onnxruntime>=1.20", "tokenizers>=0.20", "huggingface_hub>=0.25"]
+
+[project.urls]
+Homepage = "https://github.com/edsoneddy/csim-ai"
+"Source Code" = "https://github.com/edsoneddy/csim-ai"
+"Bug Tracker" = "https://github.com/edsoneddy/csim-ai/issues"
+Documentation = "https://github.com/edsoneddy/csim-ai#readme"
+
+[project.scripts]
+csim-ai = "csim_ai.cli:main"
+
+[project.optional-dependencies]
+# NOTE: [ast]'s csim>=3.3.0 pins numpy==1.26.4 exactly on paper, while
+# [torch] needs numpy>=2.0 -- looked like a hard conflict when [torch]
+# first landed (Fase 2), but Fase 4 confirmed both extras installed
+# together actually work at runtime in this venv (numpy 2.5.2): csim's
+# TED score recomputed to match a stored Fase 0 value exactly, then ran
+# at scale (tens of thousands of pairs) with no issues. Still needs a
+# real packaging decision before Fase 5 (drop the exact pin? vendor a
+# compat shim?), but it's not blocking anything today.
+ast = ["csim>=3.3.0"]   # optional TED signal via the csim package (Etapa B feature)
+train = ["pyyaml>=6.0", "csim>=3.3.0", "libcst>=1.0"]
+torch = ["torch>=2.9", "transformers>=5.0", "sentence-transformers>=3.0"]  # zero-shot eval (Fase 2), fine-tuning (Fase 3+)
+scorer = ["scikit-learn>=1.5", "joblib>=1.3"]  # GBDT fusion scorer (Fase 4)
+export = ["onnx>=1.15", "torch>=2.9", "transformers>=5.0"]  # `csim-ai setup --export-from` / training/export_onnx.py -- not needed for inference
+dev = ["pytest>=8.0"]
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/csim_ai"]
+
+[tool.hatch.build.targets.sdist]
+# Allowlist, not a denylist -- hatchling's default sdist walks the whole
+# repo (not just git-tracked files), which pulled in training/'s
+# research artifacts and even .claude/ session state on a first build.
+# Keep the sdist scoped to what's actually needed to build the wheel +
+# the files PyPI expects (README/LICENSE), same as the wheel's own
+# "lean base install" scope -- docs/ and training/ stay on GitHub only.
+include = [
+    "src/csim_ai",
+    "tests",
+    "README.md",
+    "LICENSE",
+    "pyproject.toml",
+]

csim_ai-0.0.1/src/csim_ai/__init__.py

@@ -0,0 +1,85 @@
+"""csim-ai: neural-augmented Python code plagiarism detection for
+programming judges.
+
+Base install (`pip install csim-ai`) gives bi-encoder cosine similarity
+only, via onnxruntime + tokenizers + huggingface_hub (no torch).
+`pip install csim-ai[ast,scorer]` adds the csim TED signal and the GBDT
+fusion of both -- the full hybrid scorer from Fase 4.
+
+Model weights aren't bundled in the package (the ONNX export is
+~500MB) -- `Scorer()` with no arguments downloads the pre-trained
+bi-encoder from Hugging Face Hub (`edson-eddy/csim-ai`) on first use and
+caches it there after; pass `model_path` to use a local export instead
+(from `training/export_onnx.py` or `csim-ai setup --export-from`).
+
+Recommended flow: `pip install csim-ai[ast,scorer]` then `csim-ai
+setup` once (downloads both the bi-encoder and the fusion model) --
+after that, `Scorer()` with no arguments auto-detects the cached fusion
+model and gives the full hybrid score with no further flags/arguments,
+same as the CLI's `report`/`group` without `--use-fusion`. Pass
+`use_fusion=True` to force-download the fusion model on the spot instead
+of requiring `setup` first.
+"""
+from __future__ import annotations
+
+from pathlib import Path
+
+from ._onnx_encoder import OnnxEncoder
+
+__version__ = "0.0.1"
+
+__all__ = ["Scorer", "__version__"]
+
+
+class Scorer:
+    def __init__(
+        self,
+        model_path: str | Path | None = None,
+        fusion_model_path: str | Path | None = None,
+        use_fusion: bool = False,
+    ):
+        if model_path is None:
+            from ._hub import download_model
+
+            model_path = download_model()
+        self._encoder = OnnxEncoder(model_path)
+
+        if fusion_model_path is None:
+            if use_fusion:
+                from ._hub import download_fusion_model
+
+                fusion_model_path = download_fusion_model()
+            else:
+                # Auto-use the fusion model if a prior `csim-ai setup` (or
+                # an earlier use_fusion=True call) already cached it --
+                # matches it being available without forcing a network
+                # request just to check.
+                from ._hub import cached_fusion_model
+
+                fusion_model_path = cached_fusion_model()
+
+        self._fusion = None
+        if fusion_model_path is not None:
+            try:
+                from ._fusion import FusionModel
+
+                self._fusion = FusionModel(fusion_model_path)
+            except ImportError:
+                pass  # scikit-learn not installed -- degrade to bi-encoder-only
+
+    def score(self, code_a: str, code_b: str) -> dict:
+        biencoder_cosine = self._encoder.cosine_similarity(code_a, code_b)
+
+        csim_ted = None
+        try:
+            from ._ted import ted_score
+
+            csim_ted = ted_score(code_a, code_b)
+        except ImportError:
+            pass
+
+        fusion = None
+        if self._fusion is not None and csim_ted is not None:
+            fusion = self._fusion.predict(biencoder_cosine, csim_ted)
+
+        return {"biencoder_cosine": biencoder_cosine, "csim_ted": csim_ted, "fusion": fusion}

csim_ai-0.0.1/src/csim_ai/_export.py

@@ -0,0 +1,117 @@
+"""ONNX export of a fine-tuned bi-encoder checkpoint -- shared by
+`training/export_onnx.py` (dev-side, points at
+training/artifacts/best_checkpoint by default) and `csim-ai setup
+--export-from` (works from any installed checkpoint, no access to this
+repo's training/ tree needed).
+
+Requires the `export` extra (`torch`, `transformers`, `onnx`) -- not a
+base-install dependency, imported lazily here.
+
+See docs/DEVELOPMENT.md, Fase 5, for why: legacy TorchScript exporter (not the
+dynamo-based default in torch >=2.9, which needs `onnxscript`), fp32
+only (int8 dynamic quantization measurably breaks embedding direction;
+fp16 is ~7x slower than fp32 on CPU).
+"""
+from __future__ import annotations
+
+import shutil
+from pathlib import Path
+
+
+def mean_pool(last_hidden, attention_mask):
+    import numpy as np
+
+    mask = attention_mask[..., None].astype(np.float32)
+    summed = (last_hidden * mask).sum(1)
+    counts = mask.sum(1).clip(min=1e-9)
+    pooled = summed / counts
+    return pooled / np.linalg.norm(pooled, axis=-1, keepdims=True)
+
+
+def verify(onnx_path: Path, checkpoint: Path) -> float:
+    import numpy as np
+    import onnxruntime as ort
+    import torch
+    from transformers import AutoModel, AutoTokenizer
+
+    tok = AutoTokenizer.from_pretrained(checkpoint)
+    model = AutoModel.from_pretrained(checkpoint).eval()
+    sess = ort.InferenceSession(str(onnx_path))
+
+    samples = [
+        ["def add(a, b):\n    return a + b\n"],
+        [
+            "def add(a, b):\n    return a + b\n",
+            "import math\ndef f(x):\n    return math.sqrt(x) + 1\n\nprint(f(4))\n",
+        ],
+    ]
+    max_diff = 0.0
+    for batch in samples:
+        inputs = tok(batch, return_tensors="pt", padding=True, truncation=True, max_length=512)
+        with torch.no_grad():
+            torch_out = model(**inputs).last_hidden_state.numpy()
+        torch_pooled = mean_pool(torch_out, inputs["attention_mask"].numpy())
+
+        onnx_out = sess.run(
+            None, {"input_ids": inputs["input_ids"].numpy(), "attention_mask": inputs["attention_mask"].numpy()}
+        )[0]
+        onnx_pooled = mean_pool(onnx_out, inputs["attention_mask"].numpy())
+
+        diff = float(np.abs(torch_pooled - onnx_pooled).max())
+        cos = float((torch_pooled * onnx_pooled).sum(-1).min())
+        max_diff = max(max_diff, diff)
+        print(f"  batch size {len(batch)}: max abs diff={diff:.2e}, min cosine={cos:.6f}")
+
+    return max_diff
+
+
+def export(checkpoint: Path, out_dir: Path, opset: int = 17, verify_after: bool = False) -> Path:
+    import torch
+    from transformers import AutoModel, AutoTokenizer
+
+    class _LastHiddenStateOnly(torch.nn.Module):
+        def __init__(self, model: torch.nn.Module):
+            super().__init__()
+            self.model = model
+
+        def forward(self, input_ids: torch.Tensor, attention_mask: torch.Tensor) -> torch.Tensor:
+            return self.model(input_ids=input_ids, attention_mask=attention_mask).last_hidden_state
+
+    checkpoint = Path(checkpoint)
+    out_dir = Path(out_dir)
+
+    tok = AutoTokenizer.from_pretrained(checkpoint)
+    model = AutoModel.from_pretrained(checkpoint).eval()
+    wrapped = _LastHiddenStateOnly(model)
+
+    sample = tok(["def f(x):\n    return x\n"], return_tensors="pt")
+    out_dir.mkdir(parents=True, exist_ok=True)
+    onnx_path = out_dir / "model.onnx"
+
+    torch.onnx.export(
+        wrapped,
+        (sample["input_ids"], sample["attention_mask"]),
+        str(onnx_path),
+        input_names=["input_ids", "attention_mask"],
+        output_names=["last_hidden_state"],
+        dynamic_axes={
+            "input_ids": {0: "batch", 1: "seq"},
+            "attention_mask": {0: "batch", 1: "seq"},
+            "last_hidden_state": {0: "batch", 1: "seq"},
+        },
+        opset_version=opset,
+        dynamo=False,
+    )
+    print(f"exported: {onnx_path} ({onnx_path.stat().st_size / 1e6:.1f} MB)")
+
+    for name in ("tokenizer.json", "tokenizer_config.json"):
+        shutil.copy(checkpoint / name, out_dir / name)
+    print(f"copied tokenizer files to {out_dir}")
+
+    if verify_after:
+        max_diff = verify(onnx_path, checkpoint)
+        if max_diff > 1e-3:
+            raise SystemExit(f"ONNX/PyTorch parity check failed: max abs diff {max_diff:.2e} > 1e-3")
+        print("parity OK")
+
+    return onnx_path

csim_ai-0.0.1/src/csim_ai/_fusion.py

@@ -0,0 +1,18 @@
+"""GBDT fusion of bi-encoder cosine + csim TED -- requires the `scorer`
+extra (`pip install csim-ai[scorer]`). Imported lazily by `Scorer` so
+the base install doesn't need `scikit-learn`/`joblib`.
+"""
+from __future__ import annotations
+
+from pathlib import Path
+
+
+class FusionModel:
+    def __init__(self, model_path: str | Path):
+        import joblib
+
+        self.model = joblib.load(model_path)
+
+    def predict(self, biencoder_cosine: float, csim_ted: float) -> float:
+        proba = self.model.predict_proba([[biencoder_cosine, csim_ted]])
+        return float(proba[0, 1])

csim_ai-0.0.1/src/csim_ai/_hub.py

@@ -0,0 +1,35 @@
+"""Hugging Face Hub download for the pre-trained bi-encoder + fusion
+model -- lets `Scorer()`/`csim-ai setup --download` work out of the box
+without a local checkpoint or the `[export]` extra (torch/transformers).
+
+`huggingface_hub` is a base dependency: no torch, just small utility
+deps (requests/filelock/tqdm/...), so it doesn't break the "no torch in
+the base install" promise from Fase 5.
+"""
+from __future__ import annotations
+
+DEFAULT_REPO_ID = "edson-eddy/csim-ai"
+
+
+def download_model(repo_id: str = DEFAULT_REPO_ID, revision: str | None = None) -> str:
+    from huggingface_hub import snapshot_download
+
+    return snapshot_download(repo_id=repo_id, revision=revision, allow_patterns=["model.onnx", "tokenizer*.json"])
+
+
+def download_fusion_model(repo_id: str = DEFAULT_REPO_ID, revision: str | None = None) -> str:
+    from huggingface_hub import hf_hub_download
+
+    return hf_hub_download(repo_id=repo_id, filename="fusion_model.joblib", revision=revision)
+
+
+def cached_fusion_model(repo_id: str = DEFAULT_REPO_ID, revision: str | None = None) -> str | None:
+    """Like download_fusion_model, but never hits the network -- returns
+    None if it isn't already cached locally (e.g. from a prior `csim-ai
+    setup` or `use_fusion=True` call), instead of downloading it."""
+    from huggingface_hub import hf_hub_download
+
+    try:
+        return hf_hub_download(repo_id=repo_id, filename="fusion_model.joblib", revision=revision, local_files_only=True)
+    except Exception:
+        return None

csim_ai-0.0.1/src/csim_ai/_onnx_encoder.py

@@ -0,0 +1,42 @@
+"""ONNX bi-encoder wrapper -- onnxruntime + tokenizers only, no
+torch/transformers at runtime. Mean-pool + L2-normalize matches
+`mean_pool()` in `training/train_biencoder.py` exactly, since that's
+what the exported ONNX graph was trained against.
+"""
+from __future__ import annotations
+
+from pathlib import Path
+
+import numpy as np
+import onnxruntime as ort
+from tokenizers import Tokenizer
+
+MAX_LENGTH = 512
+
+
+def _mean_pool(last_hidden: np.ndarray, attention_mask: np.ndarray) -> np.ndarray:
+    mask = attention_mask[..., None].astype(np.float32)
+    summed = (last_hidden * mask).sum(1)
+    counts = mask.sum(1).clip(min=1e-9)
+    pooled = summed / counts
+    return pooled / np.linalg.norm(pooled, axis=-1, keepdims=True)
+
+
+class OnnxEncoder:
+    def __init__(self, model_dir: str | Path):
+        model_dir = Path(model_dir)
+        self.session = ort.InferenceSession(str(model_dir / "model.onnx"))
+        self.tokenizer = Tokenizer.from_file(str(model_dir / "tokenizer.json"))
+        self.tokenizer.enable_padding()
+        self.tokenizer.enable_truncation(max_length=MAX_LENGTH)
+
+    def encode(self, texts: list[str]) -> np.ndarray:
+        encodings = self.tokenizer.encode_batch(texts)
+        input_ids = np.array([e.ids for e in encodings], dtype=np.int64)
+        attention_mask = np.array([e.attention_mask for e in encodings], dtype=np.int64)
+        last_hidden = self.session.run(None, {"input_ids": input_ids, "attention_mask": attention_mask})[0]
+        return _mean_pool(last_hidden, attention_mask)
+
+    def cosine_similarity(self, text_a: str, text_b: str) -> float:
+        embs = self.encode([text_a, text_b])
+        return float(np.dot(embs[0], embs[1]))

csim_ai-0.0.1/src/csim_ai/_ted.py

@@ -0,0 +1,31 @@
+"""csim TED (tree-edit-distance) signal -- requires the `ast` extra
+(`pip install csim-ai[ast]`). Imported lazily by `Scorer` so the base
+install doesn't need `csim`.
+"""
+from __future__ import annotations
+
+import contextlib
+import io
+
+LANG = "python_3_13"
+TED_ALGORITHM = "apted"
+
+
+def ted_score(code_a: str, code_b: str) -> float | None:
+    # csim.utils.preprocess_code does *not* raise on a syntax error --
+    # its ANTLR grammar prints "Syntax error ..." to stderr and returns a
+    # degenerate near-empty tree instead, which would otherwise silently
+    # produce a garbage similarity score. Capture stderr and treat any
+    # output as a failed parse (same fix as training/scorer/build_features.py).
+    from csim.utils import get_similarity_coefficient, preprocess_code
+
+    buf = io.StringIO()
+    try:
+        with contextlib.redirect_stderr(buf):
+            proc_a = preprocess_code("a", code_a, LANG)
+            proc_b = preprocess_code("b", code_b, LANG)
+            if buf.getvalue():
+                return None
+            return get_similarity_coefficient(proc_a, proc_b, TED_ALGORITHM)
+    except Exception:
+        return None

csim_ai-0.0.1/src/csim_ai/cli.py

@@ -0,0 +1,186 @@
+"""`csim-ai {report,group,info,setup}` -- CLI entry point (see
+[project.scripts] in pyproject.toml). Mirrors csim's CLI shape (a
+single command, an action positional, `--path` pointing at a directory
+of source files compared exhaustively) rather than a two-file-only
+interface, for consistency with the predecessor tool.
+"""
+from __future__ import annotations
+
+import argparse
+import itertools
+import sys
+from pathlib import Path
+
+from . import Scorer
+
+
+def _primary_score(result: dict) -> float | None:
+    return result["fusion"] if result["fusion"] is not None else result["biencoder_cosine"]
+
+
+def _iter_py_files(path: Path) -> list[Path]:
+    return sorted(p for p in path.iterdir() if p.suffix == ".py")
+
+
+def _score_all_pairs(files: list[Path], scorer: Scorer) -> dict[tuple[Path, Path], dict]:
+    results = {}
+    for a, b in itertools.combinations(files, 2):
+        code_a = a.read_text(encoding="utf-8", errors="ignore")
+        code_b = b.read_text(encoding="utf-8", errors="ignore")
+        results[(a, b)] = scorer.score(code_a, code_b)
+    return results
+
+
+def cmd_report(args: argparse.Namespace) -> None:
+    scorer = Scorer(args.model_path, fusion_model_path=args.fusion_model, use_fusion=args.use_fusion)
+    files = _iter_py_files(args.path)
+    results = _score_all_pairs(files, scorer)
+    for (a, b), r in results.items():
+        extra = f" (biencoder_cosine={r['biencoder_cosine']:.4f}"
+        if r["csim_ted"] is not None:
+            extra += f", csim_ted={r['csim_ted']:.4f}"
+        if r["fusion"] is not None:
+            extra += f", fusion={r['fusion']:.4f}"
+        extra += ")"
+        print(f"{b} is similar to {a} with similarity index: {_primary_score(r):.4f}{extra}")
+
+
+def cmd_group(args: argparse.Namespace) -> None:
+    scorer = Scorer(args.model_path, fusion_model_path=args.fusion_model, use_fusion=args.use_fusion)
+    files = _iter_py_files(args.path)
+    results = _score_all_pairs(files, scorer)
+
+    parent = {f: f for f in files}
+
+    def find(f: Path) -> Path:
+        while parent[f] != f:
+            parent[f] = parent[parent[f]]
+            f = parent[f]
+        return f
+
+    def union(a: Path, b: Path) -> None:
+        ra, rb = find(a), find(b)
+        if ra != rb:
+            parent[ra] = rb
+
+    for (a, b), r in results.items():
+        if _primary_score(r) >= args.threshold:
+            union(a, b)
+
+    groups: dict[Path, list[Path]] = {}
+    for f in files:
+        groups.setdefault(find(f), []).append(f)
+
+    print(f"Threshold: {args.threshold}")
+    print(f"Total files processed: {len(files)}")
+
+    group_num = 0
+    unique_files = []
+    for members in groups.values():
+        if len(members) == 1:
+            unique_files.append(members[0])
+            continue
+        group_num += 1
+        pair_scores = [
+            _primary_score(results[(a, b)] if (a, b) in results else results[(b, a)])
+            for a, b in itertools.combinations(members, 2)
+        ]
+        avg = sum(pair_scores) / len(pair_scores)
+        print(f"Group {group_num} (Average Similarity: {avg:.2f}):")
+        for f in members:
+            print(f)
+
+    if unique_files:
+        print("Unique Files (similarity below threshold):")
+        for f in unique_files:
+            print(f)
+
+
+def cmd_info(args: argparse.Namespace) -> None:
+    print("csim-ai backends")
+    print()
+
+    def check(label: str, module: str) -> None:
+        try:
+            mod = __import__(module)
+            version = getattr(mod, "__version__", "?")
+            print(f"  {label:<24} available (v{version})")
+        except ImportError:
+            print(f"  {label:<24} not installed")
+
+    check("onnxruntime (base)", "onnxruntime")
+    check("tokenizers (base)", "tokenizers")
+    check("huggingface_hub (base)", "huggingface_hub")
+    check("csim (ast extra)", "csim")
+    check("scikit-learn (scorer extra)", "sklearn")
+    check("torch (export extra)", "torch")
+
+    if args.model_path:
+        model_path = Path(args.model_path)
+        onnx_ok = (model_path / "model.onnx").exists()
+        tok_ok = (model_path / "tokenizer.json").exists()
+        print()
+        print(f"  model-path: {model_path}")
+        print(f"    model.onnx: {'found' if onnx_ok else 'MISSING'}")
+        print(f"    tokenizer.json: {'found' if tok_ok else 'MISSING'}")
+
+
+def cmd_setup(args: argparse.Namespace) -> None:
+    if args.export_from:
+        try:
+            from ._export import export
+        except ImportError:
+            print("The `export` extra is required: pip install csim-ai[export]", file=sys.stderr)
+            raise SystemExit(1)
+
+        export(args.export_from, args.out, opset=args.opset, verify_after=not args.no_verify)
+        return
+
+    # Default action: download the pre-trained weights from HF Hub, so
+    # `pip install csim-ai && csim-ai setup` is all that's needed before
+    # `report`/`group` work with the full hybrid score, no flags needed
+    # after that (Scorer() auto-detects the cache -- see __init__.py).
+    from ._hub import download_fusion_model, download_model
+
+    model_path = download_model()
+    print(f"bi-encoder cached at: {model_path}")
+    fusion_path = download_fusion_model()
+    print(f"fusion model cached at: {fusion_path}")
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description="Score Python files for plagiarism similarity.")
+    sub = parser.add_subparsers(dest="action", required=True)
+
+    def add_path_args(p: argparse.ArgumentParser) -> None:
+        p.add_argument("--path", "-p", type=Path, required=True, help="Directory of .py files to compare exhaustively.")
+        p.add_argument("--model-path", default=None, help="Directory with model.onnx/tokenizer.json. Default: download from Hugging Face Hub.")
+        p.add_argument("--fusion-model", default=None, help="Path to a fusion_model.joblib (requires the [ast,scorer] extras).")
+        p.add_argument("--use-fusion", action="store_true", help="Use the fusion score, downloading the fusion model from Hugging Face Hub if --fusion-model isn't given.")
+
+    report_p = sub.add_parser("report", help="Pairwise similarity report over a directory.")
+    add_path_args(report_p)
+    report_p.set_defaults(func=cmd_report)
+
+    group_p = sub.add_parser("group", help="Group files by similarity threshold.")
+    add_path_args(group_p)
+    group_p.add_argument("--threshold", "-t", type=float, required=True, help="Similarity threshold (0.0-1.0) for grouping.")
+    group_p.set_defaults(func=cmd_group)
+
+    info_p = sub.add_parser("info", help="Show which optional backends (onnxruntime, csim, scikit-learn, torch) are available.")
+    info_p.add_argument("--model-path", default=None, help="Optionally check a model directory for model.onnx/tokenizer.json.")
+    info_p.set_defaults(func=cmd_info)
+
+    setup_p = sub.add_parser("setup", help="Download the pre-trained weights from Hugging Face Hub (default), or export a local checkpoint to ONNX instead.")
+    setup_p.add_argument("--export-from", default=None, help="Local torch checkpoint directory to export to ONNX instead of downloading (requires the [export] extra). No network access -- see docs/DEVELOPMENT.md, Fase 5.")
+    setup_p.add_argument("--out", default="onnx_model", help="Output directory for --export-from (default: ./onnx_model).")
+    setup_p.add_argument("--opset", type=int, default=17)
+    setup_p.add_argument("--no-verify", action="store_true", help="Skip the PyTorch-vs-ONNX parity check after --export-from.")
+    setup_p.set_defaults(func=cmd_setup)
+
+    args = parser.parse_args()
+    args.func(args)
+
+
+if __name__ == "__main__":
+    main()

csim_ai-0.0.1/tests/test_hub_download.py

@@ -0,0 +1,23 @@
+"""Tests the Hugging Face Hub auto-download path (`Scorer()` with no
+model_path). Requires network access to huggingface.co -- unlike
+test_inference.py, this doesn't skip based on local artifacts, since the
+whole point is to not need any.
+"""
+from __future__ import annotations
+
+from csim_ai import Scorer
+
+DUPLICATE_A = "def add(a, b):\n    return a + b\n"
+DUPLICATE_B = "def add(x, y):\n    # add two numbers\n    return x + y\n"
+UNRELATED_A = "def add(a, b):\n    return a + b\n"
+UNRELATED_B = "import sys\nfor line in sys.stdin:\n    print(line.strip()[::-1])\n"
+
+
+def test_auto_download_full_scoring():
+    scorer = Scorer(use_fusion=True)
+    dup = scorer.score(DUPLICATE_A, DUPLICATE_B)
+    unrelated = scorer.score(UNRELATED_A, UNRELATED_B)
+
+    assert dup["fusion"] > unrelated["fusion"]
+    assert dup["fusion"] > 0.5
+    assert unrelated["fusion"] < 0.5

csim_ai-0.0.1/tests/test_inference.py

@@ -0,0 +1,51 @@
+"""Fase 5 inference smoke tests. Skips when the gitignored local
+artifacts (ONNX export, fusion model) aren't present -- they're
+regenerated via `training/export_onnx.py` and
+`training/scorer/train_fusion.py`, not part of a fresh clone.
+"""
+from __future__ import annotations
+
+from pathlib import Path
+
+import pytest
+
+from csim_ai import Scorer
+
+ROOT = Path(__file__).parent.parent
+ONNX_MODEL_DIR = ROOT / "training" / "artifacts" / "onnx_model"
+FUSION_MODEL_PATH = ROOT / "training" / "scorer" / "artifacts" / "fusion_model_v1.joblib"
+
+pytestmark = pytest.mark.skipif(
+    not (ONNX_MODEL_DIR / "model.onnx").exists(),
+    reason="ONNX model not exported locally -- run training/export_onnx.py first",
+)
+
+DUPLICATE_A = "def add(a, b):\n    return a + b\n"
+DUPLICATE_B = "def add(x, y):\n    # add two numbers\n    return x + y\n"
+UNRELATED_A = "def add(a, b):\n    return a + b\n"
+UNRELATED_B = "import sys\nfor line in sys.stdin:\n    print(line.strip()[::-1])\n"
+
+
+def test_biencoder_only_scoring():
+    # fusion_model_path=None, use_fusion=False (default): Scorer still
+    # auto-uses a fusion model if one happens to already be cached from
+    # HF Hub (e.g. a prior `csim-ai setup` or use_fusion=True call) --
+    # so this only asserts what's guaranteed regardless of cache state.
+    scorer = Scorer(ONNX_MODEL_DIR)
+    dup = scorer.score(DUPLICATE_A, DUPLICATE_B)
+    unrelated = scorer.score(UNRELATED_A, UNRELATED_B)
+
+    assert dup["biencoder_cosine"] > unrelated["biencoder_cosine"]
+    assert dup["biencoder_cosine"] > 0.8
+
+
+@pytest.mark.skipif(not FUSION_MODEL_PATH.exists(), reason="fusion model not trained locally")
+def test_fusion_scoring():
+    scorer = Scorer(ONNX_MODEL_DIR, fusion_model_path=FUSION_MODEL_PATH)
+    dup = scorer.score(DUPLICATE_A, DUPLICATE_B)
+    unrelated = scorer.score(UNRELATED_A, UNRELATED_B)
+
+    assert dup["fusion"] is not None
+    assert dup["fusion"] > unrelated["fusion"]
+    assert dup["fusion"] > 0.5
+    assert unrelated["fusion"] < 0.5