sieve-referee 1.0.0__tar.gz

sieve_referee-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Kai IWASAKI
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

sieve_referee-1.0.0/PKG-INFO

@@ -0,0 +1,207 @@
+Metadata-Version: 2.4
+Name: sieve-referee
+Version: 1.0.0
+Summary: Deterministic, zero-dependency plagiarism and text structure screening engine (Sieve-Core plugin)
+Author: Kai IWASAKI
+License: MIT
+Project-URL: Homepage, https://github.com/neguseatama/sieve-referee
+Project-URL: Repository, https://github.com/neguseatama/sieve-referee.git
+Project-URL: Related Project (Sieve-Core), https://github.com/neguseatama/sieve-core
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# Sieve Referee
+
+English | [日本語](README.ja.md)
+
+> **"Should we really use probabilistic AI models to evaluate text generated by other probabilistic AI models?"**
+> A fully local, zero-dependency, and deterministic (100% reproducible) text structure and plagiarism screening engine. Developed as an extended application of [Sieve-Core](https://github.com/neguseatama/sieve-core).
+
+---
+
+## 💡 Concept & Philosophy
+
+`Sieve-Core` is the core engine that applies the design philosophy of "Sieve-AI"—a "sieve of set theory" inspired by the Sieve of Eratosthenes—to the more general domain of text and data processing. `Sieve Referee`, designed by adding a "sieve of combinatorics" to this `Sieve-Core`, is a zero-dependency Python library that extracts reliable information from noise based on corroboration from multiple independent sources.
+
+When evaluating text that may have been generated by probabilistic models (such as LLMs), relying on another probabilistic model to calculate a similarity score makes it difficult to mathematically trace *why* that score was produced.
+
+`Sieve Referee` was built to address this problem: to ensure every judgment is mathematically traceable, 100% reproducible, and fully explainable.
+
+- **Deterministic Approach**: Evaluation is based on a 16-state Look-Up Table (LUT, of which 12 states are reachable). Identical inputs always produce identical outputs no matter how many times it is run.
+- **Explainable by Design**: Every result carries a `reason` string and a 4-bit hypothesis `mask`, so the basis of a judgment can always be inspected.
+- **Careful Classification to Avoid False Accusations**: A shared template or boilerplate format is classified separately as `COINCIDENTAL_FORMAT_MATCH` (`1100` / `YELLOW`), rather than being treated the same as an actual content match.
+- **Indirect ("Telephone Game") Chain Safety**: A pair with no direct similarity evidence (e.g. `A <-> C`, where `A <-> B` and `B <-> C` are both flagged) is never silently upgraded to a positive match solely based on cluster membership. Such pairs are reported separately on the dashboard as an "indirect chain (telephone game) warning," and the direct comparison result itself is left unmodified.
+
+---
+
+## 🎯 Primary Use Cases
+
+Built with zero external dependencies and in-memory execution, `sieve_referee` is well suited as an embedded, privacy-focused screening engine.
+
+### 1. Education & Learning Management Systems (EdTech / LMS)
+Screen student submissions without uploading personal data or copyrighted text to a third-party cloud service. Common assignment templates are classified as `YELLOW`; unattributed copying or paraphrasing is flagged as `RED`.
+
+### 2. Digital Publishing & CMS
+Detect rewritten or recycled content submitted by freelance writers at draft-save time, without incurring LLM API costs for every check.
+
+### 3. LLM Training Data Deduplication
+Use as a deterministic pre-processing stage to remove near-duplicate or paraphrased records from a fine-tuning dataset, without GPU acceleration.
+
+### 4. Legal, Contract & RFP Review
+Separate boilerplate clauses (`COINCIDENTAL_FORMAT_MATCH`) from intentionally altered wording across contract revisions or vendor proposals.
+
+### 5. CI/CD Documentation Auditing
+Run in a GitHub Actions workflow or pre-commit hook to catch unauthorized copy-pasting across internal wikis or specification documents.
+
+---
+
+## ⚡ Key Technical Advantages
+
+1. **Near-zero cold start** — no PyTorch, no Transformers, no model loading; well suited to AWS Lambda or other serverless environments.
+2. **Zero per-call cost** — everything runs on local CPU.
+3. **Zero maintenance burden** — zero third-party dependencies means no exposure to upstream CVEs or breaking API changes.
+
+---
+
+## 📦 Installation
+
+```bash
+pip install git+https://github.com/neguseatama/sieve-referee.git
+```
+
+or, from a local clone:
+
+```bash
+git clone https://github.com/neguseatama/sieve-referee.git
+cd sieve-referee
+pip install .
+```
+
+---
+
+## 🐍 Usage (Python Library)
+
+```python
+from sieve_referee import batch_evaluate, EvaluationResult
+
+# 1. Prepare input documents in memory
+documents = {
+    "doc_a.txt": "Quantum computing relies on superposition and entanglement to achieve fast factorization.",
+    "doc_b.txt": "To achieve fast factorization, quantum computing makes use of entanglement and superposition."
+}
+
+# 2. Run the screening
+results: list[EvaluationResult] = batch_evaluate(documents)
+
+# 3. Read the results
+for res in results:
+    print(f"Pair:    {res.item_id} <-> {res.matched_peer_id}")
+    print(f"Signal:  {res.evaluation_signal.value}")  # -> RED
+    print(f"Mask:    {res.mask}")                      # -> 0010
+    print(f"Pattern: {res.pattern_name}")               # -> UNFORMATTED_CONTENT_COPY
+    print(f"Reason:  {res.reason}")
+    print("-" * 40)
+```
+
+### `EvaluationResult` fields
+
+| Property | Type | Description |
+|---|---|---|
+| `item_id` | `str` | Primary document identifier |
+| `matched_peer_id` | `str` | Comparison peer document identifier |
+| `evaluation_signal` | `SignalColor` | Result signal (`.value` returns `"GREEN"`, `"YELLOW"`, or `"RED"`) |
+| `mask` | `str` | 4-bit hypothesis mask, e.g. `"1010"` |
+| `pattern_name` | `str` | LUT pattern name, e.g. `"PARAPHRASE_DETECTED"` |
+| `reason` | `str` | Human-readable explanation of the judgment |
+
+---
+
+## 💻 Usage (CLI Tool)
+
+```bash
+# Basic directory scan (writes reports/sieve_dashboard.html)
+sieve-referee ./target_documents
+
+# Single self-contained HTML report (no diff_htmls/ subfolder)
+sieve-referee ./target_documents --standalone --output-dir ./reports
+
+# JSON output for automated pipelines, with a non-zero exit code on RED
+sieve-referee ./target_documents --json --fail-on-risk
+
+# Multi-core evaluation
+sieve-referee ./target_documents -j 4
+```
+
+| Option | Type | Description |
+|---|---|---|
+| `target_dir` | positional | Directory containing the `.txt` files to screen |
+| `--output-dir` | string | Report output directory (default: `reports`) |
+| `--standalone` | flag | Emit a single self-contained HTML report instead of a report + `diff_htmls/` folder |
+| `--json` | flag | Print results as JSON to stdout |
+| `--json-file` | string | Save results as JSON to the given path |
+| `--fail-on-risk` | flag | Exit with status code `1` if any `RED` signal is found |
+| `-j`, `--jobs` | int | Number of worker processes for evaluation (default: CPU count) |
+
+---
+
+## 📊 16-State Look-Up Table (LUT)
+
+Four independent hypotheses (`H1`–`H4`) are combined into a 4-bit mask, which maps deterministically to a signal, pattern name, and reason.
+
+| Mask | Signal | Pattern Name | Description |
+|:---:|:---:|:---|:---|
+| `1000` | 🟢 GREEN | `STANDARD_STRUCTURE_UNIQUE` | Independent document, no structural or content overlap detected. |
+| `1100` | 🟡 YELLOW | `COINCIDENTAL_FORMAT_MATCH` | Shared format/template only; content itself is independent. |
+| `1010` | 🔴 RED | `PARAPHRASE_DETECTED` | Content overlap despite a different structure (paraphrasing). |
+| `1011` | 🔴 RED | `CLUSTER_PARAPHRASE_GROUP` | Paraphrased reuse confirmed within a 3+ document cluster. |
+| `1111` | 🔴 RED | `CLUSTER_EXACT_COPY` | Exact/templated copy confirmed within a 3+ document cluster. |
+
+(12 of the 16 possible mask states are reachable in practice; the remaining 4 states, where cluster membership is set without any direct similarity evidence, are structurally unreachable by design.)
+
+> **Note on `H4` (cluster membership) and indirect chains**
+> `H4` is only set when a pair has *both* direct similarity evidence (`H2` or `H3`) *and* cluster membership of size ≥ 3. A pair with no direct evidence is never upgraded through cluster membership alone — such "telephone game" chains are reported separately as an indirect-chain warning on the dashboard, and the direct pair's own result is left unmodified.
+
+---
+
+## ⚠️ Known Limitation: Threshold Calibration Is Japanese-Specific
+
+All four thresholds (`H1 = 0.35`, `H2 = 0.80`, `H3 = 0.45`, cluster size `≥ 3`)
+were calibrated using real-data testing on **Japanese** text (roughly 1,300+
+pairs across unrelated documents, paraphrase pairs, and formal academic
+prose).
+
+The `H1` density metric in particular (`len(set(text)) / len(text)`) behaves
+very differently across scripts: Japanese text draws from a large set of
+kanji, so unique-character ratios tend to be high, while English and other
+alphabetic-script text reuses a small ~26-letter alphabet and will
+systematically score lower on this metric, independent of how generic or
+templated the content actually is. As a result, non-Japanese text may be
+under- or over-classified against the current defaults.
+
+Multi-encoding file reading (UTF-8, Shift_JIS, CP932, EUC-JP) and NFKC
+normalization work for any text regardless of language. What has **not**
+been separately validated is the semantic quality of `H2`/`H3` splitting,
+or the appropriateness of the default thresholds, for languages other than
+Japanese. If you use this on non-Japanese corpora, re-run the false-positive
+check described in [ARCHITECTURE.md](docs/ARCHITECTURE.md) against your own
+data before trusting the default thresholds.
+
+Additionally, the `reason` string returned by every `EvaluationResult` is
+currently hard-coded in Japanese regardless of the input document's
+language, as shown in the example output above. There is no English (or
+other-language) localization of these messages yet.
+
+---
+
+## 📚 Documentation
+
+- 🔌 [API & Integration Guide](docs/INTEGRATION.md) — using `sieve_referee` as a Python library or FastAPI microservice
+- 🚀 [Deployment Guide](docs/DEPLOYMENT.md) — Docker and Terraform for Google Cloud Run
+- 🔐 [CI/CD & Security Guide](docs/CI_CD_SETUP.md) — keyless GCP deployment via Workload Identity Federation
+- 🏛️ [Architecture Specification](docs/ARCHITECTURE.md) — the mathematical model and safety invariants in detail
+
+## 📄 License
+
+MIT License. See [LICENSE](LICENSE).

sieve_referee-1.0.0/README.md

@@ -0,0 +1,193 @@
+# Sieve Referee
+
+English | [日本語](README.ja.md)
+
+> **"Should we really use probabilistic AI models to evaluate text generated by other probabilistic AI models?"**
+> A fully local, zero-dependency, and deterministic (100% reproducible) text structure and plagiarism screening engine. Developed as an extended application of [Sieve-Core](https://github.com/neguseatama/sieve-core).
+
+---
+
+## 💡 Concept & Philosophy
+
+`Sieve-Core` is the core engine that applies the design philosophy of "Sieve-AI"—a "sieve of set theory" inspired by the Sieve of Eratosthenes—to the more general domain of text and data processing. `Sieve Referee`, designed by adding a "sieve of combinatorics" to this `Sieve-Core`, is a zero-dependency Python library that extracts reliable information from noise based on corroboration from multiple independent sources.
+
+When evaluating text that may have been generated by probabilistic models (such as LLMs), relying on another probabilistic model to calculate a similarity score makes it difficult to mathematically trace *why* that score was produced.
+
+`Sieve Referee` was built to address this problem: to ensure every judgment is mathematically traceable, 100% reproducible, and fully explainable.
+
+- **Deterministic Approach**: Evaluation is based on a 16-state Look-Up Table (LUT, of which 12 states are reachable). Identical inputs always produce identical outputs no matter how many times it is run.
+- **Explainable by Design**: Every result carries a `reason` string and a 4-bit hypothesis `mask`, so the basis of a judgment can always be inspected.
+- **Careful Classification to Avoid False Accusations**: A shared template or boilerplate format is classified separately as `COINCIDENTAL_FORMAT_MATCH` (`1100` / `YELLOW`), rather than being treated the same as an actual content match.
+- **Indirect ("Telephone Game") Chain Safety**: A pair with no direct similarity evidence (e.g. `A <-> C`, where `A <-> B` and `B <-> C` are both flagged) is never silently upgraded to a positive match solely based on cluster membership. Such pairs are reported separately on the dashboard as an "indirect chain (telephone game) warning," and the direct comparison result itself is left unmodified.
+
+---
+
+## 🎯 Primary Use Cases
+
+Built with zero external dependencies and in-memory execution, `sieve_referee` is well suited as an embedded, privacy-focused screening engine.
+
+### 1. Education & Learning Management Systems (EdTech / LMS)
+Screen student submissions without uploading personal data or copyrighted text to a third-party cloud service. Common assignment templates are classified as `YELLOW`; unattributed copying or paraphrasing is flagged as `RED`.
+
+### 2. Digital Publishing & CMS
+Detect rewritten or recycled content submitted by freelance writers at draft-save time, without incurring LLM API costs for every check.
+
+### 3. LLM Training Data Deduplication
+Use as a deterministic pre-processing stage to remove near-duplicate or paraphrased records from a fine-tuning dataset, without GPU acceleration.
+
+### 4. Legal, Contract & RFP Review
+Separate boilerplate clauses (`COINCIDENTAL_FORMAT_MATCH`) from intentionally altered wording across contract revisions or vendor proposals.
+
+### 5. CI/CD Documentation Auditing
+Run in a GitHub Actions workflow or pre-commit hook to catch unauthorized copy-pasting across internal wikis or specification documents.
+
+---
+
+## ⚡ Key Technical Advantages
+
+1. **Near-zero cold start** — no PyTorch, no Transformers, no model loading; well suited to AWS Lambda or other serverless environments.
+2. **Zero per-call cost** — everything runs on local CPU.
+3. **Zero maintenance burden** — zero third-party dependencies means no exposure to upstream CVEs or breaking API changes.
+
+---
+
+## 📦 Installation
+
+```bash
+pip install git+https://github.com/neguseatama/sieve-referee.git
+```
+
+or, from a local clone:
+
+```bash
+git clone https://github.com/neguseatama/sieve-referee.git
+cd sieve-referee
+pip install .
+```
+
+---
+
+## 🐍 Usage (Python Library)
+
+```python
+from sieve_referee import batch_evaluate, EvaluationResult
+
+# 1. Prepare input documents in memory
+documents = {
+    "doc_a.txt": "Quantum computing relies on superposition and entanglement to achieve fast factorization.",
+    "doc_b.txt": "To achieve fast factorization, quantum computing makes use of entanglement and superposition."
+}
+
+# 2. Run the screening
+results: list[EvaluationResult] = batch_evaluate(documents)
+
+# 3. Read the results
+for res in results:
+    print(f"Pair:    {res.item_id} <-> {res.matched_peer_id}")
+    print(f"Signal:  {res.evaluation_signal.value}")  # -> RED
+    print(f"Mask:    {res.mask}")                      # -> 0010
+    print(f"Pattern: {res.pattern_name}")               # -> UNFORMATTED_CONTENT_COPY
+    print(f"Reason:  {res.reason}")
+    print("-" * 40)
+```
+
+### `EvaluationResult` fields
+
+| Property | Type | Description |
+|---|---|---|
+| `item_id` | `str` | Primary document identifier |
+| `matched_peer_id` | `str` | Comparison peer document identifier |
+| `evaluation_signal` | `SignalColor` | Result signal (`.value` returns `"GREEN"`, `"YELLOW"`, or `"RED"`) |
+| `mask` | `str` | 4-bit hypothesis mask, e.g. `"1010"` |
+| `pattern_name` | `str` | LUT pattern name, e.g. `"PARAPHRASE_DETECTED"` |
+| `reason` | `str` | Human-readable explanation of the judgment |
+
+---
+
+## 💻 Usage (CLI Tool)
+
+```bash
+# Basic directory scan (writes reports/sieve_dashboard.html)
+sieve-referee ./target_documents
+
+# Single self-contained HTML report (no diff_htmls/ subfolder)
+sieve-referee ./target_documents --standalone --output-dir ./reports
+
+# JSON output for automated pipelines, with a non-zero exit code on RED
+sieve-referee ./target_documents --json --fail-on-risk
+
+# Multi-core evaluation
+sieve-referee ./target_documents -j 4
+```
+
+| Option | Type | Description |
+|---|---|---|
+| `target_dir` | positional | Directory containing the `.txt` files to screen |
+| `--output-dir` | string | Report output directory (default: `reports`) |
+| `--standalone` | flag | Emit a single self-contained HTML report instead of a report + `diff_htmls/` folder |
+| `--json` | flag | Print results as JSON to stdout |
+| `--json-file` | string | Save results as JSON to the given path |
+| `--fail-on-risk` | flag | Exit with status code `1` if any `RED` signal is found |
+| `-j`, `--jobs` | int | Number of worker processes for evaluation (default: CPU count) |
+
+---
+
+## 📊 16-State Look-Up Table (LUT)
+
+Four independent hypotheses (`H1`–`H4`) are combined into a 4-bit mask, which maps deterministically to a signal, pattern name, and reason.
+
+| Mask | Signal | Pattern Name | Description |
+|:---:|:---:|:---|:---|
+| `1000` | 🟢 GREEN | `STANDARD_STRUCTURE_UNIQUE` | Independent document, no structural or content overlap detected. |
+| `1100` | 🟡 YELLOW | `COINCIDENTAL_FORMAT_MATCH` | Shared format/template only; content itself is independent. |
+| `1010` | 🔴 RED | `PARAPHRASE_DETECTED` | Content overlap despite a different structure (paraphrasing). |
+| `1011` | 🔴 RED | `CLUSTER_PARAPHRASE_GROUP` | Paraphrased reuse confirmed within a 3+ document cluster. |
+| `1111` | 🔴 RED | `CLUSTER_EXACT_COPY` | Exact/templated copy confirmed within a 3+ document cluster. |
+
+(12 of the 16 possible mask states are reachable in practice; the remaining 4 states, where cluster membership is set without any direct similarity evidence, are structurally unreachable by design.)
+
+> **Note on `H4` (cluster membership) and indirect chains**
+> `H4` is only set when a pair has *both* direct similarity evidence (`H2` or `H3`) *and* cluster membership of size ≥ 3. A pair with no direct evidence is never upgraded through cluster membership alone — such "telephone game" chains are reported separately as an indirect-chain warning on the dashboard, and the direct pair's own result is left unmodified.
+
+---
+
+## ⚠️ Known Limitation: Threshold Calibration Is Japanese-Specific
+
+All four thresholds (`H1 = 0.35`, `H2 = 0.80`, `H3 = 0.45`, cluster size `≥ 3`)
+were calibrated using real-data testing on **Japanese** text (roughly 1,300+
+pairs across unrelated documents, paraphrase pairs, and formal academic
+prose).
+
+The `H1` density metric in particular (`len(set(text)) / len(text)`) behaves
+very differently across scripts: Japanese text draws from a large set of
+kanji, so unique-character ratios tend to be high, while English and other
+alphabetic-script text reuses a small ~26-letter alphabet and will
+systematically score lower on this metric, independent of how generic or
+templated the content actually is. As a result, non-Japanese text may be
+under- or over-classified against the current defaults.
+
+Multi-encoding file reading (UTF-8, Shift_JIS, CP932, EUC-JP) and NFKC
+normalization work for any text regardless of language. What has **not**
+been separately validated is the semantic quality of `H2`/`H3` splitting,
+or the appropriateness of the default thresholds, for languages other than
+Japanese. If you use this on non-Japanese corpora, re-run the false-positive
+check described in [ARCHITECTURE.md](docs/ARCHITECTURE.md) against your own
+data before trusting the default thresholds.
+
+Additionally, the `reason` string returned by every `EvaluationResult` is
+currently hard-coded in Japanese regardless of the input document's
+language, as shown in the example output above. There is no English (or
+other-language) localization of these messages yet.
+
+---
+
+## 📚 Documentation
+
+- 🔌 [API & Integration Guide](docs/INTEGRATION.md) — using `sieve_referee` as a Python library or FastAPI microservice
+- 🚀 [Deployment Guide](docs/DEPLOYMENT.md) — Docker and Terraform for Google Cloud Run
+- 🔐 [CI/CD & Security Guide](docs/CI_CD_SETUP.md) — keyless GCP deployment via Workload Identity Federation
+- 🏛️ [Architecture Specification](docs/ARCHITECTURE.md) — the mathematical model and safety invariants in detail
+
+## 📄 License
+
+MIT License. See [LICENSE](LICENSE).

sieve_referee-1.0.0/pyproject.toml

@@ -0,0 +1,25 @@
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "sieve-referee"
+version = "1.0.0"
+description = "Deterministic, zero-dependency plagiarism and text structure screening engine (Sieve-Core plugin)"
+readme = "README.md"
+requires-python = ">=3.10"
+license = { text = "MIT" }
+authors = [{ name = "Kai IWASAKI" }]
+dependencies = []
+
+[project.urls]
+Homepage = "https://github.com/neguseatama/sieve-referee"
+Repository = "https://github.com/neguseatama/sieve-referee.git"
+"Related Project (Sieve-Core)" = "https://github.com/neguseatama/sieve-core"
+
+[project.scripts]
+sieve-referee = "sieve_referee.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["."]
+include = ["sieve_referee*"]

sieve_referee-1.0.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

sieve_referee-1.0.0/sieve_referee/__init__.py

@@ -0,0 +1,11 @@
+"""
+sieve_referee
+~~~~~~~~~~~~~
+Deterministic, zero-dependency plagiarism and text structure screening engine.
+"""
+from .referee import SieveReferee
+from .evaluator import batch_evaluate
+from .models import EvaluationResult, SignalColor
+
+__version__ = "1.0.0"
+__all__ = ["SieveReferee", "batch_evaluate", "EvaluationResult", "SignalColor", "__version__"]

sieve_referee-1.0.0/sieve_referee/blocker.py

@@ -0,0 +1,39 @@
+from typing import Dict, Set, Tuple
+from collections import defaultdict
+from sieve_referee.tokenizer import tokenize
+
+
+class DeterministicBlocker:
+    def __init__(self, ngram_size: int = 3, min_shared_ngrams: int = 1):
+        self.ngram_size = ngram_size
+        self.min_shared_ngrams = min_shared_ngrams
+
+    def _extract_ngrams(self, text: str) -> Set[str]:
+        tokens = [t.raw_text for t in tokenize(text)]
+        if len(tokens) < self.ngram_size:
+            return set(tokens) if tokens else {text}
+        return {"".join(tokens[i : i + self.ngram_size]) for i in range(len(tokens) - self.ngram_size + 1)}
+
+    def get_candidate_pairs(self, docs: Dict[str, str]) -> Set[Tuple[str, str]]:
+        doc_ids = sorted(docs.keys())
+        if len(doc_ids) < 10:
+            candidates = set()
+            for i in range(len(doc_ids)):
+                for j in range(i + 1, len(doc_ids)):
+                    candidates.add((doc_ids[i], doc_ids[j]))
+            return candidates
+        inverted_index: Dict[str, Set[str]] = defaultdict(set)
+        for doc_id in doc_ids:
+            ngrams = self._extract_ngrams(docs[doc_id])
+            for ngram in ngrams:
+                inverted_index[ngram].add(doc_id)
+        shared_counts = defaultdict(int)
+        for ngram, sharing_docs in inverted_index.items():
+            if len(sharing_docs) < 2:
+                continue
+            sorted_sharing = sorted(sharing_docs)
+            for i in range(len(sorted_sharing)):
+                for j in range(i + 1, len(sorted_sharing)):
+                    pair = (sorted_sharing[i], sorted_sharing[j])
+                    shared_counts[pair] += 1
+        return {pair for pair, count in shared_counts.items() if count >= self.min_shared_ngrams}

sieve_referee-1.0.0/sieve_referee/chain_detector.py

@@ -0,0 +1,29 @@
+from typing import Dict, List, Set, Tuple
+from .models import EvaluationResult
+
+
+def detect_indirect_chains(results: List[EvaluationResult], clusters: Dict[str, List[str]]) -> List[Dict]:
+    direct_pairs: Set[Tuple[str, str]] = set()
+    for res in results:
+        sig = getattr(res, "evaluation_signal", "GREEN")
+        sig_val = sig.value if hasattr(sig, "value") else str(sig)
+        if sig_val in ["RED", "YELLOW"]:
+            u, v = sorted([res.item_id, res.matched_peer_id])
+            direct_pairs.add((u, v))
+
+    seen_clusters: Set[Tuple[str, ...]] = set()
+    unique_clusters: List[List[str]] = []
+    for members in clusters.values():
+        key = tuple(sorted(members))
+        if len(key) >= 3 and key not in seen_clusters:
+            seen_clusters.add(key)
+            unique_clusters.append(list(key))
+
+    indirect_chains: List[Dict] = []
+    for cluster in unique_clusters:
+        for i in range(len(cluster)):
+            for j in range(i + 1, len(cluster)):
+                u, v = sorted([cluster[i], cluster[j]])
+                if (u, v) not in direct_pairs:
+                    indirect_chains.append({"item_id_a": u, "item_id_b": v, "cluster_members": cluster})
+    return indirect_chains

sieve_referee-1.0.0/sieve_referee/chunker.py

@@ -0,0 +1,15 @@
+import re
+from typing import List
+from .models import Chunk
+
+
+def split_into_multi_scale_windows(text: str, window_sizes: List[int] = [1, 2]) -> List[Chunk]:
+    sentences = [s.strip() + "。" for s in re.split(r"正?。|！|？|\n", text) if s.strip()]
+    chunks = []
+    chunk_id = 0
+    for w_size in window_sizes:
+        for i in range(len(sentences) - w_size + 1):
+            window = sentences[i : i + w_size]
+            chunks.append(Chunk(chunk_id=chunk_id, text="".join(window), window_size=w_size))
+            chunk_id += 1
+    return chunks

sieve_referee-1.0.0/sieve_referee/cli.py

@@ -0,0 +1,103 @@
+import argparse
+import json
+import sys
+from pathlib import Path
+from typing import List
+
+from .chain_detector import detect_indirect_chains
+from .dashboard_builder import generate_dashboard_html, generate_standalone_dashboard_html
+from .diff_builder import generate_diff_html
+from .evaluator import batch_evaluate, build_clusters
+from .io_utils import safe_read_text
+from .models import EvaluationResult
+
+
+def result_to_dict(res: EvaluationResult) -> dict:
+    signal_str = res.evaluation_signal.value if hasattr(res.evaluation_signal, "value") else str(res.evaluation_signal)
+    return {
+        "item_id": res.item_id,
+        "matched_peer_id": res.matched_peer_id,
+        "signal": signal_str,
+        "mask": res.mask,
+        "pattern_name": res.pattern_name,
+        "reason": res.reason,
+    }
+
+
+def main():
+    parser = argparse.ArgumentParser(description="sieve_referee: Deterministic Text Plagiarism Screening Engine")
+    parser.add_argument("target_dir", type=str, help="Target directory containing .txt files")
+    parser.add_argument("--output-dir", type=str, default="reports")
+    parser.add_argument("--json", action="store_true")
+    parser.add_argument("--json-file", type=str, default=None)
+    parser.add_argument("--fail-on-risk", action="store_true")
+    parser.add_argument("-j", "--jobs", type=int, default=None)
+    parser.add_argument("--standalone", action="store_true")
+    args = parser.parse_args()
+
+    target_path = Path(args.target_dir)
+    if not target_path.exists() or not target_path.is_dir():
+        print(f"Error: Directory '{args.target_dir}' does not exist.", file=sys.stderr)
+        sys.exit(2)
+
+    doc_dict = {}
+    for p in target_path.glob("*.txt"):
+        content = safe_read_text(p)
+        if content:
+            doc_dict[p.name] = content
+
+    if len(doc_dict) < 2:
+        print("Error: At least 2 valid (non-empty) .txt files are required.", file=sys.stderr)
+        sys.exit(2)
+
+    results: List[EvaluationResult] = batch_evaluate(doc_dict, max_workers=args.jobs)
+
+    if args.json:
+        print(json.dumps([result_to_dict(r) for r in results], ensure_ascii=False, indent=2))
+    if args.json_file:
+        out_json_path = Path(args.json_file)
+        out_json_path.parent.mkdir(parents=True, exist_ok=True)
+        out_json_path.write_text(
+            json.dumps([result_to_dict(r) for r in results], ensure_ascii=False, indent=2), encoding="utf-8"
+        )
+        print(f"JSON report saved to: {out_json_path}")
+
+    output_dir = Path(args.output_dir)
+    output_dir.mkdir(parents=True, exist_ok=True)
+
+    diff_map = {}
+    for res in results:
+        doc_a_text = doc_dict.get(res.item_id, "")
+        doc_b_text = doc_dict.get(res.matched_peer_id, "")
+        pair_key = f"{res.item_id}_vs_{res.matched_peer_id}"
+        diff_map[pair_key] = generate_diff_html(doc_a_text, doc_b_text, res)
+
+    clusters = build_clusters(results)
+    indirect_chains = detect_indirect_chains(results, clusters)
+
+    if args.standalone:
+        out_file = output_dir / "sieve_dashboard_standalone.html"
+        generate_standalone_dashboard_html(results, diff_map, out_file)
+        print(f"Standalone HTML report generated: {out_file}")
+    else:
+        diff_dir = output_dir / "diff_htmls"
+        diff_dir.mkdir(exist_ok=True)
+        for pair_key, diff_html_str in diff_map.items():
+            (diff_dir / f"{pair_key}.html").write_text(diff_html_str, encoding="utf-8")
+        out_file = output_dir / "sieve_dashboard.html"
+        file_names = list(doc_dict.keys())
+        generate_dashboard_html(results, indirect_chains, file_names, out_file)
+        print(f"Standard HTML report generated: {out_file}")
+
+    has_red_risk = any(
+        (r.evaluation_signal.value if hasattr(r.evaluation_signal, "value") else str(r.evaluation_signal)) == "RED"
+        for r in results
+    )
+    if args.fail_on_risk and has_red_risk:
+        print("\n[CI AUDIT FAILED] High risk plagiarism (RED signal) detected!", file=sys.stderr)
+        sys.exit(1)
+    sys.exit(0)
+
+
+if __name__ == "__main__":
+    main()