sieve-scope 1.3.0__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
|
@@ -0,0 +1,168 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: sieve-scope
|
|
3
|
+
Version: 1.3.0
|
|
4
|
+
Summary: Deterministic and Explainable Code Plagiarism Detection Engine with 7-bit Observation Space
|
|
5
|
+
Author: Kai IWASAKI
|
|
6
|
+
License: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/neguseatama/sieve-scope
|
|
8
|
+
Project-URL: Repository, https://github.com/neguseatama/sieve-scope.git
|
|
9
|
+
Project-URL: Bug Tracker, https://github.com/neguseatama/sieve-scope/issues
|
|
10
|
+
Classifier: Development Status :: 5 - Production/Stable
|
|
11
|
+
Classifier: Intended Audience :: Developers
|
|
12
|
+
Classifier: Intended Audience :: Education
|
|
13
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
14
|
+
Classifier: Programming Language :: Python :: 3
|
|
15
|
+
Classifier: Programming Language :: Python :: 3.9
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
19
|
+
Classifier: Topic :: Software Development :: Libraries :: Python Modules
|
|
20
|
+
Classifier: Topic :: Text Processing :: Linguistic
|
|
21
|
+
Requires-Python: >=3.9
|
|
22
|
+
Description-Content-Type: text/markdown
|
|
23
|
+
License-File: LICENSE
|
|
24
|
+
Dynamic: license-file
|
|
25
|
+
|
|
26
|
+
# Sieve Scope v1.3
|
|
27
|
+
|
|
28
|
+
**English** | [日本語](README.ja.md)
|
|
29
|
+
|
|
30
|
+
> **"See through the formatting, and look into the pure logic of the algorithm."**
|
|
31
|
+
> A fully local, zero-dependency, deterministic (100% reproducible) code plagiarism and structural similarity detection engine.
|
|
32
|
+
> Extends [Sieve-Core](https://github.com/neguseatama/sieve-core)'s deterministic similarity engine and [Sieve-Referee](https://github.com/neguseatama/sieve-referee)'s multi-hypothesis LUT judgment philosophy to the **source code domain**.
|
|
33
|
+
|
|
34
|
+
[](https://github.com/neguseatama/sieve-scope/actions/workflows/test.yml)
|
|
35
|
+
[](https://opensource.org/licenses/MIT)
|
|
36
|
+
[](https://www.python.org/)
|
|
37
|
+
|
|
38
|
+
---
|
|
39
|
+
|
|
40
|
+
## 💡 Concept & Philosophy
|
|
41
|
+
|
|
42
|
+
The starting point of this project is **Sieve-Core**, a core engine that applies the "sieve of set theory" design philosophy of **Sieve-AI**—inspired by the Sieve of Eratosthenes—to the more general domain of text and data processing.
|
|
43
|
+
|
|
44
|
+
Next, **Sieve Referee**, which adds a "sieve of combinatorics" to Sieve-Core, is a zero-dependency Python library that extracts reliable information from noise based on corroboration from multiple independent sources.
|
|
45
|
+
|
|
46
|
+
Then, **Sieve Scope** further adds a two-axis evaluation and redesigns the approach, making it an explainable code plagiarism detection engine that observes the **surface format** (variable names, comments, whitespace) and the **essential logic** (algorithm structure) separately.
|
|
47
|
+
|
|
48
|
+
Traditional string comparison and simple similarity measures struggle to distinguish between:
|
|
49
|
+
- Plagiarism by merely renaming variables (**variable laundering**)
|
|
50
|
+
- Coincidental matches due to shared templates (**boilerplate**)
|
|
51
|
+
|
|
52
|
+
`Sieve Scope` observes two axes independently:
|
|
53
|
+
- **AST (Abstract Syntax Tree)** — extracting the "logic skeleton" with identifiers removed
|
|
54
|
+
- **tokenize** — extracting "identifier vocabulary and order"
|
|
55
|
+
|
|
56
|
+
By inheriting the hypothesis mask (H1–H7) and LUT from Sieve-Referee, it fully explains why a judgment was made.
|
|
57
|
+
|
|
58
|
+
> The name "Sieve Scope" comes from the design philosophy of seeing through the surface of code (format) and comparing only the pure logic, like a microscope.
|
|
59
|
+
|
|
60
|
+
---
|
|
61
|
+
|
|
62
|
+
## 🔥 Key Features
|
|
63
|
+
|
|
64
|
+
- **7-bit Observation Space (H1–H7)**
|
|
65
|
+
Independently evaluates AST skeleton, identifier vocabulary, positional/order (g_POS), constants (g_CONST), and vocabulary context (g_FREQ).
|
|
66
|
+
- **v0_mask compatibility**
|
|
67
|
+
The legacy 4-bit v0_mask is preserved and regression-tested.
|
|
68
|
+
- **Strict Cluster Safety Invariant**
|
|
69
|
+
H4 is promoted to 1 only when the cluster size is at least 3 **and** there is direct similarity evidence (H2∨H3). Indirect "telephone game" chains are never mixed into direct judgments.
|
|
70
|
+
- **Short-Circuit Optimization**
|
|
71
|
+
Pairs with no structural or vocabulary similarity (H2=0 ∧ H3=0) are immediately returned as `1000-000`, skipping expensive hash computations.
|
|
72
|
+
- **Zero Dependencies & Deterministic**
|
|
73
|
+
Implemented using only the Python standard library. No external packages, no API costs, no probabilistic elements.
|
|
74
|
+
|
|
75
|
+
---
|
|
76
|
+
|
|
77
|
+
## 📐 7-bit Observation Space
|
|
78
|
+
|
|
79
|
+
The evaluation function `f_v1.3(C1, C2)` projects a code pair into seven boolean hypotheses `(H1, H2, H3, H4, H5, H6, H7)`.
|
|
80
|
+
|
|
81
|
+
| Hypothesis | Name | What it measures | Criterion / Threshold |
|
|
82
|
+
| :--- | :--- | :--- | :--- |
|
|
83
|
+
| **H1** | Parseability | Can both be parsed as Python syntax? | Success/failure of AST construction |
|
|
84
|
+
| **H2** | AST Skeleton | Do the AST node type sequences match exactly? | Exact match of skeleton strings |
|
|
85
|
+
| **H3** | Identifier Similarity | Jaccard + sequence similarity of identifiers | Score `> 0.50` |
|
|
86
|
+
| **H4** | Cluster Safety | Direct similarity + membership in a valid cluster (size ≥3) | Invariant condition satisfied |
|
|
87
|
+
| **H5** | g_POS | SHA-256 hash of recursive AST field representation | Exact hash match |
|
|
88
|
+
| **H6** | g_CONST | Constants / literals set match | Sorted constant list exact match |
|
|
89
|
+
| **H7** | g_FREQ | Weighted identifier bi-gram Dice coefficient | Score `>= 0.60` |
|
|
90
|
+
|
|
91
|
+
Mask notation is `v0_mask (4-bit) - extension (3-bit)`.
|
|
92
|
+
Example: `1100-011` → H1=1, H2=1, H3=0, H4=0 / H5=0, H6=1, H7=1
|
|
93
|
+
|
|
94
|
+
---
|
|
95
|
+
|
|
96
|
+
## 📊 Representative LUT Patterns
|
|
97
|
+
|
|
98
|
+
| v0_mask | Extended Example | Judgment Label | Description |
|
|
99
|
+
|:---:|:---:|:---|:---|
|
|
100
|
+
| `1110` | `1110-111` | `EXACT_CLONE` | Observation-space exact match |
|
|
101
|
+
| `1100` | `1100-110` | `VARIABLE_LAUNDERING` | Logic matches but surface differs |
|
|
102
|
+
| `1010` | `1010-011` | `COINCIDENTAL_FRAMEWORK_MATCH` | Logic differs but surface matches |
|
|
103
|
+
| `1000` | `1000-000` | `INDEPENDENT_LOGIC` | No direct similarity (short-circuit) |
|
|
104
|
+
| `1110` | `1110-101` | `CONSTANT_MODIFIED_CLONE` | Only constants differ |
|
|
105
|
+
| `1110` | `1110-011` | `PERMUTATED_CLONE` | Non-commutative order reversed |
|
|
106
|
+
|
|
107
|
+
---
|
|
108
|
+
|
|
109
|
+
## 💻 Quick Start
|
|
110
|
+
|
|
111
|
+
```python
|
|
112
|
+
from sieve_scope import SieveScopeEngineV1_3, SieveClusterAnalyzerV1_3
|
|
113
|
+
|
|
114
|
+
engine = SieveScopeEngineV1_3(jaccard_threshold=0.50, freq_dice_threshold=0.60)
|
|
115
|
+
|
|
116
|
+
code_a = "def calc_sum(a, b):\n return a + b"
|
|
117
|
+
code_b = "def calc_sum(x, y):\n return x + y"
|
|
118
|
+
|
|
119
|
+
result = engine.evaluate_pair(code_a, code_b)
|
|
120
|
+
|
|
121
|
+
print(result["v0_mask"]) # e.g., 1100
|
|
122
|
+
print(result["mask"]) # e.g., 1100-110
|
|
123
|
+
|
|
124
|
+
---
|
|
125
|
+
|
|
126
|
+
### Configuration / Thresholds
|
|
127
|
+
|
|
128
|
+
- `jaccard_threshold` (default: `0.50`): Threshold for H3 Multiset Jaccard + sequence similarity.
|
|
129
|
+
- `freq_dice_threshold` (default: `0.60`): Threshold for H7 Weighted identifier bi-gram Dice coefficient.
|
|
130
|
+
|
|
131
|
+
> **Note on Calibration & Defaults (v1.3)**:
|
|
132
|
+
> Exhaustive grid search experiments on synthetic benchmark datasets demonstrate optimal class separation ($F1 = 1.0000$) across a wide threshold range of `0.35` to `0.75`. However, the default threshold parameters are strictly maintained at **`0.50` / `0.60`** in order to preserve backwards compatibility with v0, maintain regression test stability, and prevent potential false positives caused by shared domain vocabulary in production codebases.
|
|
133
|
+
|
|
134
|
+
---
|
|
135
|
+
|
|
136
|
+
## 🔬 Testing & Verification
|
|
137
|
+
Cross-Python Validation Summary
|
|
138
|
+
24 regression cases (test_v1_3_full_suite.py): PASSED
|
|
139
|
+
|
|
140
|
+
5 compatibility tests (test_v1_3_regression.py): PASSED
|
|
141
|
+
|
|
142
|
+
6 edge-case & parallel determinism tests (test_v1_3_edge_cases.py): PASSED
|
|
143
|
+
|
|
144
|
+
81 synthetic threshold configurations (threshold_tuning.py): PASSED
|
|
145
|
+
|
|
146
|
+
All test suites and synthetic experiments passed completely without errors across Python 3.9, 3.10, 3.11, and 3.12.
|
|
147
|
+
|
|
148
|
+
Threshold Calibration & Synthetic Validation
|
|
149
|
+
The threshold parameters (H3 Multiset Jaccard: 0.50, H7 Weighted Bi-gram Dice: 0.60) were designed and calibrated using a synthetic validation pair set (Positive 6 pairs / Negative 3 pairs).
|
|
150
|
+
The engine's 1110-111 indicates an exact match in the current 7-dimensional observation space (Observation Clones), and does not guarantee complete semantic equivalence (Semantic Clones).
|
|
151
|
+
|
|
152
|
+
---
|
|
153
|
+
|
|
154
|
+
## 📚 Documentation
|
|
155
|
+
|
|
156
|
+
- 🏛️ Architecture Specification
|
|
157
|
+
|
|
158
|
+
- 🔌 API & Integration Guide
|
|
159
|
+
|
|
160
|
+
- 🚀 Deployment Guide
|
|
161
|
+
|
|
162
|
+
- 🔐 CI/CD & Security Guide
|
|
163
|
+
|
|
164
|
+
## 📄 License
|
|
165
|
+
MIT License. See LICENSE for details.
|
|
166
|
+
|
|
167
|
+
## 👤 Author
|
|
168
|
+
Kai IWASAKI
|
|
@@ -0,0 +1,6 @@
|
|
|
1
|
+
sieve_scope.py,sha256=xts1AlZ9csOvbg8aCz8X-n26nBG2oFbjnSQi5Cg7Rfs,12871
|
|
2
|
+
sieve_scope-1.3.0.dist-info/licenses/LICENSE,sha256=gkCel7djhBCn21V3Nwq1EM5HN6y53y-xoNZ3fRsfz5Y,1067
|
|
3
|
+
sieve_scope-1.3.0.dist-info/METADATA,sha256=V2EP_sgXTxSD5eiO2pYtdc3T9ORZGIW1Sp4uXR4yBuo,8313
|
|
4
|
+
sieve_scope-1.3.0.dist-info/WHEEL,sha256=aeYiig01lYGDzBgS8HxWXOg3uV61G9ijOsup-k9o1sk,91
|
|
5
|
+
sieve_scope-1.3.0.dist-info/top_level.txt,sha256=4PpNdcIQ5n6Bfc9wqsyuDfzitVcIGwpRh0yaBhBhl10,12
|
|
6
|
+
sieve_scope-1.3.0.dist-info/RECORD,,
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Kai IWASAKI
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
sieve_scope
|
sieve_scope.py
ADDED
|
@@ -0,0 +1,326 @@
|
|
|
1
|
+
"""
|
|
2
|
+
Sieve Scope Engine v1.3 (Production Ready)
|
|
3
|
+
- 7-bit Observation Space (H1 - H7)
|
|
4
|
+
- Short-Circuit Evaluation
|
|
5
|
+
- Tokenize-based Pure Identifier Extraction
|
|
6
|
+
- Weighted Bi-gram Dice Coefficient
|
|
7
|
+
- Parallel Pair Evaluation
|
|
8
|
+
- Indirect Chain Detection
|
|
9
|
+
"""
|
|
10
|
+
|
|
11
|
+
import ast
|
|
12
|
+
import difflib
|
|
13
|
+
import hashlib
|
|
14
|
+
import io
|
|
15
|
+
import keyword
|
|
16
|
+
import tokenize
|
|
17
|
+
from collections import Counter, defaultdict, deque
|
|
18
|
+
from concurrent.futures import ProcessPoolExecutor, as_completed
|
|
19
|
+
from typing import Any, Dict, List, Set, Tuple, Optional
|
|
20
|
+
|
|
21
|
+
# v0完全互換の除外識別子集合
|
|
22
|
+
IGNORED_WORDS_V0: Set[str] = {
|
|
23
|
+
'print', 'len', 'range', 'sum', 'int', 'str', 'self', '__init__'
|
|
24
|
+
}
|
|
25
|
+
|
|
26
|
+
|
|
27
|
+
def extract_pure_identifiers(code: str, ignored_words: Set[str] = IGNORED_WORDS_V0) -> List[str]:
|
|
28
|
+
"""
|
|
29
|
+
v0完全互換: tokenizeベースで非キーワードNAMEトークンを順序通り抽出する。
|
|
30
|
+
関数名・クラス名・変数名・属性名をすべて識別子として捕捉する。
|
|
31
|
+
"""
|
|
32
|
+
identifiers = []
|
|
33
|
+
try:
|
|
34
|
+
tokens = tokenize.generate_tokens(io.StringIO(code).readline)
|
|
35
|
+
for toknum, tokval, _, _, _ in tokens:
|
|
36
|
+
if toknum == tokenize.NAME and not keyword.iskeyword(tokval):
|
|
37
|
+
if tokval not in ignored_words:
|
|
38
|
+
identifiers.append(tokval)
|
|
39
|
+
except Exception:
|
|
40
|
+
pass
|
|
41
|
+
return identifiers
|
|
42
|
+
|
|
43
|
+
|
|
44
|
+
def build_pos_tuple(node: Any) -> Any:
|
|
45
|
+
"""
|
|
46
|
+
v1.2完全互換: ASTフィールド構造タプルを生成する。
|
|
47
|
+
value / id / attr フィールドのうち非AST値(定数・識別子名)は除外するが、
|
|
48
|
+
arg などのその他スカラーフィールドは保持する。
|
|
49
|
+
"""
|
|
50
|
+
if not isinstance(node, ast.AST):
|
|
51
|
+
if isinstance(node, list):
|
|
52
|
+
return tuple(build_pos_tuple(x) for x in node)
|
|
53
|
+
return str(node)
|
|
54
|
+
|
|
55
|
+
field_tuples = []
|
|
56
|
+
for field, value in ast.iter_fields(node):
|
|
57
|
+
if field in ("value", "id", "attr") and not isinstance(value, ast.AST):
|
|
58
|
+
continue
|
|
59
|
+
field_tuples.append((field, build_pos_tuple(value)))
|
|
60
|
+
return (type(node).__name__, tuple(field_tuples))
|
|
61
|
+
|
|
62
|
+
|
|
63
|
+
class ASTSkeletonVisitor(ast.NodeVisitor):
|
|
64
|
+
"""
|
|
65
|
+
H2: AST構造骨格の抽出。
|
|
66
|
+
generic_visit はすべてのノード型名を記録する(Load, Store, Del, 演算子ノードを含む)。
|
|
67
|
+
"""
|
|
68
|
+
def __init__(self):
|
|
69
|
+
self.skeleton = []
|
|
70
|
+
|
|
71
|
+
def generic_visit(self, node):
|
|
72
|
+
self.skeleton.append(type(node).__name__)
|
|
73
|
+
super().generic_visit(node)
|
|
74
|
+
|
|
75
|
+
|
|
76
|
+
class ASTConstantVisitor(ast.NodeVisitor):
|
|
77
|
+
"""H6: g_CONST 定数・リテラルの抽出"""
|
|
78
|
+
def __init__(self):
|
|
79
|
+
self.constants = []
|
|
80
|
+
|
|
81
|
+
def visit_Constant(self, node):
|
|
82
|
+
self.constants.append(str(node.value))
|
|
83
|
+
self.generic_visit(node)
|
|
84
|
+
|
|
85
|
+
|
|
86
|
+
class SieveScopeEngineV1_3:
|
|
87
|
+
"""7次元観測空間評価エンジン"""
|
|
88
|
+
|
|
89
|
+
def __init__(self, jaccard_threshold: float = 0.50, freq_dice_threshold: float = 0.60):
|
|
90
|
+
self.jaccard_threshold = jaccard_threshold
|
|
91
|
+
self.freq_dice_threshold = freq_dice_threshold
|
|
92
|
+
|
|
93
|
+
def evaluate_pair(self, code1: str, code2: str) -> Dict[str, Any]:
|
|
94
|
+
# H1: Parseability
|
|
95
|
+
try:
|
|
96
|
+
tree1 = ast.parse(code1)
|
|
97
|
+
tree2 = ast.parse(code2)
|
|
98
|
+
h1 = 1
|
|
99
|
+
except Exception:
|
|
100
|
+
return {
|
|
101
|
+
"h_states": {"H1": 0, "H2": 0, "H3": 0, "H4": 0, "H5": 0, "H6": 0, "H7": 0},
|
|
102
|
+
"v0_mask": "0000",
|
|
103
|
+
"mask": "0000-000",
|
|
104
|
+
"short_circuited": True,
|
|
105
|
+
}
|
|
106
|
+
|
|
107
|
+
# H2: AST Skeleton(全ノード型の列を比較)
|
|
108
|
+
vis1, vis2 = ASTSkeletonVisitor(), ASTSkeletonVisitor()
|
|
109
|
+
vis1.visit(tree1)
|
|
110
|
+
vis2.visit(tree2)
|
|
111
|
+
skeleton1 = "_".join(vis1.skeleton)
|
|
112
|
+
skeleton2 = "_".join(vis2.skeleton)
|
|
113
|
+
h2 = 1 if skeleton1 == skeleton2 else 0
|
|
114
|
+
|
|
115
|
+
# H3: 純粋識別子の類似度(Multiset Jaccard + SequenceMatcher)
|
|
116
|
+
ids1 = extract_pure_identifiers(code1)
|
|
117
|
+
ids2 = extract_pure_identifiers(code2)
|
|
118
|
+
|
|
119
|
+
if not ids1 and not ids2:
|
|
120
|
+
h3_score = 1.0
|
|
121
|
+
elif not ids1 or not ids2:
|
|
122
|
+
h3_score = 0.0
|
|
123
|
+
else:
|
|
124
|
+
c1, c2 = Counter(ids1), Counter(ids2)
|
|
125
|
+
intersection = sum((c1 & c2).values())
|
|
126
|
+
union = sum((c1 | c2).values())
|
|
127
|
+
multiset_jaccard = intersection / union if union > 0 else 0.0
|
|
128
|
+
|
|
129
|
+
seq_ratio = difflib.SequenceMatcher(None, ids1, ids2).ratio()
|
|
130
|
+
h3_score = 0.6 * multiset_jaccard + 0.4 * seq_ratio
|
|
131
|
+
|
|
132
|
+
# v0互換: 厳密な '>' 判定
|
|
133
|
+
h3 = 1 if h3_score > self.jaccard_threshold else 0
|
|
134
|
+
|
|
135
|
+
# ショートサーキット: H2=0 かつ H3=0 の場合のみ高コスト計算をスキップ
|
|
136
|
+
if h2 == 0 and h3 == 0:
|
|
137
|
+
return {
|
|
138
|
+
"h_states": {"H1": 1, "H2": 0, "H3": 0, "H4": 0, "H5": 0, "H6": 0, "H7": 0},
|
|
139
|
+
"v0_mask": "1000",
|
|
140
|
+
"mask": "1000-000",
|
|
141
|
+
"short_circuited": True,
|
|
142
|
+
"scores": {"h3_score": h3_score},
|
|
143
|
+
}
|
|
144
|
+
|
|
145
|
+
# H5: g_POS(build_pos_tuple による決定論的ハッシュ)
|
|
146
|
+
pos_tuple1 = build_pos_tuple(tree1)
|
|
147
|
+
pos_tuple2 = build_pos_tuple(tree2)
|
|
148
|
+
pos_hash1 = hashlib.sha256(str(pos_tuple1).encode('utf-8')).hexdigest()
|
|
149
|
+
pos_hash2 = hashlib.sha256(str(pos_tuple2).encode('utf-8')).hexdigest()
|
|
150
|
+
h5 = 1 if pos_hash1 == pos_hash2 else 0
|
|
151
|
+
|
|
152
|
+
# H6: g_CONST(定数集合の一致)
|
|
153
|
+
c_vis1, c_vis2 = ASTConstantVisitor(), ASTConstantVisitor()
|
|
154
|
+
c_vis1.visit(tree1)
|
|
155
|
+
c_vis2.visit(tree2)
|
|
156
|
+
h6 = 1 if sorted(c_vis1.constants) == sorted(c_vis2.constants) else 0
|
|
157
|
+
|
|
158
|
+
# H7: g_FREQ(重み付き Bi-gram Dice 係数)
|
|
159
|
+
bigrams1 = self._get_weighted_bigrams(ids1)
|
|
160
|
+
bigrams2 = self._get_weighted_bigrams(ids2)
|
|
161
|
+
|
|
162
|
+
common_keys = set(bigrams1.keys()) & set(bigrams2.keys())
|
|
163
|
+
common_weight = sum(min(bigrams1[bg], bigrams2[bg]) for bg in common_keys)
|
|
164
|
+
total_weight = sum(bigrams1.values()) + sum(bigrams2.values())
|
|
165
|
+
|
|
166
|
+
dice_score = (2.0 * common_weight) / total_weight if total_weight > 0 else 0.0
|
|
167
|
+
h7 = 1 if dice_score >= self.freq_dice_threshold else 0
|
|
168
|
+
|
|
169
|
+
# H4: ペア評価では常に0(クラスタ分析時のみ昇格)
|
|
170
|
+
h4 = 0
|
|
171
|
+
|
|
172
|
+
v0_mask = f"{h1}{h2}{h3}{h4}"
|
|
173
|
+
ext_mask = f"{h5}{h6}{h7}"
|
|
174
|
+
|
|
175
|
+
return {
|
|
176
|
+
"h_states": {"H1": h1, "H2": h2, "H3": h3, "H4": h4, "H5": h5, "H6": h6, "H7": h7},
|
|
177
|
+
"v0_mask": v0_mask,
|
|
178
|
+
"mask": f"{v0_mask}-{ext_mask}",
|
|
179
|
+
"short_circuited": False,
|
|
180
|
+
"scores": {
|
|
181
|
+
"h3_score": h3_score,
|
|
182
|
+
"h7_dice_score": dice_score,
|
|
183
|
+
},
|
|
184
|
+
}
|
|
185
|
+
|
|
186
|
+
@staticmethod
|
|
187
|
+
def _get_weighted_bigrams(identifiers: List[str]) -> Dict[str, float]:
|
|
188
|
+
"""識別子列から Bi-gram と出現頻度(重み)を生成"""
|
|
189
|
+
bigrams = defaultdict(float)
|
|
190
|
+
for i in range(len(identifiers) - 1):
|
|
191
|
+
bg = f"{identifiers[i]}___{identifiers[i+1]}"
|
|
192
|
+
bigrams[bg] += 1.0
|
|
193
|
+
return bigrams
|
|
194
|
+
|
|
195
|
+
|
|
196
|
+
def _pair_worker_func(args):
|
|
197
|
+
"""ProcessPoolExecutor用のトップレベル並列ワーカー関数"""
|
|
198
|
+
key1, code1, key2, code2, jaccard_th, freq_th = args
|
|
199
|
+
engine = SieveScopeEngineV1_3(jaccard_threshold=jaccard_th, freq_dice_threshold=freq_th)
|
|
200
|
+
result = engine.evaluate_pair(code1, code2)
|
|
201
|
+
return (key1, key2), result
|
|
202
|
+
|
|
203
|
+
|
|
204
|
+
class SieveClusterAnalyzerV1_3:
|
|
205
|
+
"""クラスタ解析・安全不変式・並列処理・間接伝搬警告エンジン"""
|
|
206
|
+
|
|
207
|
+
def __init__(self, engine: SieveScopeEngineV1_3, min_cluster_size: int = 3):
|
|
208
|
+
self.engine = engine
|
|
209
|
+
self.min_cluster_size = min_cluster_size
|
|
210
|
+
|
|
211
|
+
def analyze_corpus(self, corpus: Dict[str, str], max_workers: Optional[int] = None) -> Dict[str, Any]:
|
|
212
|
+
keys = sorted(list(corpus.keys()))
|
|
213
|
+
pair_tasks = []
|
|
214
|
+
|
|
215
|
+
for i in range(len(keys)):
|
|
216
|
+
for j in range(i + 1, len(keys)):
|
|
217
|
+
k1, k2 = keys[i], keys[j]
|
|
218
|
+
pair_tasks.append((
|
|
219
|
+
k1, corpus[k1], k2, corpus[k2],
|
|
220
|
+
self.engine.jaccard_threshold,
|
|
221
|
+
self.engine.freq_dice_threshold,
|
|
222
|
+
))
|
|
223
|
+
|
|
224
|
+
raw_pair_results = {}
|
|
225
|
+
|
|
226
|
+
if len(pair_tasks) > 10 and max_workers != 1:
|
|
227
|
+
with ProcessPoolExecutor(max_workers=max_workers) as executor:
|
|
228
|
+
futures = [executor.submit(_pair_worker_func, task) for task in pair_tasks]
|
|
229
|
+
for future in as_completed(futures):
|
|
230
|
+
pair_key, res = future.result()
|
|
231
|
+
raw_pair_results[pair_key] = res
|
|
232
|
+
else:
|
|
233
|
+
for task in pair_tasks:
|
|
234
|
+
pair_key, res = _pair_worker_func(task)
|
|
235
|
+
raw_pair_results[pair_key] = res
|
|
236
|
+
|
|
237
|
+
# 決定論的順序の保証
|
|
238
|
+
pair_results = dict(sorted(raw_pair_results.items()))
|
|
239
|
+
|
|
240
|
+
# 直接類似エッジの構築(H2=1 または H3=1)
|
|
241
|
+
direct_adj = defaultdict(set)
|
|
242
|
+
for (k1, k2), res in pair_results.items():
|
|
243
|
+
if res["h_states"]["H2"] == 1 or res["h_states"]["H3"] == 1:
|
|
244
|
+
direct_adj[k1].add(k2)
|
|
245
|
+
direct_adj[k2].add(k1)
|
|
246
|
+
|
|
247
|
+
# 連結成分抽出
|
|
248
|
+
visited = set()
|
|
249
|
+
valid_clusters = []
|
|
250
|
+
all_clusters = []
|
|
251
|
+
|
|
252
|
+
for node in keys:
|
|
253
|
+
if node not in visited:
|
|
254
|
+
component = set()
|
|
255
|
+
queue = deque([node])
|
|
256
|
+
visited.add(node)
|
|
257
|
+
while queue:
|
|
258
|
+
curr = queue.popleft()
|
|
259
|
+
component.add(curr)
|
|
260
|
+
for neighbor in sorted(list(direct_adj[curr])):
|
|
261
|
+
if neighbor not in visited:
|
|
262
|
+
visited.add(neighbor)
|
|
263
|
+
queue.append(neighbor)
|
|
264
|
+
|
|
265
|
+
all_clusters.append(component)
|
|
266
|
+
if len(component) >= self.min_cluster_size:
|
|
267
|
+
valid_clusters.append(component)
|
|
268
|
+
|
|
269
|
+
# H4 安全不変式の適用(有効クラスタ内・直接類似のあるペアのみ昇格)
|
|
270
|
+
valid_nodes = set().union(*valid_clusters) if valid_clusters else set()
|
|
271
|
+
|
|
272
|
+
for (k1, k2), res in pair_results.items():
|
|
273
|
+
is_direct = (res["h_states"]["H2"] == 1 or res["h_states"]["H3"] == 1)
|
|
274
|
+
in_valid_cluster = (k1 in valid_nodes and k2 in valid_nodes and
|
|
275
|
+
any(k1 in cl and k2 in cl for cl in valid_clusters))
|
|
276
|
+
|
|
277
|
+
if in_valid_cluster and is_direct:
|
|
278
|
+
res["h_states"]["H4"] = 1
|
|
279
|
+
h = res["h_states"]
|
|
280
|
+
res["v0_mask"] = f"{h['H1']}{h['H2']}{h['H3']}1"
|
|
281
|
+
res["mask"] = f"{res['v0_mask']}-{h['H5']}{h['H6']}{h['H7']}"
|
|
282
|
+
|
|
283
|
+
# 間接伝搬チェーン警告の検出
|
|
284
|
+
warnings = self.detect_indirect_chains(keys, direct_adj, valid_clusters)
|
|
285
|
+
|
|
286
|
+
return {
|
|
287
|
+
"valid_clusters": [sorted(list(c)) for c in valid_clusters],
|
|
288
|
+
"all_clusters": [sorted(list(c)) for c in all_clusters],
|
|
289
|
+
"pair_results": pair_results,
|
|
290
|
+
"warnings": warnings,
|
|
291
|
+
}
|
|
292
|
+
|
|
293
|
+
@staticmethod
|
|
294
|
+
def detect_indirect_chains(nodes, direct_adj, valid_clusters):
|
|
295
|
+
warnings = []
|
|
296
|
+
for cluster in valid_clusters:
|
|
297
|
+
cluster_nodes = sorted(list(cluster))
|
|
298
|
+
for i in range(len(cluster_nodes)):
|
|
299
|
+
for j in range(i + 1, len(cluster_nodes)):
|
|
300
|
+
u, v = cluster_nodes[i], cluster_nodes[j]
|
|
301
|
+
if v not in direct_adj[u]:
|
|
302
|
+
queue = deque([[u]])
|
|
303
|
+
visited = {u}
|
|
304
|
+
found_path = None
|
|
305
|
+
while queue:
|
|
306
|
+
path = queue.popleft()
|
|
307
|
+
curr = path[-1]
|
|
308
|
+
if curr == v:
|
|
309
|
+
found_path = path
|
|
310
|
+
break
|
|
311
|
+
for neighbor in sorted(list(direct_adj[curr])):
|
|
312
|
+
if neighbor not in visited and neighbor in cluster:
|
|
313
|
+
visited.add(neighbor)
|
|
314
|
+
queue.append(path + [neighbor])
|
|
315
|
+
if found_path and len(found_path) > 2:
|
|
316
|
+
warnings.append({
|
|
317
|
+
"code": "INDIRECT_PLAGIARISM_CHAIN_WARNING",
|
|
318
|
+
"pair": (u, v),
|
|
319
|
+
"message": (
|
|
320
|
+
f"Direct similarity not confirmed between '{u}' and '{v}', "
|
|
321
|
+
f"but indirect chain detected via {len(found_path)-1} hops."
|
|
322
|
+
),
|
|
323
|
+
"propagation_path": " -> ".join(found_path),
|
|
324
|
+
"hops": len(found_path) - 1,
|
|
325
|
+
})
|
|
326
|
+
return warnings
|