sieve-core-engine 1.0.0__tar.gz

sieve_core_engine-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Kai IWASAKI
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

sieve_core_engine-1.0.0/PKG-INFO

@@ -0,0 +1,304 @@
+Metadata-Version: 2.4
+Name: sieve-core-engine
+Version: 1.0.0
+Summary: Core logic and base components for the Sieve ecosystem
+Author: Kai IWASAKI
+License: MIT
+Project-URL: Homepage, https://github.com/neguseatama/sieve-core
+Project-URL: Repository, https://github.com/neguseatama/sieve-core.git
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# Sieve-Core (v1.0.0-MVP)
+
+**English | [日本語](./README.ja.md)**
+
+**A deterministic similarity / confidence verification engine (Explainable, General-Purpose)**  
+*Deterministic collapsing of copy-paste and bot amplification, and noise reduction via multi-source corroboration*
+
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
+[![Dependencies](https://img.shields.io/badge/dependencies-zero-success.svg)](#)
+[![Tests](https://github.com/neguseatama/sieve-core/actions/workflows/tests.yml/badge.svg)](https://github.com/neguseatama/sieve-core/actions/workflows/tests.yml)
+
+<!-- NOTE: this badge assumes the repo is named "sieve-core" under the neguseatama account. If you name the repo something else on GitHub, update the URL above to match. -->
+
+---
+
+## 📜 Developer Notes: Why "Deterministic"?
+
+### 1. Background: information volume and noise grow together
+The combination of falling copy-paste costs (digitization), faster amplification (social media), and large-scale summarization/generation (generative AI) means the total volume of information online keeps increasing. At the same time, the share of that volume made up of copy-paste, bot amplification, and unsubstantiated claims is also increasing. Spin control by companies or individuals (flooding a topic with information to shift the conversation) and coordinated bot-driven opinion manipulation are both techniques that exploit this growth in sheer volume. This project was started to provide a practical technical approach to accurately extracting and filtering the true signal of text and data in response to these challenges of information overload and growing noise.
+
+### 2. The problem: "summarization" doesn't solve this
+AI-generated "summarization" compresses information, but if the source material itself contains a lot of noise or bias, the summary inherits that bias. Summarization is also, by nature, a process of "generating a new sentence," which carries an inherent risk of hallucination.
+
+This project instead focuses on "extraction": rather than generating new sentences, it mechanically selects only the elements from an existing data set that satisfy given conditions. This idea is a direct extension of the "sieve of set theory" design used in an earlier project of mine, a music-identification AI called "Sieve-AI."
+
+### 3. Design decision: LLMs are not the decision-maker
+Many existing implementations use an LLM directly to judge "is this true or not." But since an LLM can itself hallucinate, putting the LLM at the center of that judgment creates a fundamental contradiction.
+
+So this project deliberately keeps LLMs and other probabilistic methods out of the decision-making core. Judgments are made entirely through graph theory (Tarjan's strongly connected components), set operations (n-gram Jaccard / overlap coefficient), and a deterministic scoring function based on the count of independent sources — so that a human can trace the entire decision path. This lets us explain "why this conclusion was reached" without going through any probabilistic internal state. (To be clear: being deterministic does not guarantee that the output is true. It is a design choice that guarantees the explainability of the decision process, not the correctness of its content.)
+
+`Sieve-Core` is the core engine that applies the design philosophy of "Sieve-AI"—a "sieve of set theory" inspired by the Sieve of Eratosthenes—to the more general domain of text and data processing.
+
+---
+
+## 💡 Overview
+
+**Sieve-Core** is an open-source **deterministic similarity / confidence verification engine** that uses deterministic mathematical models to collapse copy-paste and bot amplification within a large data set, and surface items that are corroborated by multiple independent sources. Fact-checking is one representative application, not the only one.
+
+Instead of relying on black-box AI judgments, it provides a way to detect and reduce the influence of spin control and bot amplification using **"an algorithm a human can trace and explain, that produces the same result no matter who runs it."** (This is not a tool that judges the truth of content itself — it mechanically visualizes whether or not something is corroborated by multiple independent sources.)
+
+---
+
+## 💎 Core Virtues
+
+1. **Auditability, with no black box**
+   * Every conclusion can be traced by a human through explicit graph structure (SCC) and explicit thresholds (Jaccard / Sigmoid). This design fits domains such as plagiarism detection or fact-checking where the reasoning behind a judgment needs to be explainable (this MVP has only been validated on limited data sets, so any legal or institutional use would require separate validation).
+2. **Zero-Dependencies as a constraint on the core**
+   * The core engine (`sieve_core.py`) is self-contained using only Python's standard library, with no external dependencies. This gives it strong portability and removes, for the core itself, any risk from third-party package vulnerabilities. This does not restrict you from injecting external libraries (e.g. an embedding model) into `density_fn` / `similarity_fn` — see the plugin examples below.
+3. **Determinism as a controller**
+   * A design philosophy in which AI is not the source of the final output, but is instead constrained by the outermost algorithm (a competitive-programming-style approach) acting as an "evaluation function / type converter."
+
+---
+
+## 🛡️ Pipeline Architecture
+
+```
+[ Raw Inputs ] (a large volume of posts, reports, copy-paste, bot data)
+      │
+      ▼
+┌────────────────────────────────────────┐
+│ 1. Density score filter (`density_fn`)  │  ← removes smokescreen text, filler, low-information noise
+└────────────────────────────────────────┘
+      │
+      ▼
+┌────────────────────────────────────────┐
+│ 2. Tarjan SCC graph reduction           │  ← deterministic strongly-connected-components collapse of copy-paste / bot loops
+└────────────────────────────────────────┘
+      │
+      ▼
+┌────────────────────────────────────────┐
+│ 3. Deterministic average-linkage merge  │  ← no probabilistic (LSH) step; merges paraphrases via inter-cluster average similarity
+└────────────────────────────────────────┘
+      │
+      ▼
+┌────────────────────────────────────────┐
+│ 4. Sigmoid confidence scoring           │  ← mathematical scoring based on independent source count (k_roles)
+└────────────────────────────────────────┘
+      │
+      ▼
+[ Verified Outputs ] (verified fact candidates with a confidence score based on independent source count)
+```
+
+---
+
+## 💻 Engine Architecture
+
+The engine itself lives in [`sieve_core.py`](./sieve_core.py) (no external dependencies, single file).
+Pasting the full source into the README would create a second copy that drifts out of sync every time the code changes, so only a minimal usage example is shown here.
+
+```python
+from sieve_core import GenericItemInput, SieveCoreEngine
+
+items = [
+    GenericItemInput(item_id="p1", author_id="user_a", author_role="citizen", content="..."),
+    GenericItemInput(item_id="p2", author_id="reporter_b", author_role="media", content="..."),
+    # ...
+]
+
+engine = SieveCoreEngine()
+results, stats = engine.process(items)
+
+for fact in results:
+    print(fact.fact_id, fact.confidence_score, fact.independent_sources_count)
+```
+
+This snippet is the minimal shape of the API call (the input data shape passed to `process()`, and how results are read back). With only a handful of single, unrelated posts, the independent source count stays at 1, so the confidence score will be low. To see the actual behavior that Sieve-Core is built around — confidence spiking once multiple independent sources corroborate the same fact — run [`example.py`](./example.py).
+
+There are three main classes:
+
+| Class | Role |
+| :--- | :--- |
+| `GenericItemInput` | One unit of input data (ID, author, independent attribute, content) |
+| `SieveCoreEngine` | The 4-stage pipeline body: density filter → SCC reduction → cluster merge → confidence scoring |
+| `FactClusterOutput` | The extraction result (representative text, item count, independent source count, confidence) |
+
+For the internal implementation details (Tarjan SCC, average-linkage Jaccard / overlap coefficient, the sigmoid confidence function, etc.), read `sieve_core.py` directly. Its behavior is backed by two regression test suites:
+
+* `test_sieve_core.py` — behavior of the core engine in isolation (duplicate-ID detection, density filter, SCC reduction, chaining mitigation, empty input, etc.)
+* `test_multi_domain.py` — cross-domain validation across three qualitatively different domains: short-form SNS text, source code, and long-form news articles
+
+```bash
+python3 -m unittest test_sieve_core.py test_multi_domain.py -v
+```
+
+---
+
+## ⚙️ Plugin Extension Examples (Advanced Usage: Custom Evaluators)
+
+`Sieve-Core` lets you inject custom functions into both `density_fn` (information density evaluation) and `similarity_fn` (similarity evaluation), so anything from a morphological analyzer to an LLM, or a code-specific comparison routine, can be plugged in as "an evaluation part that answers to the outermost algorithm."
+
+### Example 1: Injecting an LLM as a lightweight fact-density evaluator
+```python
+def llm_based_density_evaluator(text: str) -> float:
+    # Don't let the LLM summarize or judge truth — have it output only
+    # "what fraction of this text is concrete, factual content (0.0-1.0)"
+    # so its output becomes a controlled input value inside the deterministic pipeline.
+    response = call_mini_llm(f"Rate factual density of this text from 0.0 to 1.0: {text}")
+    try:
+        return float(response.strip())
+    except ValueError:
+        return 0.0
+
+# Inject into the engine
+results, stats = engine.process(items, density_fn=llm_based_density_evaluator)
+```
+
+### Example 2: Injecting density/similarity evaluators for Japanese
+The default `density_fn` (unique-character ratio) and `similarity_fn` (character n-gram Jaccard) are language-agnostic, general-purpose fallbacks, and they do not adequately filter Japanese chit-chat or emotional noise. For real-world use with Japanese text, we recommend injecting a language-specific heuristic (kanji/numeral density) together with a similarity function that strips particles and boilerplate phrases before comparing (see `example.py` for a complete working example).
+
+```python
+import re
+
+STOP_PATTERNS = re.compile(r"(拡散希望|拡散|速報|本日|現在|確認中|発生|です|ます|から|より|で|に|を|が|は|の|と)")
+
+def japanese_density_fn(text: str) -> float:
+    """Heuristic: the higher the ratio of kanji/numerals, the more likely
+    this is a concrete report rather than an opinion or reaction."""
+    if not text:
+        return 0.0
+    return len(re.findall(r"[一-龥0-9]", text)) / len(text)
+
+def japanese_similarity_fn(a: str, b: str) -> float:
+    """2-gram overlap coefficient after stripping particles/boilerplate.
+    We measured that plain Jaccard underestimates similarity for
+    asymmetric-length pairs (a short breaking-news post vs. a long
+    eyewitness account) even when they share the core vocabulary,
+    because the union grows with the extra detail. The overlap
+    coefficient (dividing by the smaller set's size) avoids this."""
+    def grams(text):
+        cleaned = STOP_PATTERNS.sub("", text)
+        return {cleaned[i:i+2] for i in range(len(cleaned) - 1)} if len(cleaned) >= 2 else {cleaned}
+    ga, gb = grams(a), grams(b)
+    if not ga or not gb:
+        return 0.0
+    return len(ga & gb) / min(len(ga), len(gb))
+
+results, stats = engine.process(items, density_fn=japanese_density_fn, similarity_fn=japanese_similarity_fn)
+```
+
+### Example 3: Injecting `similarity_fn` for source-code duplicate detection
+```python
+import difflib
+
+def normalized_code_similarity(a: str, b: str) -> float:
+    def normalize(code: str) -> str:
+        code = re.sub(r"#.*", "", code)          # strip comments
+        code = re.sub(r"\s+", " ", code).strip()  # normalize whitespace
+        return code
+    return difflib.SequenceMatcher(None, normalize(a), normalize(b)).ratio()
+
+# For code, exact_sim_threshold / jaccard_merge_threshold need to be raised.
+# We measured that `def add(a,b): return a+b` and `def multiply(a,b): return a*b`
+# differ by a single operator, yet because they share so much boilerplate
+# (def, parentheses, return), their character-diff ratio reaches 0.75 —
+# high enough that the default thresholds (0.65 / 0.15) would incorrectly
+# merge them into the same fact.
+code_engine = SieveCoreEngine(exact_sim_threshold=0.9, jaccard_merge_threshold=0.8)
+results, stats = code_engine.process(code_items, similarity_fn=normalized_code_similarity)
+```
+
+> **Note**: character-level diff ratios are fundamentally fragile in domains like source
+> code, where boilerplate structure is heavily shared (the example above relies on a narrow
+> 0.75–0.94 gap between thresholds). If you need rigorous code-similarity comparison in
+> production, we recommend injecting an AST-based structural comparison into `similarity_fn`
+> (e.g. matching syntax trees after normalizing function/variable names).
+
+---
+
+## ⚠️ Known Limitations
+
+These are structural limitations of the current MVP (v1.0), found through actual implementation and testing.
+
+1. **Chaining in cluster merging**
+   Single-linkage merging (deciding to merge two clusters based on the *maximum* similarity between any pair of members) can transitively merge unrelated items A and C just because A-B and B-C are each partially similar. The current implementation mitigates this by using average-linkage (deciding based on the *average* similarity across all cross-cluster pairs), but the problem can still resurface depending on threshold tuning.
+2. **Jaccard similarity underestimates asymmetric-length pairs**
+   When one text is much more detailed than the other (e.g. a short breaking-news post vs. a long eyewitness account), Jaccard similarity (`intersection / union`) tends to underestimate similarity because the union grows with the extra length, even when the core vocabulary is shared. We confirmed, empirically, that switching to the overlap coefficient (`intersection / min(set sizes)`) mitigates this (see `test_multi_domain.py`).
+3. **Fragility of similarity scoring in boilerplate-heavy domains (e.g. code)**
+   Character-level or naive n-gram-based Jaccard/diff similarity tends to misfire in domains like source code, which shares a lot of fixed syntax (`def ...`, parentheses, reserved words). You need to inject an AST-based structural comparison into `similarity_fn` for that domain.
+4. **Threshold parameters are domain-dependent**
+   `exact_sim_threshold` and `jaccard_merge_threshold` both require empirical tuning for any given domain; there is no universally safe default. We have validated behavior across three domains (short-form SNS text, source code, long-form news — see `test_multi_domain.py`), but each test case is only a handful of items, and robustness at real-world scale is untested.
+5. **Scale**: the engine currently uses $O(N^2)$ pairwise comparison. Measured (see the benchmark below), processing time grows roughly with $N^2$ as the item count increases, so it stays comfortable up to roughly a few hundred items; past 1,000 items you start seeing wait times of several seconds to tens of seconds. If you need to process on the order of thousands of items in one batch, the pre-filtering block structure described in the roadmap below becomes effectively mandatory.
+
+**Measured benchmark** (a single measurement from a local development environment; actual numbers will vary with your environment and the cost of your `density_fn` / `similarity_fn` implementations. Note also that the true complexity is closer to $O(N^2 \times L)$, where $L$ is average text length — these numbers were measured on English text, which needs roughly 2-3x more characters than Japanese to express the same content, so per-pair comparison cost is correspondingly higher than the Japanese-text benchmark in [README.ja.md](./README.ja.md))
+
+| N (items) | Time (sec) |
+| :--- | :--- |
+| 50 | ~0.04 |
+| 100 | ~0.14 |
+| 250 | ~0.87 |
+| 500 | ~3.5 |
+| 1,000 | ~14.3 |
+| 2,000 | ~57.2 |
+
+Going from N=1,000 to N=2,000 roughly quadruples the processing time, which is consistent with the theoretical $O(N^2)$ scaling. This table is the output of [`benchmark.py`](./benchmark.py); run `python3 benchmark.py` to reproduce it on your own machine.
+
+---
+
+## 🚀 Roadmap (v2.0 Performance Roadmap)
+
+The current MVP (v1.0) prioritizes deterministic precision and uses $O(N^2)$ pairwise comparison. Measured performance is practical up to roughly a few hundred items, but processing time grows noticeably past 1,000 items (see the benchmark table under "Known Limitations"). To scale to tens of thousands of items or more, the following deterministic pre-screening structure is planned:
+
+* **Deterministic Pre-bucket Blocking**
+  * Instead of relying on LSH (probabilistic hashing), pre-partition items into deterministic buckets keyed on low-frequency feature keywords (rare n-grams) or context attributes.
+  * By limiting comparisons to pairs within the same bucket, this compresses the effective complexity to roughly $O(N)$–$O(N \log N)$ while preserving full deterministic observability.
+
+---
+
+## 🛠️ Parameter Guide
+
+| Parameter | Default | Purpose |
+| :--- | :--- | :--- |
+| `density_threshold` | `0.35` | Minimum information density. Filters out smokescreen jargon and emotional noise. |
+| `exact_sim_threshold` | `0.65` | The similarity threshold above which Tarjan SCC treats two items as strongly connected (bot copy-paste / astroturfing). |
+| `jaccard_merge_threshold` | `0.15` | The threshold at which average-linkage merges clusters to account for paraphrasing/variation. |
+| `sigmoid_alpha` | `3.0` | The steepness of the confidence curve's response to an increase in independent role count. |
+| `sigmoid_k0` | `1.5` | The inflection point: the independent role count at which confidence crosses 50%. |
+
+---
+
+## 🌐 Intended Use Cases
+
+`Sieve-Core` is designed to run on the user's own machine, rather than depending on judgments made by a centralized platform.
+
+Concretely, it is intended for cases like: given a large volume of posts or comments, discount the "apparent volume" created by copy-paste and bot amplification, and mechanically narrow down to items corroborated by multiple independent sources (this is not a silver-bullet defense — it is meant as a first-pass filter).
+
+---
+
+## 📄 License
+
+This project is released under the [MIT License](LICENSE).
+
+---
+
+## 🤝 Contributing
+
+Bug reports, feature proposals, and pull requests are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for the project's design principles and the contribution workflow, and [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md) for the standards we hold community interactions to. If you find a security vulnerability, please follow the process in [SECURITY.md](.github/SECURITY.md) rather than filing a public issue.
+
+---
+
+## 👤 Author
+
+* **Kai IWASAKI**

sieve_core_engine-1.0.0/README.md

@@ -0,0 +1,283 @@
+# Sieve-Core (v1.0.0-MVP)
+
+**English | [日本語](./README.ja.md)**
+
+**A deterministic similarity / confidence verification engine (Explainable, General-Purpose)**  
+*Deterministic collapsing of copy-paste and bot amplification, and noise reduction via multi-source corroboration*
+
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
+[![Dependencies](https://img.shields.io/badge/dependencies-zero-success.svg)](#)
+[![Tests](https://github.com/neguseatama/sieve-core/actions/workflows/tests.yml/badge.svg)](https://github.com/neguseatama/sieve-core/actions/workflows/tests.yml)
+
+<!-- NOTE: this badge assumes the repo is named "sieve-core" under the neguseatama account. If you name the repo something else on GitHub, update the URL above to match. -->
+
+---
+
+## 📜 Developer Notes: Why "Deterministic"?
+
+### 1. Background: information volume and noise grow together
+The combination of falling copy-paste costs (digitization), faster amplification (social media), and large-scale summarization/generation (generative AI) means the total volume of information online keeps increasing. At the same time, the share of that volume made up of copy-paste, bot amplification, and unsubstantiated claims is also increasing. Spin control by companies or individuals (flooding a topic with information to shift the conversation) and coordinated bot-driven opinion manipulation are both techniques that exploit this growth in sheer volume. This project was started to provide a practical technical approach to accurately extracting and filtering the true signal of text and data in response to these challenges of information overload and growing noise.
+
+### 2. The problem: "summarization" doesn't solve this
+AI-generated "summarization" compresses information, but if the source material itself contains a lot of noise or bias, the summary inherits that bias. Summarization is also, by nature, a process of "generating a new sentence," which carries an inherent risk of hallucination.
+
+This project instead focuses on "extraction": rather than generating new sentences, it mechanically selects only the elements from an existing data set that satisfy given conditions. This idea is a direct extension of the "sieve of set theory" design used in an earlier project of mine, a music-identification AI called "Sieve-AI."
+
+### 3. Design decision: LLMs are not the decision-maker
+Many existing implementations use an LLM directly to judge "is this true or not." But since an LLM can itself hallucinate, putting the LLM at the center of that judgment creates a fundamental contradiction.
+
+So this project deliberately keeps LLMs and other probabilistic methods out of the decision-making core. Judgments are made entirely through graph theory (Tarjan's strongly connected components), set operations (n-gram Jaccard / overlap coefficient), and a deterministic scoring function based on the count of independent sources — so that a human can trace the entire decision path. This lets us explain "why this conclusion was reached" without going through any probabilistic internal state. (To be clear: being deterministic does not guarantee that the output is true. It is a design choice that guarantees the explainability of the decision process, not the correctness of its content.)
+
+`Sieve-Core` is the core engine that applies the design philosophy of "Sieve-AI"—a "sieve of set theory" inspired by the Sieve of Eratosthenes—to the more general domain of text and data processing.
+
+---
+
+## 💡 Overview
+
+**Sieve-Core** is an open-source **deterministic similarity / confidence verification engine** that uses deterministic mathematical models to collapse copy-paste and bot amplification within a large data set, and surface items that are corroborated by multiple independent sources. Fact-checking is one representative application, not the only one.
+
+Instead of relying on black-box AI judgments, it provides a way to detect and reduce the influence of spin control and bot amplification using **"an algorithm a human can trace and explain, that produces the same result no matter who runs it."** (This is not a tool that judges the truth of content itself — it mechanically visualizes whether or not something is corroborated by multiple independent sources.)
+
+---
+
+## 💎 Core Virtues
+
+1. **Auditability, with no black box**
+   * Every conclusion can be traced by a human through explicit graph structure (SCC) and explicit thresholds (Jaccard / Sigmoid). This design fits domains such as plagiarism detection or fact-checking where the reasoning behind a judgment needs to be explainable (this MVP has only been validated on limited data sets, so any legal or institutional use would require separate validation).
+2. **Zero-Dependencies as a constraint on the core**
+   * The core engine (`sieve_core.py`) is self-contained using only Python's standard library, with no external dependencies. This gives it strong portability and removes, for the core itself, any risk from third-party package vulnerabilities. This does not restrict you from injecting external libraries (e.g. an embedding model) into `density_fn` / `similarity_fn` — see the plugin examples below.
+3. **Determinism as a controller**
+   * A design philosophy in which AI is not the source of the final output, but is instead constrained by the outermost algorithm (a competitive-programming-style approach) acting as an "evaluation function / type converter."
+
+---
+
+## 🛡️ Pipeline Architecture
+
+```
+[ Raw Inputs ] (a large volume of posts, reports, copy-paste, bot data)
+      │
+      ▼
+┌────────────────────────────────────────┐
+│ 1. Density score filter (`density_fn`)  │  ← removes smokescreen text, filler, low-information noise
+└────────────────────────────────────────┘
+      │
+      ▼
+┌────────────────────────────────────────┐
+│ 2. Tarjan SCC graph reduction           │  ← deterministic strongly-connected-components collapse of copy-paste / bot loops
+└────────────────────────────────────────┘
+      │
+      ▼
+┌────────────────────────────────────────┐
+│ 3. Deterministic average-linkage merge  │  ← no probabilistic (LSH) step; merges paraphrases via inter-cluster average similarity
+└────────────────────────────────────────┘
+      │
+      ▼
+┌────────────────────────────────────────┐
+│ 4. Sigmoid confidence scoring           │  ← mathematical scoring based on independent source count (k_roles)
+└────────────────────────────────────────┘
+      │
+      ▼
+[ Verified Outputs ] (verified fact candidates with a confidence score based on independent source count)
+```
+
+---
+
+## 💻 Engine Architecture
+
+The engine itself lives in [`sieve_core.py`](./sieve_core.py) (no external dependencies, single file).
+Pasting the full source into the README would create a second copy that drifts out of sync every time the code changes, so only a minimal usage example is shown here.
+
+```python
+from sieve_core import GenericItemInput, SieveCoreEngine
+
+items = [
+    GenericItemInput(item_id="p1", author_id="user_a", author_role="citizen", content="..."),
+    GenericItemInput(item_id="p2", author_id="reporter_b", author_role="media", content="..."),
+    # ...
+]
+
+engine = SieveCoreEngine()
+results, stats = engine.process(items)
+
+for fact in results:
+    print(fact.fact_id, fact.confidence_score, fact.independent_sources_count)
+```
+
+This snippet is the minimal shape of the API call (the input data shape passed to `process()`, and how results are read back). With only a handful of single, unrelated posts, the independent source count stays at 1, so the confidence score will be low. To see the actual behavior that Sieve-Core is built around — confidence spiking once multiple independent sources corroborate the same fact — run [`example.py`](./example.py).
+
+There are three main classes:
+
+| Class | Role |
+| :--- | :--- |
+| `GenericItemInput` | One unit of input data (ID, author, independent attribute, content) |
+| `SieveCoreEngine` | The 4-stage pipeline body: density filter → SCC reduction → cluster merge → confidence scoring |
+| `FactClusterOutput` | The extraction result (representative text, item count, independent source count, confidence) |
+
+For the internal implementation details (Tarjan SCC, average-linkage Jaccard / overlap coefficient, the sigmoid confidence function, etc.), read `sieve_core.py` directly. Its behavior is backed by two regression test suites:
+
+* `test_sieve_core.py` — behavior of the core engine in isolation (duplicate-ID detection, density filter, SCC reduction, chaining mitigation, empty input, etc.)
+* `test_multi_domain.py` — cross-domain validation across three qualitatively different domains: short-form SNS text, source code, and long-form news articles
+
+```bash
+python3 -m unittest test_sieve_core.py test_multi_domain.py -v
+```
+
+---
+
+## ⚙️ Plugin Extension Examples (Advanced Usage: Custom Evaluators)
+
+`Sieve-Core` lets you inject custom functions into both `density_fn` (information density evaluation) and `similarity_fn` (similarity evaluation), so anything from a morphological analyzer to an LLM, or a code-specific comparison routine, can be plugged in as "an evaluation part that answers to the outermost algorithm."
+
+### Example 1: Injecting an LLM as a lightweight fact-density evaluator
+```python
+def llm_based_density_evaluator(text: str) -> float:
+    # Don't let the LLM summarize or judge truth — have it output only
+    # "what fraction of this text is concrete, factual content (0.0-1.0)"
+    # so its output becomes a controlled input value inside the deterministic pipeline.
+    response = call_mini_llm(f"Rate factual density of this text from 0.0 to 1.0: {text}")
+    try:
+        return float(response.strip())
+    except ValueError:
+        return 0.0
+
+# Inject into the engine
+results, stats = engine.process(items, density_fn=llm_based_density_evaluator)
+```
+
+### Example 2: Injecting density/similarity evaluators for Japanese
+The default `density_fn` (unique-character ratio) and `similarity_fn` (character n-gram Jaccard) are language-agnostic, general-purpose fallbacks, and they do not adequately filter Japanese chit-chat or emotional noise. For real-world use with Japanese text, we recommend injecting a language-specific heuristic (kanji/numeral density) together with a similarity function that strips particles and boilerplate phrases before comparing (see `example.py` for a complete working example).
+
+```python
+import re
+
+STOP_PATTERNS = re.compile(r"(拡散希望|拡散|速報|本日|現在|確認中|発生|です|ます|から|より|で|に|を|が|は|の|と)")
+
+def japanese_density_fn(text: str) -> float:
+    """Heuristic: the higher the ratio of kanji/numerals, the more likely
+    this is a concrete report rather than an opinion or reaction."""
+    if not text:
+        return 0.0
+    return len(re.findall(r"[一-龥0-9]", text)) / len(text)
+
+def japanese_similarity_fn(a: str, b: str) -> float:
+    """2-gram overlap coefficient after stripping particles/boilerplate.
+    We measured that plain Jaccard underestimates similarity for
+    asymmetric-length pairs (a short breaking-news post vs. a long
+    eyewitness account) even when they share the core vocabulary,
+    because the union grows with the extra detail. The overlap
+    coefficient (dividing by the smaller set's size) avoids this."""
+    def grams(text):
+        cleaned = STOP_PATTERNS.sub("", text)
+        return {cleaned[i:i+2] for i in range(len(cleaned) - 1)} if len(cleaned) >= 2 else {cleaned}
+    ga, gb = grams(a), grams(b)
+    if not ga or not gb:
+        return 0.0
+    return len(ga & gb) / min(len(ga), len(gb))
+
+results, stats = engine.process(items, density_fn=japanese_density_fn, similarity_fn=japanese_similarity_fn)
+```
+
+### Example 3: Injecting `similarity_fn` for source-code duplicate detection
+```python
+import difflib
+
+def normalized_code_similarity(a: str, b: str) -> float:
+    def normalize(code: str) -> str:
+        code = re.sub(r"#.*", "", code)          # strip comments
+        code = re.sub(r"\s+", " ", code).strip()  # normalize whitespace
+        return code
+    return difflib.SequenceMatcher(None, normalize(a), normalize(b)).ratio()
+
+# For code, exact_sim_threshold / jaccard_merge_threshold need to be raised.
+# We measured that `def add(a,b): return a+b` and `def multiply(a,b): return a*b`
+# differ by a single operator, yet because they share so much boilerplate
+# (def, parentheses, return), their character-diff ratio reaches 0.75 —
+# high enough that the default thresholds (0.65 / 0.15) would incorrectly
+# merge them into the same fact.
+code_engine = SieveCoreEngine(exact_sim_threshold=0.9, jaccard_merge_threshold=0.8)
+results, stats = code_engine.process(code_items, similarity_fn=normalized_code_similarity)
+```
+
+> **Note**: character-level diff ratios are fundamentally fragile in domains like source
+> code, where boilerplate structure is heavily shared (the example above relies on a narrow
+> 0.75–0.94 gap between thresholds). If you need rigorous code-similarity comparison in
+> production, we recommend injecting an AST-based structural comparison into `similarity_fn`
+> (e.g. matching syntax trees after normalizing function/variable names).
+
+---
+
+## ⚠️ Known Limitations
+
+These are structural limitations of the current MVP (v1.0), found through actual implementation and testing.
+
+1. **Chaining in cluster merging**
+   Single-linkage merging (deciding to merge two clusters based on the *maximum* similarity between any pair of members) can transitively merge unrelated items A and C just because A-B and B-C are each partially similar. The current implementation mitigates this by using average-linkage (deciding based on the *average* similarity across all cross-cluster pairs), but the problem can still resurface depending on threshold tuning.
+2. **Jaccard similarity underestimates asymmetric-length pairs**
+   When one text is much more detailed than the other (e.g. a short breaking-news post vs. a long eyewitness account), Jaccard similarity (`intersection / union`) tends to underestimate similarity because the union grows with the extra length, even when the core vocabulary is shared. We confirmed, empirically, that switching to the overlap coefficient (`intersection / min(set sizes)`) mitigates this (see `test_multi_domain.py`).
+3. **Fragility of similarity scoring in boilerplate-heavy domains (e.g. code)**
+   Character-level or naive n-gram-based Jaccard/diff similarity tends to misfire in domains like source code, which shares a lot of fixed syntax (`def ...`, parentheses, reserved words). You need to inject an AST-based structural comparison into `similarity_fn` for that domain.
+4. **Threshold parameters are domain-dependent**
+   `exact_sim_threshold` and `jaccard_merge_threshold` both require empirical tuning for any given domain; there is no universally safe default. We have validated behavior across three domains (short-form SNS text, source code, long-form news — see `test_multi_domain.py`), but each test case is only a handful of items, and robustness at real-world scale is untested.
+5. **Scale**: the engine currently uses $O(N^2)$ pairwise comparison. Measured (see the benchmark below), processing time grows roughly with $N^2$ as the item count increases, so it stays comfortable up to roughly a few hundred items; past 1,000 items you start seeing wait times of several seconds to tens of seconds. If you need to process on the order of thousands of items in one batch, the pre-filtering block structure described in the roadmap below becomes effectively mandatory.
+
+**Measured benchmark** (a single measurement from a local development environment; actual numbers will vary with your environment and the cost of your `density_fn` / `similarity_fn` implementations. Note also that the true complexity is closer to $O(N^2 \times L)$, where $L$ is average text length — these numbers were measured on English text, which needs roughly 2-3x more characters than Japanese to express the same content, so per-pair comparison cost is correspondingly higher than the Japanese-text benchmark in [README.ja.md](./README.ja.md))
+
+| N (items) | Time (sec) |
+| :--- | :--- |
+| 50 | ~0.04 |
+| 100 | ~0.14 |
+| 250 | ~0.87 |
+| 500 | ~3.5 |
+| 1,000 | ~14.3 |
+| 2,000 | ~57.2 |
+
+Going from N=1,000 to N=2,000 roughly quadruples the processing time, which is consistent with the theoretical $O(N^2)$ scaling. This table is the output of [`benchmark.py`](./benchmark.py); run `python3 benchmark.py` to reproduce it on your own machine.
+
+---
+
+## 🚀 Roadmap (v2.0 Performance Roadmap)
+
+The current MVP (v1.0) prioritizes deterministic precision and uses $O(N^2)$ pairwise comparison. Measured performance is practical up to roughly a few hundred items, but processing time grows noticeably past 1,000 items (see the benchmark table under "Known Limitations"). To scale to tens of thousands of items or more, the following deterministic pre-screening structure is planned:
+
+* **Deterministic Pre-bucket Blocking**
+  * Instead of relying on LSH (probabilistic hashing), pre-partition items into deterministic buckets keyed on low-frequency feature keywords (rare n-grams) or context attributes.
+  * By limiting comparisons to pairs within the same bucket, this compresses the effective complexity to roughly $O(N)$–$O(N \log N)$ while preserving full deterministic observability.
+
+---
+
+## 🛠️ Parameter Guide
+
+| Parameter | Default | Purpose |
+| :--- | :--- | :--- |
+| `density_threshold` | `0.35` | Minimum information density. Filters out smokescreen jargon and emotional noise. |
+| `exact_sim_threshold` | `0.65` | The similarity threshold above which Tarjan SCC treats two items as strongly connected (bot copy-paste / astroturfing). |
+| `jaccard_merge_threshold` | `0.15` | The threshold at which average-linkage merges clusters to account for paraphrasing/variation. |
+| `sigmoid_alpha` | `3.0` | The steepness of the confidence curve's response to an increase in independent role count. |
+| `sigmoid_k0` | `1.5` | The inflection point: the independent role count at which confidence crosses 50%. |
+
+---
+
+## 🌐 Intended Use Cases
+
+`Sieve-Core` is designed to run on the user's own machine, rather than depending on judgments made by a centralized platform.
+
+Concretely, it is intended for cases like: given a large volume of posts or comments, discount the "apparent volume" created by copy-paste and bot amplification, and mechanically narrow down to items corroborated by multiple independent sources (this is not a silver-bullet defense — it is meant as a first-pass filter).
+
+---
+
+## 📄 License
+
+This project is released under the [MIT License](LICENSE).
+
+---
+
+## 🤝 Contributing
+
+Bug reports, feature proposals, and pull requests are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for the project's design principles and the contribution workflow, and [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md) for the standards we hold community interactions to. If you find a security vulnerability, please follow the process in [SECURITY.md](.github/SECURITY.md) rather than filing a public issue.
+
+---
+
+## 👤 Author
+
+* **Kai IWASAKI**

sieve_core_engine-1.0.0/pyproject.toml

@@ -0,0 +1,32 @@
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "sieve-core-engine"
+version = "1.0.0"
+description = "Core logic and base components for the Sieve ecosystem"
+readme = "README.md"
+authors = [
+    { name = "Kai IWASAKI" }
+]
+license = { text = "MIT" }
+classifiers = [
+    "Development Status :: 5 - Production/Stable",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12"
+]
+requires-python = ">=3.9"
+dependencies = []
+
+[project.urls]
+Homepage = "https://github.com/neguseatama/sieve-core"
+Repository = "https://github.com/neguseatama/sieve-core.git"
+
+[tool.setuptools]
+py-modules = ["sieve_core"]

sieve_core_engine-1.0.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

sieve_core_engine-1.0.0/sieve_core.py

@@ -0,0 +1,295 @@
+"""
+Sieve-Core (v1.0.0-MVP)
+Deterministic Fact-Checking & Extraction Engine
+
+Author: Kai IWASAKI
+License: MIT
+Description:
+    A core engine that, in response to the flood of misinformation, copy-paste,
+    and spin control driven by AI (generative AI and black-box judgments),
+    deliberately avoids using AI and instead extracts only verified facts using
+    a fully deterministic mathematical model and graph theory
+    (Tarjan SCC / average-linkage Jaccard).
+"""
+
+import math
+from typing import Callable, List, Dict, Set, Tuple, Any
+
+
+class GenericItemInput:
+    """
+    A single input data element to feed into the Sieve-Core Engine.
+
+    Attributes:
+        item_id (str): unique identifier for this item
+        author_id (str): identifier of the poster/source
+        author_role (str): the poster's independent attribute (e.g. "citizen", "media", "official")
+        content (str): the text content to analyze
+        payload (Any): arbitrary metadata (optional)
+    """
+    def __init__(
+        self,
+        item_id: str,
+        author_id: str,
+        author_role: str,
+        content: str,
+        payload: Any = None
+    ):
+        self.item_id = item_id
+        self.author_id = author_id
+        self.author_role = author_role
+        self.content = content
+        self.payload = payload
+
+
+class FactClusterOutput:
+    """
+    A cluster of facts verified and extracted by the Sieve-Core Engine.
+
+    Attributes:
+        fact_id (str): ID of the extracted fact
+        representative_content (str): the representative (longest) text in the cluster
+        item_ids (List[str]): the input item IDs aggregated into this fact
+        independent_sources_count (int): number of independent verifying sources (roles)
+        confidence_score (float): confidence score computed by the sigmoid function (0.0-1.0)
+    """
+    def __init__(
+        self,
+        fact_id: str,
+        representative_content: str,
+        item_ids: List[str],
+        independent_sources_count: int,
+        confidence_score: float
+    ):
+        self.fact_id = fact_id
+        self.representative_content = representative_content
+        self.item_ids = item_ids
+        self.independent_sources_count = independent_sources_count
+        self.confidence_score = confidence_score
+
+
+class SieveCoreEngine:
+    """
+    A fully deterministic fact-checking / extraction core engine.
+    """
+    def __init__(
+        self,
+        density_threshold: float = 0.35,
+        exact_sim_threshold: float = 0.65,
+        jaccard_merge_threshold: float = 0.15,
+        sigmoid_alpha: float = 3.0,
+        sigmoid_k0: float = 1.5
+    ):
+        """
+        Initialize the engine's various thresholds.
+
+        Args:
+            density_threshold (float): minimum information density; removes low-density
+                text (filler expressions, smokescreen noise).
+            exact_sim_threshold (float): similarity threshold above which Tarjan SCC treats
+                two items as identical wording / a copy-paste cycle.
+            jaccard_merge_threshold (float): threshold at which clusters are merged to
+                account for paraphrasing/variation.
+            sigmoid_alpha (float): the steepness of the confidence sigmoid curve.
+            sigmoid_k0 (float): the inflection point (independent role count) at which
+                confidence crosses 0.5.
+        """
+        self.density_threshold = density_threshold
+        self.exact_sim_threshold = exact_sim_threshold
+        self.jaccard_merge_threshold = jaccard_merge_threshold
+        self.sigmoid_alpha = sigmoid_alpha
+        self.sigmoid_k0 = sigmoid_k0
+
+    def _default_density_fn(self, text: str) -> float:
+        """Default information-density calculation, based on the ratio of unique characters."""
+        if not text:
+            return 0.0
+        return len(set(text)) / len(text)
+
+    def _ngram_extract(self, text: str, n: int = 2) -> Set[str]:
+        """Extract n-gram features."""
+        if len(text) < n:
+            return {text}
+        return {text[i:i + n] for i in range(len(text) - n + 1)}
+
+    def _jaccard_similarity(self, set_a: Set[str], set_b: Set[str]) -> float:
+        """Exact Jaccard similarity computation (O(1)-O(N))."""
+        intersection = len(set_a & set_b)
+        union = len(set_a | set_b)
+        return intersection / union if union > 0 else 0.0
+
+    def calculate_confidence(self, k_roles: int) -> float:
+        """
+        Compute a confidence score (0.0-1.0) from the independent source count (k_roles).
+
+        Formula:
+            f(k) = 1 / (1 + exp(-alpha * (k - k0)))
+
+        This is factored out as its own method rather than inlined in process() so
+        that the sigmoid curve's mathematical properties (monotonic increase, point
+        symmetry around the inflection point, saturating behavior at the extremes)
+        can be unit-tested directly, without going through an end-to-end run.
+        """
+        confidence = 1.0 / (1.0 + math.exp(-self.sigmoid_alpha * (k_roles - self.sigmoid_k0)))
+        return round(confidence, 4)
+
+    def _tarjan_scc(self, nodes: List[str], adj_list: Dict[str, Set[str]]) -> List[List[str]]:
+        """
+        Tarjan's strongly-connected-components (SCC) algorithm, implemented
+        iteratively (an explicit call-stack simulation) instead of recursively,
+        to avoid hitting Python's recursion limit on deep input graphs.
+        """
+        index = 0
+        stack: List[str] = []
+        indices: Dict[str, int] = {}
+        lowlink: Dict[str, int] = {}
+        on_stack: Dict[str, bool] = {node: False for node in nodes}
+        sccs: List[List[str]] = []
+
+        for node in nodes:
+            if node not in indices:
+                work_stack = [(node, 0, list(adj_list.get(node, [])))]
+                indices[node] = index
+                lowlink[node] = index
+                index += 1
+                stack.append(node)
+                on_stack[node] = True
+
+                while work_stack:
+                    v, i, neighbors = work_stack[-1]
+                    if i < len(neighbors):
+                        w = neighbors[i]
+                        work_stack[-1] = (v, i + 1, neighbors)
+                        if w not in indices:
+                            indices[w] = index
+                            lowlink[w] = index
+                            index += 1
+                            stack.append(w)
+                            on_stack[w] = True
+                            work_stack.append((w, 0, list(adj_list.get(w, []))))
+                        elif on_stack[w]:
+                            lowlink[v] = min(lowlink[v], indices[w])
+                    else:
+                        work_stack.pop()
+                        if work_stack:
+                            parent = work_stack[-1][0]
+                            lowlink[parent] = min(lowlink[parent], lowlink[v])
+
+                        if lowlink[v] == indices[v]:
+                            scc = []
+                            while True:
+                                w = stack.pop()
+                                on_stack[w] = False
+                                scc.append(w)
+                                if w == v:
+                                    break
+                            sccs.append(scc)
+        return sccs
+
+    def process(
+        self,
+        items: List[GenericItemInput],
+        density_fn: Callable[[str], float] = None,
+        similarity_fn: Callable[[str, str], float] = None
+    ) -> Tuple[List[FactClusterOutput], Dict[str, Any]]:
+        """
+        Run the 4-stage pipeline to extract and score facts.
+        """
+        if density_fn is None:
+            density_fn = self._default_density_fn
+        if similarity_fn is None:
+            similarity_fn = lambda a, b: self._jaccard_similarity(
+                self._ngram_extract(a), self._ngram_extract(b)
+            )
+
+        # 1. Density score filter
+        filtered_items = [item for item in items if density_fn(item.content) >= self.density_threshold]
+
+        # Explicitly detect duplicate item_ids (to preserve full auditability).
+        seen_ids: Set[str] = set()
+        duplicate_ids: Set[str] = set()
+        for item in filtered_items:
+            if item.item_id in seen_ids:
+                duplicate_ids.add(item.item_id)
+            seen_ids.add(item.item_id)
+        if duplicate_ids:
+            raise ValueError(
+                f"Duplicate item_id(s) found: {sorted(duplicate_ids)}."
+                " item_id must be unique across all input items."
+            )
+
+        item_map = {item.item_id: item for item in filtered_items}
+        node_ids = list(item_map.keys())
+
+        # 2. Build a directed graph & collapse copy-paste/bot-amplification cycles via Tarjan SCC
+        adj_list: Dict[str, Set[str]] = {nid: set() for nid in node_ids}
+
+        for i in range(len(node_ids)):
+            for j in range(i + 1, len(node_ids)):
+                id_a, id_b = node_ids[i], node_ids[j]
+                sim = similarity_fn(item_map[id_a].content, item_map[id_b].content)
+                if sim >= self.exact_sim_threshold:
+                    adj_list[id_a].add(id_b)
+                    adj_list[id_b].add(id_a)
+
+        scc_clusters = self._tarjan_scc(node_ids, adj_list)
+
+        # 3. Deterministic cluster merging (average-linkage, to mitigate chaining)
+        clusters: List[Set[str]] = [set(scc) for scc in scc_clusters]
+        merged = True
+        while merged:
+            merged = False
+            new_clusters: List[Set[str]] = []
+            visited = [False] * len(clusters)
+
+            for i in range(len(clusters)):
+                if visited[i]:
+                    continue
+                current_cluster = set(clusters[i])
+                for j in range(i + 1, len(clusters)):
+                    if visited[j]:
+                        continue
+                    total_sim = sum(
+                        similarity_fn(item_map[id_a].content, item_map[id_b].content)
+                        for id_a in current_cluster
+                        for id_b in clusters[j]
+                    )
+                    avg_sim = total_sim / (len(current_cluster) * len(clusters[j]))
+                    if avg_sim >= self.jaccard_merge_threshold:
+                        current_cluster.update(clusters[j])
+                        visited[j] = True
+                        merged = True
+
+                visited[i] = True
+                new_clusters.append(current_cluster)
+            clusters = new_clusters
+
+        # 4. Sigmoid confidence scoring and result assembly
+        results: List[FactClusterOutput] = []
+        for idx, cluster in enumerate(clusters):
+            cluster_items = [item_map[nid] for nid in cluster]
+            unique_roles = {item.author_role for item in cluster_items}
+            k_roles = len(unique_roles)
+
+            confidence = self.calculate_confidence(k_roles)
+            rep_item = max(cluster_items, key=lambda x: len(x.content))
+
+            results.append(
+                FactClusterOutput(
+                    fact_id=f"FACT-{idx+1:03d}",
+                    representative_content=rep_item.content,
+                    item_ids=[item.item_id for item in cluster_items],
+                    independent_sources_count=k_roles,
+                    confidence_score=round(confidence, 4),
+                )
+            )
+
+        results.sort(key=lambda x: x.confidence_score, reverse=True)
+
+        stats = {
+            "raw_input_count": len(items),
+            "filtered_input_count": len(filtered_items),
+            "extracted_facts_count": len(results),
+        }
+
+        return results, stats

sieve_core_engine-1.0.0/sieve_core_engine.egg-info/PKG-INFO

@@ -0,0 +1,304 @@
+Metadata-Version: 2.4
+Name: sieve-core-engine
+Version: 1.0.0
+Summary: Core logic and base components for the Sieve ecosystem
+Author: Kai IWASAKI
+License: MIT
+Project-URL: Homepage, https://github.com/neguseatama/sieve-core
+Project-URL: Repository, https://github.com/neguseatama/sieve-core.git
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# Sieve-Core (v1.0.0-MVP)
+
+**English | [日本語](./README.ja.md)**
+
+**A deterministic similarity / confidence verification engine (Explainable, General-Purpose)**  
+*Deterministic collapsing of copy-paste and bot amplification, and noise reduction via multi-source corroboration*
+
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
+[![Dependencies](https://img.shields.io/badge/dependencies-zero-success.svg)](#)
+[![Tests](https://github.com/neguseatama/sieve-core/actions/workflows/tests.yml/badge.svg)](https://github.com/neguseatama/sieve-core/actions/workflows/tests.yml)
+
+<!-- NOTE: this badge assumes the repo is named "sieve-core" under the neguseatama account. If you name the repo something else on GitHub, update the URL above to match. -->
+
+---
+
+## 📜 Developer Notes: Why "Deterministic"?
+
+### 1. Background: information volume and noise grow together
+The combination of falling copy-paste costs (digitization), faster amplification (social media), and large-scale summarization/generation (generative AI) means the total volume of information online keeps increasing. At the same time, the share of that volume made up of copy-paste, bot amplification, and unsubstantiated claims is also increasing. Spin control by companies or individuals (flooding a topic with information to shift the conversation) and coordinated bot-driven opinion manipulation are both techniques that exploit this growth in sheer volume. This project was started to provide a practical technical approach to accurately extracting and filtering the true signal of text and data in response to these challenges of information overload and growing noise.
+
+### 2. The problem: "summarization" doesn't solve this
+AI-generated "summarization" compresses information, but if the source material itself contains a lot of noise or bias, the summary inherits that bias. Summarization is also, by nature, a process of "generating a new sentence," which carries an inherent risk of hallucination.
+
+This project instead focuses on "extraction": rather than generating new sentences, it mechanically selects only the elements from an existing data set that satisfy given conditions. This idea is a direct extension of the "sieve of set theory" design used in an earlier project of mine, a music-identification AI called "Sieve-AI."
+
+### 3. Design decision: LLMs are not the decision-maker
+Many existing implementations use an LLM directly to judge "is this true or not." But since an LLM can itself hallucinate, putting the LLM at the center of that judgment creates a fundamental contradiction.
+
+So this project deliberately keeps LLMs and other probabilistic methods out of the decision-making core. Judgments are made entirely through graph theory (Tarjan's strongly connected components), set operations (n-gram Jaccard / overlap coefficient), and a deterministic scoring function based on the count of independent sources — so that a human can trace the entire decision path. This lets us explain "why this conclusion was reached" without going through any probabilistic internal state. (To be clear: being deterministic does not guarantee that the output is true. It is a design choice that guarantees the explainability of the decision process, not the correctness of its content.)
+
+`Sieve-Core` is the core engine that applies the design philosophy of "Sieve-AI"—a "sieve of set theory" inspired by the Sieve of Eratosthenes—to the more general domain of text and data processing.
+
+---
+
+## 💡 Overview
+
+**Sieve-Core** is an open-source **deterministic similarity / confidence verification engine** that uses deterministic mathematical models to collapse copy-paste and bot amplification within a large data set, and surface items that are corroborated by multiple independent sources. Fact-checking is one representative application, not the only one.
+
+Instead of relying on black-box AI judgments, it provides a way to detect and reduce the influence of spin control and bot amplification using **"an algorithm a human can trace and explain, that produces the same result no matter who runs it."** (This is not a tool that judges the truth of content itself — it mechanically visualizes whether or not something is corroborated by multiple independent sources.)
+
+---
+
+## 💎 Core Virtues
+
+1. **Auditability, with no black box**
+   * Every conclusion can be traced by a human through explicit graph structure (SCC) and explicit thresholds (Jaccard / Sigmoid). This design fits domains such as plagiarism detection or fact-checking where the reasoning behind a judgment needs to be explainable (this MVP has only been validated on limited data sets, so any legal or institutional use would require separate validation).
+2. **Zero-Dependencies as a constraint on the core**
+   * The core engine (`sieve_core.py`) is self-contained using only Python's standard library, with no external dependencies. This gives it strong portability and removes, for the core itself, any risk from third-party package vulnerabilities. This does not restrict you from injecting external libraries (e.g. an embedding model) into `density_fn` / `similarity_fn` — see the plugin examples below.
+3. **Determinism as a controller**
+   * A design philosophy in which AI is not the source of the final output, but is instead constrained by the outermost algorithm (a competitive-programming-style approach) acting as an "evaluation function / type converter."
+
+---
+
+## 🛡️ Pipeline Architecture
+
+```
+[ Raw Inputs ] (a large volume of posts, reports, copy-paste, bot data)
+      │
+      ▼
+┌────────────────────────────────────────┐
+│ 1. Density score filter (`density_fn`)  │  ← removes smokescreen text, filler, low-information noise
+└────────────────────────────────────────┘
+      │
+      ▼
+┌────────────────────────────────────────┐
+│ 2. Tarjan SCC graph reduction           │  ← deterministic strongly-connected-components collapse of copy-paste / bot loops
+└────────────────────────────────────────┘
+      │
+      ▼
+┌────────────────────────────────────────┐
+│ 3. Deterministic average-linkage merge  │  ← no probabilistic (LSH) step; merges paraphrases via inter-cluster average similarity
+└────────────────────────────────────────┘
+      │
+      ▼
+┌────────────────────────────────────────┐
+│ 4. Sigmoid confidence scoring           │  ← mathematical scoring based on independent source count (k_roles)
+└────────────────────────────────────────┘
+      │
+      ▼
+[ Verified Outputs ] (verified fact candidates with a confidence score based on independent source count)
+```
+
+---
+
+## 💻 Engine Architecture
+
+The engine itself lives in [`sieve_core.py`](./sieve_core.py) (no external dependencies, single file).
+Pasting the full source into the README would create a second copy that drifts out of sync every time the code changes, so only a minimal usage example is shown here.
+
+```python
+from sieve_core import GenericItemInput, SieveCoreEngine
+
+items = [
+    GenericItemInput(item_id="p1", author_id="user_a", author_role="citizen", content="..."),
+    GenericItemInput(item_id="p2", author_id="reporter_b", author_role="media", content="..."),
+    # ...
+]
+
+engine = SieveCoreEngine()
+results, stats = engine.process(items)
+
+for fact in results:
+    print(fact.fact_id, fact.confidence_score, fact.independent_sources_count)
+```
+
+This snippet is the minimal shape of the API call (the input data shape passed to `process()`, and how results are read back). With only a handful of single, unrelated posts, the independent source count stays at 1, so the confidence score will be low. To see the actual behavior that Sieve-Core is built around — confidence spiking once multiple independent sources corroborate the same fact — run [`example.py`](./example.py).
+
+There are three main classes:
+
+| Class | Role |
+| :--- | :--- |
+| `GenericItemInput` | One unit of input data (ID, author, independent attribute, content) |
+| `SieveCoreEngine` | The 4-stage pipeline body: density filter → SCC reduction → cluster merge → confidence scoring |
+| `FactClusterOutput` | The extraction result (representative text, item count, independent source count, confidence) |
+
+For the internal implementation details (Tarjan SCC, average-linkage Jaccard / overlap coefficient, the sigmoid confidence function, etc.), read `sieve_core.py` directly. Its behavior is backed by two regression test suites:
+
+* `test_sieve_core.py` — behavior of the core engine in isolation (duplicate-ID detection, density filter, SCC reduction, chaining mitigation, empty input, etc.)
+* `test_multi_domain.py` — cross-domain validation across three qualitatively different domains: short-form SNS text, source code, and long-form news articles
+
+```bash
+python3 -m unittest test_sieve_core.py test_multi_domain.py -v
+```
+
+---
+
+## ⚙️ Plugin Extension Examples (Advanced Usage: Custom Evaluators)
+
+`Sieve-Core` lets you inject custom functions into both `density_fn` (information density evaluation) and `similarity_fn` (similarity evaluation), so anything from a morphological analyzer to an LLM, or a code-specific comparison routine, can be plugged in as "an evaluation part that answers to the outermost algorithm."
+
+### Example 1: Injecting an LLM as a lightweight fact-density evaluator
+```python
+def llm_based_density_evaluator(text: str) -> float:
+    # Don't let the LLM summarize or judge truth — have it output only
+    # "what fraction of this text is concrete, factual content (0.0-1.0)"
+    # so its output becomes a controlled input value inside the deterministic pipeline.
+    response = call_mini_llm(f"Rate factual density of this text from 0.0 to 1.0: {text}")
+    try:
+        return float(response.strip())
+    except ValueError:
+        return 0.0
+
+# Inject into the engine
+results, stats = engine.process(items, density_fn=llm_based_density_evaluator)
+```
+
+### Example 2: Injecting density/similarity evaluators for Japanese
+The default `density_fn` (unique-character ratio) and `similarity_fn` (character n-gram Jaccard) are language-agnostic, general-purpose fallbacks, and they do not adequately filter Japanese chit-chat or emotional noise. For real-world use with Japanese text, we recommend injecting a language-specific heuristic (kanji/numeral density) together with a similarity function that strips particles and boilerplate phrases before comparing (see `example.py` for a complete working example).
+
+```python
+import re
+
+STOP_PATTERNS = re.compile(r"(拡散希望|拡散|速報|本日|現在|確認中|発生|です|ます|から|より|で|に|を|が|は|の|と)")
+
+def japanese_density_fn(text: str) -> float:
+    """Heuristic: the higher the ratio of kanji/numerals, the more likely
+    this is a concrete report rather than an opinion or reaction."""
+    if not text:
+        return 0.0
+    return len(re.findall(r"[一-龥0-9]", text)) / len(text)
+
+def japanese_similarity_fn(a: str, b: str) -> float:
+    """2-gram overlap coefficient after stripping particles/boilerplate.
+    We measured that plain Jaccard underestimates similarity for
+    asymmetric-length pairs (a short breaking-news post vs. a long
+    eyewitness account) even when they share the core vocabulary,
+    because the union grows with the extra detail. The overlap
+    coefficient (dividing by the smaller set's size) avoids this."""
+    def grams(text):
+        cleaned = STOP_PATTERNS.sub("", text)
+        return {cleaned[i:i+2] for i in range(len(cleaned) - 1)} if len(cleaned) >= 2 else {cleaned}
+    ga, gb = grams(a), grams(b)
+    if not ga or not gb:
+        return 0.0
+    return len(ga & gb) / min(len(ga), len(gb))
+
+results, stats = engine.process(items, density_fn=japanese_density_fn, similarity_fn=japanese_similarity_fn)
+```
+
+### Example 3: Injecting `similarity_fn` for source-code duplicate detection
+```python
+import difflib
+
+def normalized_code_similarity(a: str, b: str) -> float:
+    def normalize(code: str) -> str:
+        code = re.sub(r"#.*", "", code)          # strip comments
+        code = re.sub(r"\s+", " ", code).strip()  # normalize whitespace
+        return code
+    return difflib.SequenceMatcher(None, normalize(a), normalize(b)).ratio()
+
+# For code, exact_sim_threshold / jaccard_merge_threshold need to be raised.
+# We measured that `def add(a,b): return a+b` and `def multiply(a,b): return a*b`
+# differ by a single operator, yet because they share so much boilerplate
+# (def, parentheses, return), their character-diff ratio reaches 0.75 —
+# high enough that the default thresholds (0.65 / 0.15) would incorrectly
+# merge them into the same fact.
+code_engine = SieveCoreEngine(exact_sim_threshold=0.9, jaccard_merge_threshold=0.8)
+results, stats = code_engine.process(code_items, similarity_fn=normalized_code_similarity)
+```
+
+> **Note**: character-level diff ratios are fundamentally fragile in domains like source
+> code, where boilerplate structure is heavily shared (the example above relies on a narrow
+> 0.75–0.94 gap between thresholds). If you need rigorous code-similarity comparison in
+> production, we recommend injecting an AST-based structural comparison into `similarity_fn`
+> (e.g. matching syntax trees after normalizing function/variable names).
+
+---
+
+## ⚠️ Known Limitations
+
+These are structural limitations of the current MVP (v1.0), found through actual implementation and testing.
+
+1. **Chaining in cluster merging**
+   Single-linkage merging (deciding to merge two clusters based on the *maximum* similarity between any pair of members) can transitively merge unrelated items A and C just because A-B and B-C are each partially similar. The current implementation mitigates this by using average-linkage (deciding based on the *average* similarity across all cross-cluster pairs), but the problem can still resurface depending on threshold tuning.
+2. **Jaccard similarity underestimates asymmetric-length pairs**
+   When one text is much more detailed than the other (e.g. a short breaking-news post vs. a long eyewitness account), Jaccard similarity (`intersection / union`) tends to underestimate similarity because the union grows with the extra length, even when the core vocabulary is shared. We confirmed, empirically, that switching to the overlap coefficient (`intersection / min(set sizes)`) mitigates this (see `test_multi_domain.py`).
+3. **Fragility of similarity scoring in boilerplate-heavy domains (e.g. code)**
+   Character-level or naive n-gram-based Jaccard/diff similarity tends to misfire in domains like source code, which shares a lot of fixed syntax (`def ...`, parentheses, reserved words). You need to inject an AST-based structural comparison into `similarity_fn` for that domain.
+4. **Threshold parameters are domain-dependent**
+   `exact_sim_threshold` and `jaccard_merge_threshold` both require empirical tuning for any given domain; there is no universally safe default. We have validated behavior across three domains (short-form SNS text, source code, long-form news — see `test_multi_domain.py`), but each test case is only a handful of items, and robustness at real-world scale is untested.
+5. **Scale**: the engine currently uses $O(N^2)$ pairwise comparison. Measured (see the benchmark below), processing time grows roughly with $N^2$ as the item count increases, so it stays comfortable up to roughly a few hundred items; past 1,000 items you start seeing wait times of several seconds to tens of seconds. If you need to process on the order of thousands of items in one batch, the pre-filtering block structure described in the roadmap below becomes effectively mandatory.
+
+**Measured benchmark** (a single measurement from a local development environment; actual numbers will vary with your environment and the cost of your `density_fn` / `similarity_fn` implementations. Note also that the true complexity is closer to $O(N^2 \times L)$, where $L$ is average text length — these numbers were measured on English text, which needs roughly 2-3x more characters than Japanese to express the same content, so per-pair comparison cost is correspondingly higher than the Japanese-text benchmark in [README.ja.md](./README.ja.md))
+
+| N (items) | Time (sec) |
+| :--- | :--- |
+| 50 | ~0.04 |
+| 100 | ~0.14 |
+| 250 | ~0.87 |
+| 500 | ~3.5 |
+| 1,000 | ~14.3 |
+| 2,000 | ~57.2 |
+
+Going from N=1,000 to N=2,000 roughly quadruples the processing time, which is consistent with the theoretical $O(N^2)$ scaling. This table is the output of [`benchmark.py`](./benchmark.py); run `python3 benchmark.py` to reproduce it on your own machine.
+
+---
+
+## 🚀 Roadmap (v2.0 Performance Roadmap)
+
+The current MVP (v1.0) prioritizes deterministic precision and uses $O(N^2)$ pairwise comparison. Measured performance is practical up to roughly a few hundred items, but processing time grows noticeably past 1,000 items (see the benchmark table under "Known Limitations"). To scale to tens of thousands of items or more, the following deterministic pre-screening structure is planned:
+
+* **Deterministic Pre-bucket Blocking**
+  * Instead of relying on LSH (probabilistic hashing), pre-partition items into deterministic buckets keyed on low-frequency feature keywords (rare n-grams) or context attributes.
+  * By limiting comparisons to pairs within the same bucket, this compresses the effective complexity to roughly $O(N)$–$O(N \log N)$ while preserving full deterministic observability.
+
+---
+
+## 🛠️ Parameter Guide
+
+| Parameter | Default | Purpose |
+| :--- | :--- | :--- |
+| `density_threshold` | `0.35` | Minimum information density. Filters out smokescreen jargon and emotional noise. |
+| `exact_sim_threshold` | `0.65` | The similarity threshold above which Tarjan SCC treats two items as strongly connected (bot copy-paste / astroturfing). |
+| `jaccard_merge_threshold` | `0.15` | The threshold at which average-linkage merges clusters to account for paraphrasing/variation. |
+| `sigmoid_alpha` | `3.0` | The steepness of the confidence curve's response to an increase in independent role count. |
+| `sigmoid_k0` | `1.5` | The inflection point: the independent role count at which confidence crosses 50%. |
+
+---
+
+## 🌐 Intended Use Cases
+
+`Sieve-Core` is designed to run on the user's own machine, rather than depending on judgments made by a centralized platform.
+
+Concretely, it is intended for cases like: given a large volume of posts or comments, discount the "apparent volume" created by copy-paste and bot amplification, and mechanically narrow down to items corroborated by multiple independent sources (this is not a silver-bullet defense — it is meant as a first-pass filter).
+
+---
+
+## 📄 License
+
+This project is released under the [MIT License](LICENSE).
+
+---
+
+## 🤝 Contributing
+
+Bug reports, feature proposals, and pull requests are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for the project's design principles and the contribution workflow, and [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md) for the standards we hold community interactions to. If you find a security vulnerability, please follow the process in [SECURITY.md](.github/SECURITY.md) rather than filing a public issue.
+
+---
+
+## 👤 Author
+
+* **Kai IWASAKI**

sieve_core_engine-1.0.0/sieve_core_engine.egg-info/SOURCES.txt

@@ -0,0 +1,8 @@
+LICENSE
+README.md
+pyproject.toml
+sieve_core.py
+sieve_core_engine.egg-info/PKG-INFO
+sieve_core_engine.egg-info/SOURCES.txt
+sieve_core_engine.egg-info/dependency_links.txt
+sieve_core_engine.egg-info/top_level.txt

sieve_core_engine-1.0.0/sieve_core_engine.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

sieve_core_engine-1.0.0/sieve_core_engine.egg-info/top_level.txt

@@ -0,0 +1 @@
+sieve_core