gibsondedup 0.1.0__tar.gz

gibsondedup-0.1.0/PKG-INFO

@@ -0,0 +1,322 @@
+Metadata-Version: 2.4
+Name: gibsondedup
+Version: 0.1.0
+Summary: A high-performance search result deduplication engine that groups duplicate results by canonical URL and semantic similarity, returning clean batches with the most authoritative source and full traceability to all original sources.
+Author: Gibson Kwabena Aseda Mensah
+Author-email: logosyninc@gmail.com
+Classifier: Programming Language :: Python :: 3
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Dynamic: author-email
+Dynamic: requires-python
+
+# gibsondedup
+
+A high-performance search result deduplication engine that groups duplicate results by canonical URL and semantic similarity, returning clean batches with the most authoritative source and full traceability to all original sources.
+
+---
+
+## The Problem
+
+When aggregating search results from multiple providers (Google, Bing, DuckDuckGo), 40–60% of results are duplicates — same content, different URLs, different titles, different tracking parameters.
+
+```
+Source 1 (Google):
+  url: https://www.python.org/docs?utm_source=google
+  title: "Python Official Documentation"
+
+Source 2 (Bing):
+  url: http://python.org/docs/
+  title: "Python Docs"
+
+Source 3 (DuckDuckGo):
+  url: https://python.org/docs?utm_medium=cpc
+  title: "Learn Python - Official Docs"
+```
+
+These are the same resource. Without deduplication, users see noise instead of signal.
+
+---
+
+## The Solution
+
+`gibsondedup` processes raw search results through a multi-stage pipeline:
+
+```
+Raw Results
+    ↓
+Parse & Validate         (malformed records skipped gracefully)
+    ↓
+URL Normalization        (remove tracking params, www, trailing slashes)
+    ↓
+Exact URL Grouping       (hash-based, O(n) complexity)
+    ↓
+Semantic Merge           (Jaccard similarity, configurable threshold)
+    ↓
+Canonicalization         (select best result per group)
+    ↓
+Clean Output             (deduplicated results + metadata)
+```
+
+**Result:** 100 noisy inputs → 60–70 clean, canonical results.
+
+---
+
+## Installation
+
+```bash
+pip install gibsondedup
+```
+
+---
+
+## Quick Start
+
+```python
+from app.orchestration.pipeline import DeduplicationEngine
+
+engine = DeduplicationEngine()
+
+results = engine.process([
+    {
+        "title": "Python Official Documentation",
+        "url": "https://www.python.org/docs?utm_source=google",
+        "description": "Official Python docs",
+        "source": "google"
+    },
+    {
+        "title": "Python Docs",
+        "url": "http://python.org/docs/",
+        "description": "Python documentation",
+        "source": "bing"
+    },
+    {
+        "title": "Stack Overflow Python",
+        "url": "https://stackoverflow.com/questions/tagged/python",
+        "description": "Python questions",
+        "source": "google"
+    },
+])
+
+print(results)
+```
+
+**Output:**
+
+```json
+{
+  "results": [
+    {
+      "title": "Python Official Documentation",
+      "url": "https://www.python.org/docs?utm_source=google",
+      "canonical_url": "python.org/docs",
+      "description": "Official Python docs",
+      "sources": ["google", "bing"],
+      "duplicates_removed": 1
+    },
+    {
+      "title": "Stack Overflow Python",
+      "url": "https://stackoverflow.com/questions/tagged/python",
+      "canonical_url": "stackoverflow.com/questions/tagged/python",
+      "description": "Python questions",
+      "sources": ["google"],
+      "duplicates_removed": 0
+    }
+  ],
+  "meta": {
+    "total_input": 3,
+    "total_parsed": 3,
+    "total_output": 2,
+    "duplicates_removed": 1,
+    "processing_time_ms": 0.09,
+    "similarity_threshold": 0.8
+  }
+}
+```
+
+---
+
+## Configuration
+
+### Custom Similarity Threshold
+
+Control how aggressively similar-titled results are merged:
+
+```python
+# Conservative (default) — only merge highly similar titles
+engine = DeduplicationEngine(similarity_threshold=0.8)
+
+# Moderate — merge titles with moderate overlap
+engine = DeduplicationEngine(similarity_threshold=0.5)
+
+# Aggressive — merge titles with minimal overlap
+engine = DeduplicationEngine(similarity_threshold=0.3)
+```
+
+### Input Contract
+
+Each result in the input array accepts:
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `title` | string | ✅ | Result title |
+| `url` | string | ✅ | Result URL |
+| `description` | string | ❌ | Result description |
+| `source` | string | ❌ | Source provider (google, bing, etc.) |
+
+Malformed records (missing title or URL) are **skipped gracefully** — the pipeline continues processing valid records.
+
+---
+
+## Architecture
+
+### Pipeline Stages
+
+**Stage 1: Parser**
+Converts raw JSON payloads into structured internal contracts. Validates required fields. Skips malformed records without crashing.
+
+**Stage 2: URL Normalizer**
+Converts URLs to canonical form by:
+- Removing protocol (`http://`, `https://`)
+- Removing `www` prefix
+- Removing tracking parameters (`utm_*`, `fbclid`, `gclid`)
+- Removing trailing slashes
+- Stripping default ports (80, 443)
+- Preserving meaningful query parameters
+- Lowercasing everything
+
+**Stage 3: Exact Grouping**
+Groups results by normalized canonical URL using a hash map. Time complexity: O(n). Avoids the O(n²) pairwise comparison trap.
+
+**Stage 4: Semantic Merger**
+Within groups sharing the same domain, compares representative titles using Jaccard similarity. Groups exceeding the similarity threshold are merged into one. Only compares within the same exact domain — `python.org` and `docs.python.org` are treated as separate domains.
+
+**Stage 5: Canonicalizer**
+For each group, selects the single best result using a combined title + description length heuristic. Collects source traceability from all merged results.
+
+### Data Contracts
+
+```python
+# Input
+RawSearchResult(title, url, description?, source?)
+
+# Internal
+NormalizedSearchResult(title, url, canonical_url, description, source)
+
+# Output
+CanonicalResult(title, url, canonical_url, description, sources[], duplicates_removed)
+```
+
+### System Invariants
+
+1. Same normalized URL → same duplicate group (always)
+2. Canonical results preserve source traceability (always)
+3. Pipeline stages do not mutate upstream data (always)
+4. Normalization happens once per result (always)
+
+---
+
+## Engineering Decisions
+
+### Why hash-based grouping (O(n)) over pairwise comparison (O(n²))?
+
+With 1000 results, O(n²) means 1,000,000 comparisons. O(n) means 1000. At scale this is the difference between milliseconds and seconds.
+
+### Why exact domain matching for semantic merging?
+
+`python.org/docs` and `docs.python.org` serve genuinely different content despite sharing a base domain. Exact domain matching prevents false merges while still catching real duplicates like `python.org/docs` and `python.org/reference`.
+
+### Why Jaccard similarity over Levenshtein distance?
+
+Jaccard operates on token sets (words), making it robust to word order changes and additions. Levenshtein operates on character sequences, making it sensitive to trivial differences like "Docs" vs "Documentation".
+
+### Why keep normalization deterministic?
+
+Same input must always produce same output. This ensures:
+- Results are reproducible across runs
+- Caching normalized URLs is safe
+- Tests are reliable and meaningful
+
+---
+
+## Performance
+
+| Input Size | Processing Time | Memory |
+|------------|----------------|--------|
+| 100 results | ~1ms | ~1MB |
+| 1,000 results | ~10ms | ~5MB |
+| 10,000 results | ~100ms | ~50MB |
+
+*Benchmarked on Ubuntu 22.04, Python 3.10, Intel i5*
+
+---
+
+## Testing
+
+```bash
+# Install dev dependencies
+pip install pytest pytest-cov
+
+# Run all tests
+pytest tests/ -v
+
+# Run with coverage
+pytest tests/ --cov=app --cov-report=term-missing
+```
+
+**Test coverage: 59 tests across 5 modules**
+
+```
+tests/test_normalizer.py    14 tests  (URL normalization)
+tests/test_grouping.py       4 tests  (exact grouping)
+tests/test_canonicalizer.py  7 tests  (best result selection)
+tests/test_merger.py        11 tests  (semantic merging)
+tests/test_similarity.py    13 tests  (Jaccard similarity)
+tests/test_pipeline.py      10 tests  (end-to-end pipeline)
+```
+
+---
+
+## Roadmap
+
+### Phase 1 (Complete)
+- URL normalization
+- Exact duplicate grouping
+- Canonicalization with source traceability
+
+### Phase 2 (Complete)
+- Jaccard similarity engine
+- Domain-based semantic merging
+- Configurable similarity threshold
+
+### Phase 3 (Planned)
+- Persistent caching (Redis)
+- Database storage (PostgreSQL)
+- REST API exposure
+- Rails wrapper (gem)
+
+### Phase 4 (Planned)
+- Semantic embeddings (replace Jaccard with vector similarity)
+- Distributed processing
+- Production observability
+
+---
+
+## License
+
+MIT License. See `LICENSE` file.
+
+---
+
+## Author
+
+**Aseda Gibson**
+Computer Engineering, University of Energy and Natural Resources (UENR), Ghana.
+Backend systems, distributed architecture, infrastructure tooling.
+
+GitHub: [github.com/asedagibson](https://github.com/asedagibson)
+
+---
+
+*Built with correctness-oriented engineering. Every architectural decision is documented. Every invariant is tested.*

gibsondedup-0.1.0/README.md

@@ -0,0 +1,309 @@
+# gibsondedup
+
+A high-performance search result deduplication engine that groups duplicate results by canonical URL and semantic similarity, returning clean batches with the most authoritative source and full traceability to all original sources.
+
+---
+
+## The Problem
+
+When aggregating search results from multiple providers (Google, Bing, DuckDuckGo), 40–60% of results are duplicates — same content, different URLs, different titles, different tracking parameters.
+
+```
+Source 1 (Google):
+  url: https://www.python.org/docs?utm_source=google
+  title: "Python Official Documentation"
+
+Source 2 (Bing):
+  url: http://python.org/docs/
+  title: "Python Docs"
+
+Source 3 (DuckDuckGo):
+  url: https://python.org/docs?utm_medium=cpc
+  title: "Learn Python - Official Docs"
+```
+
+These are the same resource. Without deduplication, users see noise instead of signal.
+
+---
+
+## The Solution
+
+`gibsondedup` processes raw search results through a multi-stage pipeline:
+
+```
+Raw Results
+    ↓
+Parse & Validate         (malformed records skipped gracefully)
+    ↓
+URL Normalization        (remove tracking params, www, trailing slashes)
+    ↓
+Exact URL Grouping       (hash-based, O(n) complexity)
+    ↓
+Semantic Merge           (Jaccard similarity, configurable threshold)
+    ↓
+Canonicalization         (select best result per group)
+    ↓
+Clean Output             (deduplicated results + metadata)
+```
+
+**Result:** 100 noisy inputs → 60–70 clean, canonical results.
+
+---
+
+## Installation
+
+```bash
+pip install gibsondedup
+```
+
+---
+
+## Quick Start
+
+```python
+from app.orchestration.pipeline import DeduplicationEngine
+
+engine = DeduplicationEngine()
+
+results = engine.process([
+    {
+        "title": "Python Official Documentation",
+        "url": "https://www.python.org/docs?utm_source=google",
+        "description": "Official Python docs",
+        "source": "google"
+    },
+    {
+        "title": "Python Docs",
+        "url": "http://python.org/docs/",
+        "description": "Python documentation",
+        "source": "bing"
+    },
+    {
+        "title": "Stack Overflow Python",
+        "url": "https://stackoverflow.com/questions/tagged/python",
+        "description": "Python questions",
+        "source": "google"
+    },
+])
+
+print(results)
+```
+
+**Output:**
+
+```json
+{
+  "results": [
+    {
+      "title": "Python Official Documentation",
+      "url": "https://www.python.org/docs?utm_source=google",
+      "canonical_url": "python.org/docs",
+      "description": "Official Python docs",
+      "sources": ["google", "bing"],
+      "duplicates_removed": 1
+    },
+    {
+      "title": "Stack Overflow Python",
+      "url": "https://stackoverflow.com/questions/tagged/python",
+      "canonical_url": "stackoverflow.com/questions/tagged/python",
+      "description": "Python questions",
+      "sources": ["google"],
+      "duplicates_removed": 0
+    }
+  ],
+  "meta": {
+    "total_input": 3,
+    "total_parsed": 3,
+    "total_output": 2,
+    "duplicates_removed": 1,
+    "processing_time_ms": 0.09,
+    "similarity_threshold": 0.8
+  }
+}
+```
+
+---
+
+## Configuration
+
+### Custom Similarity Threshold
+
+Control how aggressively similar-titled results are merged:
+
+```python
+# Conservative (default) — only merge highly similar titles
+engine = DeduplicationEngine(similarity_threshold=0.8)
+
+# Moderate — merge titles with moderate overlap
+engine = DeduplicationEngine(similarity_threshold=0.5)
+
+# Aggressive — merge titles with minimal overlap
+engine = DeduplicationEngine(similarity_threshold=0.3)
+```
+
+### Input Contract
+
+Each result in the input array accepts:
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `title` | string | ✅ | Result title |
+| `url` | string | ✅ | Result URL |
+| `description` | string | ❌ | Result description |
+| `source` | string | ❌ | Source provider (google, bing, etc.) |
+
+Malformed records (missing title or URL) are **skipped gracefully** — the pipeline continues processing valid records.
+
+---
+
+## Architecture
+
+### Pipeline Stages
+
+**Stage 1: Parser**
+Converts raw JSON payloads into structured internal contracts. Validates required fields. Skips malformed records without crashing.
+
+**Stage 2: URL Normalizer**
+Converts URLs to canonical form by:
+- Removing protocol (`http://`, `https://`)
+- Removing `www` prefix
+- Removing tracking parameters (`utm_*`, `fbclid`, `gclid`)
+- Removing trailing slashes
+- Stripping default ports (80, 443)
+- Preserving meaningful query parameters
+- Lowercasing everything
+
+**Stage 3: Exact Grouping**
+Groups results by normalized canonical URL using a hash map. Time complexity: O(n). Avoids the O(n²) pairwise comparison trap.
+
+**Stage 4: Semantic Merger**
+Within groups sharing the same domain, compares representative titles using Jaccard similarity. Groups exceeding the similarity threshold are merged into one. Only compares within the same exact domain — `python.org` and `docs.python.org` are treated as separate domains.
+
+**Stage 5: Canonicalizer**
+For each group, selects the single best result using a combined title + description length heuristic. Collects source traceability from all merged results.
+
+### Data Contracts
+
+```python
+# Input
+RawSearchResult(title, url, description?, source?)
+
+# Internal
+NormalizedSearchResult(title, url, canonical_url, description, source)
+
+# Output
+CanonicalResult(title, url, canonical_url, description, sources[], duplicates_removed)
+```
+
+### System Invariants
+
+1. Same normalized URL → same duplicate group (always)
+2. Canonical results preserve source traceability (always)
+3. Pipeline stages do not mutate upstream data (always)
+4. Normalization happens once per result (always)
+
+---
+
+## Engineering Decisions
+
+### Why hash-based grouping (O(n)) over pairwise comparison (O(n²))?
+
+With 1000 results, O(n²) means 1,000,000 comparisons. O(n) means 1000. At scale this is the difference between milliseconds and seconds.
+
+### Why exact domain matching for semantic merging?
+
+`python.org/docs` and `docs.python.org` serve genuinely different content despite sharing a base domain. Exact domain matching prevents false merges while still catching real duplicates like `python.org/docs` and `python.org/reference`.
+
+### Why Jaccard similarity over Levenshtein distance?
+
+Jaccard operates on token sets (words), making it robust to word order changes and additions. Levenshtein operates on character sequences, making it sensitive to trivial differences like "Docs" vs "Documentation".
+
+### Why keep normalization deterministic?
+
+Same input must always produce same output. This ensures:
+- Results are reproducible across runs
+- Caching normalized URLs is safe
+- Tests are reliable and meaningful
+
+---
+
+## Performance
+
+| Input Size | Processing Time | Memory |
+|------------|----------------|--------|
+| 100 results | ~1ms | ~1MB |
+| 1,000 results | ~10ms | ~5MB |
+| 10,000 results | ~100ms | ~50MB |
+
+*Benchmarked on Ubuntu 22.04, Python 3.10, Intel i5*
+
+---
+
+## Testing
+
+```bash
+# Install dev dependencies
+pip install pytest pytest-cov
+
+# Run all tests
+pytest tests/ -v
+
+# Run with coverage
+pytest tests/ --cov=app --cov-report=term-missing
+```
+
+**Test coverage: 59 tests across 5 modules**
+
+```
+tests/test_normalizer.py    14 tests  (URL normalization)
+tests/test_grouping.py       4 tests  (exact grouping)
+tests/test_canonicalizer.py  7 tests  (best result selection)
+tests/test_merger.py        11 tests  (semantic merging)
+tests/test_similarity.py    13 tests  (Jaccard similarity)
+tests/test_pipeline.py      10 tests  (end-to-end pipeline)
+```
+
+---
+
+## Roadmap
+
+### Phase 1 (Complete)
+- URL normalization
+- Exact duplicate grouping
+- Canonicalization with source traceability
+
+### Phase 2 (Complete)
+- Jaccard similarity engine
+- Domain-based semantic merging
+- Configurable similarity threshold
+
+### Phase 3 (Planned)
+- Persistent caching (Redis)
+- Database storage (PostgreSQL)
+- REST API exposure
+- Rails wrapper (gem)
+
+### Phase 4 (Planned)
+- Semantic embeddings (replace Jaccard with vector similarity)
+- Distributed processing
+- Production observability
+
+---
+
+## License
+
+MIT License. See `LICENSE` file.
+
+---
+
+## Author
+
+**Aseda Gibson**
+Computer Engineering, University of Energy and Natural Resources (UENR), Ghana.
+Backend systems, distributed architecture, infrastructure tooling.
+
+GitHub: [github.com/asedagibson](https://github.com/asedagibson)
+
+---
+
+*Built with correctness-oriented engineering. Every architectural decision is documented. Every invariant is tested.*

gibsondedup-0.1.0/gibsondedup/canonicalizer/canonicalizer.py

@@ -0,0 +1,24 @@
+from gibsondedup.contracts.models import NormalizedSearchResult, CanonicalResult
+from typing import List, Dict
+
+def canonicalize(groups: Dict[str, List[NormalizedSearchResult]]) -> List[CanonicalResult]:
+    canonical_results = []
+    for canonical_url, results in groups.items():
+        if not results:
+            continue
+        # Find the result with the longest description to use as the canonical result and longest title
+        canonical_result = max(results, key=lambda r: len(r.description or "") + len(r.title or ""))
+        # Create a CanonicalResult with the sources and count of duplicates removed
+        canonical_results.append(CanonicalResult(
+            title=canonical_result.title,
+            url=canonical_result.url,
+            canonical_url=canonical_result.canonical_url,
+            description=canonical_result.description,
+            sources=[r.source for r in results],
+            duplicates_removed=len(results) - 1
+        ))
+    return canonical_results
+
+__all__ = [
+    "canonicalize"
+]

gibsondedup-0.1.0/gibsondedup/contracts/models.py

@@ -0,0 +1,32 @@
+from dataclasses import dataclass, field
+from typing import Optional
+
+@dataclass
+class RawSearchResult:
+    title: str                        # Required
+    url: str                          # Required
+    description: Optional[str] = ""   # Optional
+    source: Optional[str] = "unknown" # Optional
+
+@dataclass
+class NormalizedSearchResult:
+    title: str
+    url: str                          # Original URL
+    canonical_url: str                # Normalized URL
+    description: Optional[str] = ""
+    source: Optional[str] = "unknown"
+
+@dataclass
+class CanonicalResult:
+    title: str
+    url: str
+    canonical_url: str
+    description: Optional[str] = ""
+    sources: list = field(default_factory=list)
+    duplicates_removed: int = 0
+
+__all__ = [
+    "RawSearchResult",
+    "NormalizedSearchResult",
+    "CanonicalResult"
+]