benchcraft-clean 0.1.0__py3-none-any.whl

benchcraft_clean-0.1.0.dist-info/METADATA

@@ -0,0 +1,242 @@
+Metadata-Version: 2.4
+Name: benchcraft-clean
+Version: 0.1.0
+Summary: Benchcraft Data-Quality: ONNX Runtime (PyTorch-free) text embeddings feeding naive cosine-similarity near-duplicate detection (D4 scaffold).
+Author: Benchcraft
+License: MIT
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.11
+Requires-Python: >=3.11
+Requires-Dist: numpy>=1.24
+Requires-Dist: onnx>=1.15
+Requires-Dist: onnxruntime>=1.17
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# benchcraft-clean
+
+A scaffold-depth implementation of one signature capability from
+Benchcraft's LazyClean module (architecture doc Part 3, "Module 2:
+LazyClean"): **embedding generation via native ONNX Runtime feeding a
+near-duplicate detection check** -- a minimal version of the
+Density-Based Semantic Deduplication (D4) idea.
+
+## What this package does (and doesn't) implement
+
+This is a **scaffold-depth pass, not a full implementation** of LazyClean.
+In scope:
+
+1. Embed a batch of text rows via an ONNX Runtime session
+   (`benchcraft_lazyclean.embeddings`).
+2. Flag near-duplicate row-index pairs via cosine-similarity thresholding
+   over those embeddings (`benchcraft_lazyclean.dedup`).
+
+Explicitly **out of scope** for this pass (tracked as future work per the
+architecture doc, not silently dropped):
+
+- The **IVF-HNSW approximate-nearest-neighbor index** and **spherical
+  mini-batch k-means** clustering step that the real D4 design uses to
+  avoid O(n²) pairwise cosine-similarity cost at scale.
+- The **DeCoLe tabular label-error detector** (per-subpopulation confident
+  learning).
+- The **train/test contamination auditor**.
+- The aggregate **"Dataset Integrity Score"**.
+
+## Zero-vector rows: "not comparable", not a silent duplicate or distinct call
+
+`hashing_bag_of_words_vectorizer` tokenizes with a simple `[a-z0-9]+` regex.
+Any text with **zero regex-matching tokens** -- not just genuinely empty or
+whitespace-only strings, but also punctuation-only text (`"!!!"`, `"???"`)
+and non-ASCII text the regex can't match (`"日本語"`) -- embeds to the
+identical all-zero vector. A zero embedding means *the vectorizer extracted
+no features*, not that the source rows are equal, and not that they're
+distinct either -- a hashing bag-of-words vectorizer with zero extracted
+features genuinely has no basis to compare two such rows.
+
+`cosine_similarity_matrix` and `find_near_duplicates` treat this honestly:
+every pairwise entry involving at least one zero-vector row (including a
+zero-vector row against itself, and against a genuinely non-zero row) is
+`nan` -- undefined, not silently `0.0` and not silently `1.0` -- and
+`find_near_duplicates` never flags such a pair as a duplicate by score.
+Instead, `DedupReport.zero_vector_row_indices` lists every row that
+produced no extractable features and could not be compared at all, as a
+third category distinct from both "confirmed duplicate" and "confirmed
+distinct" rows in `report.pairs`/`report.flagged_indices()`.
+
+This is itself the fix for two earlier bugs that got this wrong in
+opposite directions: originally, two genuinely-empty rows read similarity
+`0.0` against each other and were silently missed as duplicates; a
+follow-up fix over-corrected by making *any* two zero-vector rows read
+`1.0`, which falsely flagged unrelated zero-feature rows (e.g. `"!!!"` and
+`"???"`) as duplicates of each other at every valid threshold. Neither
+silent guess is right -- reporting "not comparable" separately is the
+honest answer given what this vectorizer can actually tell you. Genuinely
+identical non-empty text is unaffected by any of this and is still
+correctly flagged as a duplicate via its (non-zero) embedding vector.
+
+## The naive O(n²) caveat
+
+`dedup.py`'s `find_near_duplicates` computes the **full pairwise
+cosine-similarity matrix** over the embedded batch (`cosine_similarity_matrix`)
+and scans it directly. This is O(n²) in both time and memory. It is a
+correct, simple stand-in for the ANN index at this scaffold's small-batch
+depth, but it is **not** the production path -- the architecture doc calls
+out IVF-HNSW specifically to avoid this cost once dataset sizes grow past a
+small batch. Replacing this brute-force check with a real IVF-HNSW index is
+explicit follow-up work, not implemented here.
+
+## The PyTorch-free / <100MB constraint, and why
+
+Per the architecture doc (Part 3, Appendix A) and `CLAUDE.md`'s packaging
+section (§2.7): LazyClean is deliberately **PyTorch-free**. This package's
+runtime dependencies are exactly `onnxruntime`, `onnx`, `numpy`, and
+`lazycore` -- **no `torch`, no `transformers`, anywhere, including optional
+extras.** This is enforced as a hard rule (see `tests/test_embeddings.py::test_no_pytorch_or_transformers_imported`),
+not a soft preference, because:
+
+- PyTorch + HuggingFace `transformers` together pull in a multi-hundred-MB
+  to multi-GB dependency tree (CUDA/MPS backends, tokenizer binaries,
+  model-hub client code). LazyClean's design target is to stay under
+  **~100MB** total footprint so a data-cleaning pass doesn't require
+  installing the platform's heaviest dependency stack just to check for
+  duplicate rows.
+- It is also this module's specific differentiator against the PyTorch/HF
+  stack every other embedding-generation tool in this space defaults to.
+
+Embeddings are produced by loading a `.onnx` model file directly via the
+`onnxruntime` Python package and running lightweight, tokenizer-adjacent
+preprocessing ourselves in plain Python/NumPy (see
+`hashing_bag_of_words_vectorizer` in `embeddings.py`) -- never via
+`AutoTokenizer`/`AutoModel` or any `transformers` call.
+
+## Solving "we need a real `.onnx` model" without network access or a bundled checkpoint
+
+Real sentence-embedding ONNX models are tens to hundreds of MB, which is
+wrong to check into this repo and wrong to require for `pytest` to pass in
+an offline CI environment. This package handles that with two paths:
+
+1. **Hermetic (used by tests and the example):**
+   `embeddings.build_synthetic_embedding_onnx` hand-builds a tiny ONNX
+   graph on the fly, directly via the `onnx` package's graph-builder API
+   (`onnx.helper`/`onnx.numpy_helper`) -- a linear projection plus L2
+   normalization over a hashed bag-of-words feature vector. No network
+   access, no multi-hundred-MB file, deterministic given a seed.
+   `embeddings.build_synthetic_embedding_model()` wraps this plus the
+   default preprocessor into a ready-to-use `EmbeddingModel` in one call.
+   This is **not** a semantically meaningful sentence embedding -- it exists
+   solely to exercise the embed → cosine-similarity dedup pipeline
+   end-to-end in tests and the example without any external dependency.
+2. **Production (documented, not exercised by tests):** see "Wiring in a
+   real production model" below.
+
+### Wiring in a real production model
+
+`benchcraft_lazyclean.embeddings.MODEL_ALLOWLIST` (a per-module
+`lazycore.licensing.Allowlist` instance, per architecture doc §2.10)
+registers one recommended Tier-1 checkpoint:
+
+```python
+from benchcraft_lazyclean.embeddings import MODEL_ALLOWLIST
+entry = MODEL_ALLOWLIST.check("Xenova/all-MiniLM-L6-v2")
+# ModelLicenseEntry(name='Xenova/all-MiniLM-L6-v2', tier=ModelTier.TIER_1,
+#                    license_identifier='Apache-2.0', ...)
+```
+
+**Xenova/all-MiniLM-L6-v2** is an ONNX-exported build of
+`sentence-transformers/all-MiniLM-L6-v2` (384-dim mean-pooled embeddings,
+~90MB fp32 / ~23MB int8-quantized -- comfortably under the <100MB target),
+Apache-2.0 licensed, auto-usable under the Tier-1 policy with no opt-in
+gate. It is **not bundled** with this package. `embeddings.download_recommended_model()`
+is the optional, lazy download path (never called by tests or the example,
+matching the platform's local-only-by-default posture) that fetches and
+caches its `.onnx` graph.
+
+To actually use it for real inference, pair it with a real tokenizer
+instead of the default `hashing_bag_of_words_vectorizer`:
+
+```python
+from pathlib import Path
+from tokenizers import Tokenizer  # standalone Rust-backed tokenizer, no torch/transformers
+from benchcraft_lazyclean.embeddings import EmbeddingModel, download_recommended_model
+
+onnx_path = download_recommended_model()  # requires network access; opt-in
+tokenizer = Tokenizer.from_pretrained("Xenova/all-MiniLM-L6-v2")
+
+def real_preprocessor(text: str):
+    encoding = tokenizer.encode(text)
+    # Feed input_ids/attention_mask into the model's actual ONNX inputs and
+    # mean-pool the token embeddings over attention_mask here -- the exact
+    # shape depends on the checkpoint's input signature (use
+    # onnxruntime.InferenceSession(...).get_inputs() to inspect it).
+    ...
+
+model = EmbeddingModel.from_onnx_file(
+    onnx_path, preprocessor=real_preprocessor, embedding_dim=384,
+)
+```
+
+`tokenizers` (the standalone HuggingFace tokenizer library, Rust-backed,
+**no PyTorch/`transformers` dependency**) is a fine choice here per the
+task's constraints, but it is deliberately **not** a hard dependency of
+this package -- it is only needed if you wire in a real subword-tokenized
+model. The default hashing preprocessor has zero extra dependencies.
+
+## Public API
+
+```python
+from benchcraft_lazyclean import (
+    EmbeddingModel,               # wraps an onnxruntime.InferenceSession + preprocessor
+    build_synthetic_embedding_model,  # hermetic test/example fixture
+    detect_near_duplicate_text,   # rows -> (embeddings, DedupReport)
+    find_near_duplicates,         # embeddings -> DedupReport (naive O(n^2))
+    DedupReport, DuplicatePair,
+    MODEL_ALLOWLIST,              # lazycore.licensing.Allowlist for this module
+)
+
+model = build_synthetic_embedding_model()
+embeddings, report = detect_near_duplicate_text(
+    ["some text", "some txt", "totally different"], model, threshold=0.9,
+)
+for pair in report.pairs:
+    print(pair.index_a, pair.index_b, pair.similarity)
+
+# Rows that produced no extractable features (e.g. empty/punctuation-only/
+# non-ASCII text under the hashing vectorizer) are reported separately --
+# see "Zero-vector rows" above -- rather than silently folded into `pairs`.
+print("could not compare:", report.zero_vector_row_indices)
+```
+
+`detect_near_duplicate_text` accepts a plain `Iterable[str]`, or a Tier-1
+Arrow-backed `pandas.Series` / `polars.Series` (a single text column), per
+`lazycore.data`'s §2.1 conventions -- it uses
+`lazycore.data.is_arrow_backed_pandas` to check (and warn, not fail, if
+not) rather than re-implementing that check.
+
+## Installation
+
+`lazycore` is a local sibling package under `packages/lazycore` and is not
+declared as a formal `pyproject.toml` dependency (hatchling/pip have no
+portable relative-path dependency syntax) -- install it first, matching the
+convention already established by `packages/automl`:
+
+```bash
+pip install -e packages/lazycore
+pip install -e "packages/lazyclean[dev]"
+```
+
+## Running tests
+
+```bash
+pytest packages/lazyclean/tests
+```
+
+Fully hermetic -- no network access required. Uses
+`build_synthetic_embedding_model()` throughout.
+
+## Running the example
+
+```bash
+python packages/lazyclean/examples/dedup_example.py
+```

benchcraft_clean-0.1.0.dist-info/RECORD

@@ -0,0 +1,6 @@
+benchcraft_lazyclean/__init__.py,sha256=TRm5ek7dQbfcSlIjG40LkCTWjqj9xvO5E71ZVfjEifI,5357
+benchcraft_lazyclean/dedup.py,sha256=TA_GqVcQfJvmXq7KSZDlDiO8galyje8Fv6mLthSrRRw,8565
+benchcraft_lazyclean/embeddings.py,sha256=nrk72hGbdhbUowWXNXgXEcxsiBvdTvQe4LNeI7Jazes,20381
+benchcraft_clean-0.1.0.dist-info/METADATA,sha256=4cfzuucC-7dh7AsfNAaGWnNFOszuyDc4zEPvNmIc9is,11230
+benchcraft_clean-0.1.0.dist-info/WHEEL,sha256=lCkmxWfQsSc9CfIClYeavTdQeEX2toPqufh9gI35EQA,87
+benchcraft_clean-0.1.0.dist-info/RECORD,,

benchcraft_clean-0.1.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.31.0
+Root-Is-Purelib: true
+Tag: py3-none-any

benchcraft_lazyclean/__init__.py

@@ -0,0 +1,139 @@
+"""benchcraft_lazyclean -- LazyClean scaffold: ONNX Runtime embeddings + near-duplicate detection.
+
+Public API surface for the one signature capability implemented at this
+scaffold depth (architecture doc Part 3, "Module 2: LazyClean"): embed a
+batch of text rows via native ONNX Runtime (no PyTorch, no `transformers`),
+then flag near-duplicate row pairs via cosine-similarity thresholding over
+those embeddings -- a minimal stand-in for the Density-Based Semantic
+Deduplication (D4) idea. See ``dedup.py`` for the explicit naive-O(n^2)
+scope boundary and ``embeddings.py`` for the PyTorch-free embedding path.
+
+Everything else described for LazyClean in the architecture doc (the
+IVF-HNSW ANN index, spherical mini-batch k-means, the DeCoLe tabular
+label-error detector, train/test contamination auditing, the aggregate
+"Dataset Integrity Score") is out of scope for this pass -- see README.
+"""
+
+from __future__ import annotations
+
+import warnings
+from typing import TYPE_CHECKING
+
+import numpy as np
+
+from lazycore.data import is_arrow_backed_pandas
+
+from .dedup import DedupReport, DuplicatePair, cosine_similarity_matrix, find_near_duplicates
+from .embeddings import (
+    MODEL_ALLOWLIST,
+    RECOMMENDED_MODEL_NAME,
+    EmbeddingModel,
+    build_synthetic_embedding_model,
+    build_synthetic_embedding_onnx,
+    download_recommended_model,
+    hashing_bag_of_words_vectorizer,
+)
+
+if TYPE_CHECKING:  # pragma: no cover - type-checking-only imports
+    import pandas as pd
+    import polars as pl
+
+__all__ = [
+    "EmbeddingModel",
+    "MODEL_ALLOWLIST",
+    "RECOMMENDED_MODEL_NAME",
+    "build_synthetic_embedding_model",
+    "build_synthetic_embedding_onnx",
+    "download_recommended_model",
+    "hashing_bag_of_words_vectorizer",
+    "DedupReport",
+    "DuplicatePair",
+    "cosine_similarity_matrix",
+    "find_near_duplicates",
+    "detect_near_duplicate_text",
+]
+
+#: Text rows may be supplied as a plain iterable of strings, or as a
+#: Tier-1 Arrow-backed column (a pandas Series or a Polars Series), per
+#: lazycore.data's §2.1 conventions.
+TextRows = "Iterable[str] | pd.Series | pl.Series"
+
+
+def _coerce_text_rows(rows: object) -> list[str]:
+    """Normalize the accepted input shapes down to a plain ``list[str]``.
+
+    Accepts a plain iterable of strings, or a Tier-1 Arrow-backed pandas
+    Series / Polars Series (see ``lazycore.data``, architecture doc §2.1).
+    A pandas Series that is *not* Arrow-backed is still accepted (this
+    package does not hard-require Tier-1 storage to function), but emits a
+    warning pointing at the convention, using
+    ``lazycore.data.is_arrow_backed_pandas`` for the check -- reusing
+    lazycore's Tier-1 helper rather than re-implementing an Arrow-dtype
+    check here.
+    """
+    try:
+        import pandas as pd
+    except ImportError:
+        pd = None  # type: ignore[assignment]
+
+    if pd is not None and isinstance(rows, pd.DataFrame):
+        raise TypeError(
+            "Expected a single text column (a pandas Series), not a full "
+            "DataFrame. Pass e.g. `frame['text_column']`."
+        )
+
+    if pd is not None and isinstance(rows, pd.Series):
+        if not is_arrow_backed_pandas(rows.to_frame()):
+            warnings.warn(
+                "Input pandas Series is not Arrow-backed (pandas 2.x "
+                "ArrowDtype). benchcraft_lazyclean follows lazycore's Tier-1 "
+                "convention (architecture doc §2.1) for zero-copy interop "
+                "across Benchcraft modules; consider "
+                "`series.convert_dtypes(dtype_backend='pyarrow')`. "
+                "Proceeding anyway -- this is not a hard requirement.",
+                stacklevel=3,
+            )
+        return [str(value) for value in rows.tolist()]
+
+    try:
+        import polars as pl
+    except ImportError:
+        pl = None  # type: ignore[assignment]
+
+    if pl is not None and isinstance(rows, pl.Series):
+        return [str(value) for value in rows.to_list()]
+
+    return [str(value) for value in rows]  # type: ignore[union-attr]
+
+
+def detect_near_duplicate_text(
+    rows: object,
+    model: EmbeddingModel,
+    *,
+    threshold: float = 0.92,
+) -> tuple[np.ndarray, DedupReport]:
+    """Embed ``rows`` via ONNX Runtime and flag near-duplicate row pairs.
+
+    This is the one canonical entrypoint tying the embedding path
+    (``embeddings.py``) to the dedup path (``dedup.py``) together -- prefer
+    it over calling ``model.embed`` + ``find_near_duplicates`` separately
+    unless you need the embeddings for something else in between.
+
+    Args:
+        rows: an iterable of text strings, or a Tier-1 Arrow-backed pandas
+            Series / Polars Series (a single text column).
+        model: an :class:`EmbeddingModel` (e.g. from
+            :func:`build_synthetic_embedding_model` for hermetic use, or a
+            real production model wired via
+            :meth:`EmbeddingModel.from_onnx_file`).
+        threshold: cosine-similarity cutoff in ``(0.0, 1.0]`` -- see
+            :func:`find_near_duplicates`.
+
+    Returns:
+        ``(embeddings, report)`` -- the ``(n_rows, embedding_dim)`` float32
+        embedding array and the :class:`DedupReport` of flagged pairs.
+    """
+    texts = _coerce_text_rows(rows)
+    embeddings = model.embed(texts)
+    report = find_near_duplicates(embeddings, threshold=threshold)
+    return embeddings, report

benchcraft_lazyclean/dedup.py

@@ -0,0 +1,187 @@
+"""Cosine-similarity near-duplicate detection over embeddings.
+
+This is a minimal stand-in for the Density-Based Semantic Deduplication
+(D4) capability described in the architecture doc's Part 3 ("Module 2:
+LazyClean"). The real D4 design uses spherical mini-batch k-means to bucket
+embeddings into clusters, then an IVF-HNSW approximate-nearest-neighbor
+index within/across clusters, specifically to *avoid* O(n^2) pairwise
+cosine-similarity cost at scale.
+
+**This module deliberately implements only the naive O(n^2) brute-force
+version.** For a small batch this is a perfectly correct and simple
+stand-in for the ANN index; it does not scale past a few thousand rows
+before the pairwise similarity matrix becomes the bottleneck the
+architecture doc calls out IVF-HNSW as solving. Replacing
+:func:`cosine_similarity_matrix` (and the O(n^2) loop in
+:func:`find_near_duplicates`) with a real IVF-HNSW index is tracked as
+follow-up work, not implemented in this scaffold-depth pass -- see the
+package README for the explicit scope boundary.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+
+import numpy as np
+
+__all__ = [
+    "DuplicatePair",
+    "DedupReport",
+    "cosine_similarity_matrix",
+    "find_near_duplicates",
+]
+
+
+@dataclass(frozen=True)
+class DuplicatePair:
+    """One flagged near-duplicate row pair."""
+
+    index_a: int
+    index_b: int
+    similarity: float
+
+
+@dataclass
+class DedupReport:
+    """Result of a near-duplicate scan over a batch of embedded rows.
+
+    ``zero_vector_row_indices`` is a distinct third category alongside
+    "flagged as a duplicate pair" and "not flagged" -- see
+    :func:`find_near_duplicates` for why an all-zero embedding row cannot be
+    scored as either a confirmed duplicate or a confirmed distinct row.
+    """
+
+    pairs: list[DuplicatePair]
+    threshold: float
+    num_rows: int
+    zero_vector_row_indices: list[int] = field(default_factory=list)
+
+    def flagged_indices(self) -> set[int]:
+        """The set of row indices that appear in at least one flagged pair.
+
+        Rows in :attr:`zero_vector_row_indices` never appear here: a
+        zero-vector row's similarity to *anything* (including another
+        zero-vector row) is undefined, so it can never be scored above
+        ``threshold`` and flagged as a duplicate. Check
+        ``zero_vector_row_indices`` separately to see which rows produced no
+        extractable features and could not be compared at all.
+        """
+        flagged: set[int] = set()
+        for pair in self.pairs:
+            flagged.add(pair.index_a)
+            flagged.add(pair.index_b)
+        return flagged
+
+
+def cosine_similarity_matrix(embeddings: np.ndarray) -> np.ndarray:
+    """Compute the full pairwise cosine-similarity matrix for ``embeddings``.
+
+    **Naive O(n^2) implementation.** This materializes an ``(n, n)`` dense
+    similarity matrix, which is the exact cost profile the architecture
+    doc's IVF-HNSW approximate-nearest-neighbor index (Part 3, "Module 2:
+    LazyClean") exists to avoid at scale. Acceptable and simple for the
+    small batches this scaffold-depth pass targets; not a substitute for
+    that index once real dataset sizes are in play.
+
+    **Zero-vector rows are undefined, not 0.0 or 1.0.** A genuinely all-zero
+    embedding row (e.g. ``hashing_bag_of_words_vectorizer`` on text with no
+    regex-matching tokens at all -- empty/whitespace-only text, but also
+    punctuation-only text like ``"!!!"``/``"???"`` or non-ASCII text the
+    tokenizer's ``[a-z0-9]+`` regex can't match) has no defined direction. A
+    zero embedding means the vectorizer extracted *no features*, not that
+    the source rows are equal or that they are distinct -- two different
+    zero-feature texts (``"!!!"`` and ``"???"``) are not duplicates of each
+    other just because they hash to the same all-zero vector, and treating
+    that as similarity 1.0 falsely flags unrelated rows as duplicates at
+    every valid threshold. Silently reading it as 0.0 (the plain
+    normalized-dot-product result) is equally wrong in the other direction:
+    it silently misses genuinely-identical zero-feature rows (e.g. two
+    empty strings) as duplicates. Both are silent guesses this vectorizer
+    has no basis for. This function reports the honest answer instead:
+    every entry involving at least one zero-vector row (including a
+    zero-vector row against itself) is ``np.nan`` -- "not comparable" --
+    rather than a guessed 0.0 or 1.0. Callers that need pairwise duplicate
+    decisions should treat ``nan`` as "cannot say" and handle zero-vector
+    rows as a separate category (see :func:`find_near_duplicates` and
+    :attr:`DedupReport.zero_vector_row_indices`).
+    """
+    if embeddings.ndim != 2:
+        raise ValueError(f"Expected a 2D (n_rows, dim) array, got shape {embeddings.shape!r}.")
+    norms = np.linalg.norm(embeddings, axis=1, keepdims=True)
+    is_zero_vector = (norms == 0.0).reshape(-1)
+    safe_norms = np.where(norms == 0.0, 1.0, norms)
+    normalized = embeddings / safe_norms
+    similarities = normalized @ normalized.T
+
+    if is_zero_vector.any():
+        # Any pair where *either* row is a zero vector is undefined -- not
+        # just the zero-vector-against-zero-vector case. See the docstring
+        # above for why silently choosing 0.0 or 1.0 here is wrong either
+        # way.
+        undefined_mask = np.logical_or.outer(is_zero_vector, is_zero_vector)
+        similarities = np.where(undefined_mask, np.nan, similarities)
+    return similarities
+
+
+def find_near_duplicates(embeddings: np.ndarray, *, threshold: float = 0.92) -> DedupReport:
+    """Flag row-index pairs whose cosine similarity is at or above ``threshold``.
+
+    Brute-force O(n^2): computes the full pairwise similarity matrix via
+    :func:`cosine_similarity_matrix` and scans its upper triangle. See the
+    module docstring for why this is an intentional, documented scope
+    boundary rather than the production-scale IVF-HNSW path.
+
+    **Zero-vector rows are never flagged by similarity score.** A row whose
+    embedding is all-zero (see :func:`cosine_similarity_matrix`) has an
+    undefined (``nan``) similarity to every other row, including another
+    zero-vector row, so it is skipped when scanning for duplicate pairs --
+    it can never be silently swept in as a "duplicate" of an unrelated
+    zero-vector row, nor silently cleared as "distinct". Instead, every such
+    row's index is reported separately in
+    :attr:`DedupReport.zero_vector_row_indices`, meaning "this row produced
+    no extractable features and could not be compared" -- a third category
+    distinct from both "confirmed duplicate" and "confirmed distinct".
+
+    Args:
+        embeddings: ``(n_rows, dim)`` array, typically from
+            :meth:`benchcraft_lazyclean.embeddings.EmbeddingModel.embed`.
+        threshold: cosine-similarity cutoff in ``(0.0, 1.0]`` above which a
+            pair is flagged as a near-duplicate.
+
+    Returns:
+        A :class:`DedupReport` with pairs sorted by descending similarity,
+        plus ``zero_vector_row_indices`` for rows that could not be
+        compared at all.
+    """
+    if not (0.0 < threshold <= 1.0):
+        raise ValueError(f"threshold must be in (0.0, 1.0], got {threshold!r}.")
+    if embeddings.ndim != 2:
+        raise ValueError(f"Expected a 2D (n_rows, dim) array, got shape {embeddings.shape!r}.")
+    if not np.isfinite(embeddings).all():
+        raise ValueError("embeddings must contain only finite values")
+
+    num_rows = embeddings.shape[0]
+    norms = np.linalg.norm(embeddings, axis=1)
+    zero_vector_row_indices = [i for i in range(num_rows) if norms[i] == 0.0]
+    similarities = cosine_similarity_matrix(embeddings)
+
+    pairs: list[DuplicatePair] = []
+    for i in range(num_rows):
+        for j in range(i + 1, num_rows):
+            similarity = similarities[i, j]
+            if np.isnan(similarity):
+                # Undefined -- at least one of these two rows is a zero
+                # vector. Not comparable, so never flagged as a duplicate
+                # here; see zero_vector_row_indices instead.
+                continue
+            similarity = float(similarity)
+            if similarity >= threshold:
+                pairs.append(DuplicatePair(index_a=i, index_b=j, similarity=similarity))
+
+    pairs.sort(key=lambda pair: pair.similarity, reverse=True)
+    return DedupReport(
+        pairs=pairs,
+        threshold=threshold,
+        num_rows=num_rows,
+        zero_vector_row_indices=zero_vector_row_indices,
+    )

benchcraft_lazyclean/embeddings.py

@@ -0,0 +1,442 @@
+"""ONNX Runtime text embedding generation (architecture doc Part 3, "Module 2: LazyClean").
+
+The one hard constraint this module exists to enforce: embeddings are
+produced by loading a ``.onnx`` model file directly via the ``onnxruntime``
+Python package and running our own lightweight, tokenizer-adjacent
+preprocessing in plain Python/NumPy -- never via PyTorch or the HuggingFace
+``transformers`` package. That keeps this package's runtime dependency
+footprint under the architecture doc's ~100MB target and avoids pulling in
+the PyTorch/HuggingFace stack, which is LazyClean's specific differentiator
+per Part 3 and Appendix A ("LazyClean"). Do not import ``torch`` or
+``transformers`` anywhere in this package, including for type hints.
+
+Two embedding-model sources are provided:
+
+1. :func:`build_synthetic_embedding_model` -- hand-builds a tiny ONNX graph
+   on the fly via the ``onnx`` package's graph-builder API (a linear
+   projection + L2 normalization over a hashed bag-of-words feature vector).
+   This is fully hermetic (no network access, no bundled multi-hundred-MB
+   model file) and is what the test suite and ``examples/dedup_example.py``
+   use. It is a stand-in for a real sentence-embedding model, good enough to
+   demonstrate the embed -> dedup pipeline end-to-end, and is **not**
+   intended to produce semantically meaningful embeddings for production use.
+2. :func:`download_recommended_model` -- documents (and, given network
+   access, performs) the production wiring: fetching the Tier-1-allowlisted
+   ONNX sentence-embedding checkpoint referenced in :data:`MODEL_ALLOWLIST`
+   and caching it locally. This path is optional and lazy -- it is never
+   called by tests or the example, and nothing in this package requires
+   network access to import or to run its test suite.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import os
+import re
+import tempfile
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Callable, Iterable, Mapping, Sequence, Union
+
+import numpy as np
+import onnx
+import onnxruntime as ort
+from onnx import TensorProto, helper, numpy_helper
+
+from lazycore.licensing import Allowlist, ModelTier
+
+__all__ = [
+    "MODEL_ALLOWLIST",
+    "EmbeddingModel",
+    "hashing_bag_of_words_vectorizer",
+    "build_synthetic_embedding_onnx",
+    "build_synthetic_embedding_model",
+    "download_recommended_model",
+]
+
+_TOKEN_RE = re.compile(r"[a-z0-9]+")
+
+#: A preprocessor may return either a single ``(feature_dim,)`` array fed to
+#: the model's one input (:data:`EmbeddingModel.input_name` -- what the
+#: synthetic single-input fixture and ``hashing_bag_of_words_vectorizer``
+#: use), or a mapping of ONNX input name -> ``(feature_dim,)`` array for a
+#: real multi-input sentence-transformer checkpoint (e.g.
+#: ``{"input_ids": ..., "attention_mask": ..., "token_type_ids": ...}``).
+#: See :meth:`EmbeddingModel.embed`.
+PreprocessorOutput = Union[np.ndarray, Mapping[str, np.ndarray]]
+
+# ---------------------------------------------------------------------------
+# Model licensing allowlist (architecture doc §2.10) -- LazyClean's own
+# instance, per lazycore.licensing's documented per-module ownership pattern.
+# ---------------------------------------------------------------------------
+
+#: LazyClean's model-checkpoint allowlist (architecture doc §2.10). Starts
+#: empty per lazycore's contract; LazyClean populates it with the one
+#: production embedding checkpoint this module documents wiring for. This
+#: is a *recommendation*, not a bundled artifact -- see
+#: :func:`download_recommended_model`.
+MODEL_ALLOWLIST = Allowlist()
+
+RECOMMENDED_MODEL_NAME = "Xenova/all-MiniLM-L6-v2"
+
+MODEL_ALLOWLIST.register(
+    name=RECOMMENDED_MODEL_NAME,
+    tier=ModelTier.TIER_1,
+    license_identifier="Apache-2.0",
+    notes=(
+        "Recommended production checkpoint for benchcraft_lazyclean.embeddings.EmbeddingModel: "
+        "an ONNX-exported sentence-transformer (384-dim mean-pooled embeddings, "
+        "~90MB fp32 / ~23MB int8-quantized -- well under this module's <100MB "
+        "ONNX Runtime footprint target). Apache-2.0 licensed, auto-usable "
+        "(Tier 1), no opt-in gate required. Not bundled with this package and "
+        "not downloaded by default -- see download_recommended_model() and the "
+        "README's 'Wiring in a real production model' section for how to point "
+        "EmbeddingModel.from_onnx_file at a local copy of its model.onnx, paired "
+        "with a real WordPiece tokenizer (e.g. via the standalone `tokenizers` "
+        "library) and mean-pooling in place of this module's default hashing "
+        "bag-of-words preprocessor. Loading it never requires torch or "
+        "transformers -- only onnxruntime plus a tokenizer/vocab file."
+    ),
+)
+
+# A specific, pinned commit of the recommended checkpoint's HF repo -- NOT
+# "main". "main" is a mutable branch ref: the repo owner can force-push a
+# different model (different weights, a different license, or a broken
+# export) to it at any time, silently changing what a previously-verified
+# "Xenova/all-MiniLM-L6-v2" download resolves to. Pinning to an immutable
+# commit SHA means download_recommended_model() always fetches the exact,
+# previously-reviewed artifact this module's Tier-1/Apache-2.0
+# MODEL_ALLOWLIST entry above was actually reviewed against. Re-verify and
+# bump this SHA deliberately (a human decision, not automatic) if the
+# upstream repo needs to be updated.
+_RECOMMENDED_MODEL_REVISION = "751bff37182d3f1213fa05d7196b954e230abad9"
+
+# A direct HTTP source for the recommended checkpoint's ONNX export and its
+# tokenizer config. Referenced only by the optional, lazy download path in
+# download_recommended_model() -- never touched by import or by tests.
+_RECOMMENDED_MODEL_ONNX_URL = (
+    f"https://huggingface.co/Xenova/all-MiniLM-L6-v2/resolve/"
+    f"{_RECOMMENDED_MODEL_REVISION}/onnx/model.onnx"
+)
+
+
+# ---------------------------------------------------------------------------
+# Preprocessing ("tokenization-adjacent" -- runs in plain Python/NumPy, not
+# inside the ONNX graph, and never touches PyTorch/transformers).
+# ---------------------------------------------------------------------------
+
+
+def hashing_bag_of_words_vectorizer(vocab_dim: int = 128) -> Callable[[str], np.ndarray]:
+    """Return a text -> fixed-length float32 vector feature-hashing function.
+
+    This is the default preprocessor for the synthetic test/example model.
+    It lower-cases and word-tokenizes with a simple regex, hashes each token
+    into one of ``vocab_dim`` buckets via SHA-256 (stable across Python
+    processes, unlike the builtin ``hash()``), accumulates token counts per
+    bucket, and normalizes by token count so sentence length doesn't dominate
+    the resulting vector's magnitude. It has no PyTorch/transformers
+    dependency and no learned vocabulary -- it is intentionally simple,
+    matching the ~O(n^2)-for-now scaffold depth of this package (see
+    dedup.py). A real production model (see README) would instead use a
+    proper subword tokenizer (e.g. the standalone `tokenizers` library) and
+    feed token IDs/attention masks into a transformer ONNX graph.
+    """
+
+    def _tokenize(text: str) -> list[str]:
+        return _TOKEN_RE.findall(text.lower())
+
+    def _vectorize(text: str) -> np.ndarray:
+        vector = np.zeros(vocab_dim, dtype=np.float32)
+        tokens = _tokenize(text)
+        if not tokens:
+            return vector
+        for token in tokens:
+            digest = hashlib.sha256(token.encode("utf-8")).digest()
+            bucket = int.from_bytes(digest[:8], "big") % vocab_dim
+            vector[bucket] += 1.0
+        vector /= float(len(tokens))
+        return vector
+
+    return _vectorize
+
+
+# ---------------------------------------------------------------------------
+# Synthetic ONNX graph builder (hermetic test/example fixture)
+# ---------------------------------------------------------------------------
+
+
+def build_synthetic_embedding_onnx(
+    path: str | Path,
+    *,
+    vocab_dim: int = 128,
+    embedding_dim: int = 32,
+    seed: int = 0,
+) -> Path:
+    """Hand-build a tiny ONNX graph and save it to ``path``.
+
+    This is a **test/example fixture, not a real embedding model**. It is
+    built directly via the ``onnx`` package's graph-builder API
+    (``onnx.helper``/``onnx.numpy_helper``) so that tests and the example in
+    this package are fully self-contained: no network access is required,
+    and no multi-hundred-MB model file needs to be checked into the repo.
+
+    Graph shape: ``embedding = L2Normalize(input @ weight + bias)``, where
+    ``weight``/``bias`` are fixed (seeded) random initializers baked into
+    the graph. This is enough to demonstrate the embed -> cosine-similarity
+    dedup pipeline end-to-end -- near-identical input text produces
+    near-identical hashed feature vectors (see
+    :func:`hashing_bag_of_words_vectorizer`), and a linear map (even a
+    random one) preserves that similarity closely enough for near-duplicate
+    detection to work in tests. It is not a semantically meaningful sentence
+    embedding and must not be used for anything beyond tests/examples.
+
+    Returns the resolved ``Path`` the model was written to.
+    """
+    path = Path(path)
+    path.parent.mkdir(parents=True, exist_ok=True)
+
+    rng = np.random.default_rng(seed)
+    weight = rng.normal(scale=1.0 / np.sqrt(vocab_dim), size=(vocab_dim, embedding_dim)).astype(
+        np.float32
+    )
+    bias = np.zeros((embedding_dim,), dtype=np.float32)
+    eps = np.array([1e-12], dtype=np.float32)
+
+    input_info = helper.make_tensor_value_info("input", TensorProto.FLOAT, [None, vocab_dim])
+    output_info = helper.make_tensor_value_info(
+        "embedding", TensorProto.FLOAT, [None, embedding_dim]
+    )
+
+    weight_init = numpy_helper.from_array(weight, name="weight")
+    bias_init = numpy_helper.from_array(bias, name="bias")
+    eps_init = numpy_helper.from_array(eps, name="eps")
+
+    nodes = [
+        helper.make_node("MatMul", ["input", "weight"], ["linear_raw"], name="matmul"),
+        helper.make_node("Add", ["linear_raw", "bias"], ["linear_out"], name="add_bias"),
+        helper.make_node("Mul", ["linear_out", "linear_out"], ["squared"], name="square"),
+        helper.make_node(
+            "ReduceSum",
+            ["squared"],
+            ["sum_squared"],
+            name="reduce_sum",
+            axes=[1],
+            keepdims=1,
+        ),
+        helper.make_node("Sqrt", ["sum_squared"], ["norm"], name="sqrt"),
+        helper.make_node("Add", ["norm", "eps"], ["norm_eps"], name="add_eps"),
+        helper.make_node("Div", ["linear_out", "norm_eps"], ["embedding"], name="l2_normalize"),
+    ]
+
+    graph = helper.make_graph(
+        nodes,
+        "lazyclean_synthetic_embedding",
+        [input_info],
+        [output_info],
+        initializer=[weight_init, bias_init, eps_init],
+    )
+    # Opset 11: ReduceSum's `axes` is still an attribute here (it moved to
+    # an optional second input starting at opset 13), which keeps this
+    # graph-construction code simple. All other ops used below (MatMul,
+    # Add, Mul, Sqrt, Div) are stable well before and after opset 11.
+    model = helper.make_model(
+        graph,
+        producer_name="benchcraft-lazyclean",
+        opset_imports=[helper.make_opsetid("", 11)],
+    )
+    onnx.checker.check_model(model)
+    onnx.save(model, str(path))
+    return path
+
+
+# ---------------------------------------------------------------------------
+# EmbeddingModel -- the one canonical embedding path
+# ---------------------------------------------------------------------------
+
+
+@dataclass
+class EmbeddingModel:
+    """Wraps an ``onnxruntime.InferenceSession`` plus a text preprocessor.
+
+    This is the single canonical way this package turns text into
+    embeddings, whether the underlying ``.onnx`` graph is the synthetic
+    test fixture from :func:`build_synthetic_embedding_onnx` (single input,
+    single output -- e.g. ``hashing_bag_of_words_vectorizer``) or a real
+    production sentence-embedding checkpoint (see README), which typically
+    has *multiple* named inputs (``input_ids``/``attention_mask``/
+    ``token_type_ids``). There is deliberately no second/parallel embedding
+    code path -- both shapes go through :attr:`preprocessor` and
+    :meth:`embed` below; see :data:`PreprocessorOutput` for how a
+    preprocessor selects between them.
+    """
+
+    session: ort.InferenceSession
+    input_name: str
+    output_name: str
+    preprocessor: Callable[[str], PreprocessorOutput]
+    embedding_dim: int
+
+    @classmethod
+    def from_onnx_file(
+        cls,
+        model_path: str | Path,
+        *,
+        preprocessor: Callable[[str], PreprocessorOutput],
+        embedding_dim: int,
+        input_name: str | None = None,
+        output_name: str | None = None,
+        providers: Sequence[str] | None = None,
+    ) -> "EmbeddingModel":
+        """Load a ``.onnx`` embedding model via ``onnxruntime`` directly.
+
+        No PyTorch, no `transformers` -- ``onnxruntime.InferenceSession`` is
+        the only inference runtime this package ever touches.
+
+        ``input_name`` (like ``session.get_inputs()[0].name`` inferred by
+        default) only matters for a **single-input** graph, i.e. when
+        ``preprocessor`` returns a plain array per row -- see
+        :meth:`embed`. A real multi-input sentence-transformer ONNX
+        checkpoint has more than one required input
+        (``input_ids``/``attention_mask``/``token_type_ids``, typically);
+        for that case, write ``preprocessor`` to return a
+        ``{input_name: array}`` mapping instead, and this inferred/passed
+        ``input_name`` is simply unused (the mapping's own keys are fed to
+        the session directly). This function does not validate that a
+        single inferred input name covers every input the graph actually
+        requires -- if you load a multi-input model with a preprocessor
+        that returns a single array, ``onnxruntime`` will raise a missing-
+        input error at ``embed()`` time, not here.
+        """
+        session = ort.InferenceSession(
+            str(model_path), providers=list(providers) if providers else ["CPUExecutionProvider"]
+        )
+        resolved_input = input_name or session.get_inputs()[0].name
+        resolved_output = output_name or session.get_outputs()[0].name
+        return cls(
+            session=session,
+            input_name=resolved_input,
+            output_name=resolved_output,
+            preprocessor=preprocessor,
+            embedding_dim=embedding_dim,
+        )
+
+    def embed(self, texts: Iterable[str]) -> np.ndarray:
+        """Embed a batch of text rows, returning a ``(n, embedding_dim)`` float32 array.
+
+        ``preprocessor`` may return, per row, either:
+
+        - a plain ``(feature_dim,)`` array -- fed as the single named input
+          ``self.input_name`` (the synthetic fixture / hashing-vectorizer
+          shape), or
+        - a ``{input_name: (feature_dim,) array}`` mapping -- each named
+          input is stacked across the batch and fed to the session under
+          its own name (the real multi-input sentence-transformer shape,
+          e.g. ``input_ids``/``attention_mask``/``token_type_ids``).
+
+        Mixing the two shapes across rows in the same call is not
+        supported; ``preprocessor``'s return type is assumed to be
+        consistent for a given :class:`EmbeddingModel`.
+        """
+        rows = list(texts)
+        if not rows:
+            return np.zeros((0, self.embedding_dim), dtype=np.float32)
+        raw_features = [self.preprocessor(text) for text in rows]
+        if isinstance(raw_features[0], Mapping):
+            # Multi-input ONNX graph: each preprocessed row is a dict of
+            # named arrays (e.g. input_ids/attention_mask/token_type_ids)
+            # rather than a single array for self.input_name. Stack each
+            # named input across the batch and feed the session all of them
+            # at once, by name -- not just self.input_name.
+            input_names = raw_features[0].keys()
+            feed = {
+                name: np.stack([row[name] for row in raw_features]) for name in input_names  # type: ignore[index]
+            }
+            (output,) = self.session.run([self.output_name], feed)
+        else:
+            features = np.stack(raw_features).astype(np.float32)
+            (output,) = self.session.run([self.output_name], {self.input_name: features})
+        return np.asarray(output, dtype=np.float32)
+
+
+def build_synthetic_embedding_model(
+    *,
+    cache_dir: str | Path | None = None,
+    vocab_dim: int = 128,
+    embedding_dim: int = 32,
+    seed: int = 0,
+) -> EmbeddingModel:
+    """Build (or reuse a cached) synthetic ONNX embedding model, ready to use.
+
+    Convenience wrapper around :func:`build_synthetic_embedding_onnx` +
+    :func:`hashing_bag_of_words_vectorizer` + :meth:`EmbeddingModel.from_onnx_file`
+    so tests and the example call one function instead of duplicating this
+    three-step setup. Fully hermetic: no network access, writes a small
+    (a few KB) ``.onnx`` file to a temp/cache directory.
+    """
+    cache_dir_path = Path(cache_dir) if cache_dir is not None else Path(tempfile.gettempdir())
+    onnx_path = cache_dir_path / f"lazyclean_synthetic_v{vocab_dim}x{embedding_dim}_seed{seed}.onnx"
+    if not onnx_path.exists():
+        build_synthetic_embedding_onnx(
+            onnx_path, vocab_dim=vocab_dim, embedding_dim=embedding_dim, seed=seed
+        )
+    preprocessor = hashing_bag_of_words_vectorizer(vocab_dim=vocab_dim)
+    return EmbeddingModel.from_onnx_file(
+        onnx_path, preprocessor=preprocessor, embedding_dim=embedding_dim
+    )
+
+
+def download_recommended_model(
+    *,
+    cache_dir: str | Path | None = None,
+    accept_restricted_licenses: bool = False,
+) -> Path:
+    """Lazily download the Tier-1 recommended production checkpoint (optional).
+
+    This is the documented production wiring path -- **never called by
+    tests, the example, or any import-time code in this package.** It
+    requires network access, which is deliberately not a hard requirement
+    for installing or testing this package (per CLAUDE.md's local-only
+    constraint: any model-download path must be optional/lazy).
+
+    Looks the checkpoint up in :data:`MODEL_ALLOWLIST` first (demonstrating
+    the shared lazycore.licensing.Allowlist usage pattern) before touching
+    the network -- this call would raise ``RestrictedLicenseNotAcceptedError``
+    for a Tier 2 model without the opt-in flag, though the currently
+    registered recommended model is Tier 1 and does not require it.
+
+    After downloading, wire the result into :meth:`EmbeddingModel.from_onnx_file`
+    together with a real tokenizer (see README) -- this function only
+    fetches and caches the ``.onnx`` graph itself.
+
+    The download is written to a temporary file in ``cache_dir`` first and
+    atomically renamed into place only once it completes successfully. If
+    the download is interrupted (network error, process kill, etc.) the
+    temporary file is removed and ``dest`` is left untouched, so a later
+    call never mistakes a truncated/corrupt partial download for a valid
+    cached model just because a file already exists at ``dest``. The
+    fetched URL is pinned to a specific immutable commit of the upstream
+    repo (see ``_RECOMMENDED_MODEL_REVISION``), not a mutable branch ref,
+    so repeated calls (and calls across machines) always fetch the exact
+    same, previously-reviewed artifact.
+    """
+    MODEL_ALLOWLIST.check(
+        RECOMMENDED_MODEL_NAME, accept_restricted_licenses=accept_restricted_licenses
+    )
+    cache_dir_path = Path(cache_dir) if cache_dir is not None else Path.home() / ".cache" / "benchcraft" / "lazyclean"
+    cache_dir_path.mkdir(parents=True, exist_ok=True)
+    dest = cache_dir_path / "all-MiniLM-L6-v2.onnx"
+    if dest.exists():
+        return dest
+
+    import urllib.request
+
+    fd, tmp_name = tempfile.mkstemp(dir=cache_dir_path, suffix=".onnx.part")
+    tmp_path = Path(tmp_name)
+    try:
+        os.close(fd)
+        urllib.request.urlretrieve(_RECOMMENDED_MODEL_ONNX_URL, tmp_path)  # noqa: S310
+        tmp_path.replace(dest)  # atomic on the same filesystem
+    except BaseException:
+        tmp_path.unlink(missing_ok=True)
+        raise
+    return dest