verascan 0.1.0__tar.gz

verascan-0.1.0/.gitignore

@@ -0,0 +1,87 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Ruff
+.ruff_cache/
+
+# IDEs and Editors
+.vscode/
+.idea/
+*.swp
+*.swo
+*~
+.DS_Store
+
+# User generated outputs & scratch reports
+*.html
+*.json
+!pyproject.toml
+quickstart_report.html
+quickstart_report.json
+report.html
+report.json

verascan-0.1.0/CHANGELOG.md

@@ -0,0 +1,29 @@
+# Changelog
+
+All notable changes to **Verascan** will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+---
+
+## [0.1.0] - 2026-08-22
+
+### Added
+- **Core Detection Engines**:
+  - `exact`: Ultra-fast SHA-256 hash-based matching with case and whitespace normalisation.
+  - `fuzzy`: Near-duplicate detection using MinHash and Locality-Sensitive Hashing (LSH) via `datasketch`.
+  - `semantic`: Embedding-based semantic similarity search with `sentence-transformers` and FAISS vector indexing.
+- **Unified Data Loading**:
+  - Direct support for Python `list[str]`, `pandas.DataFrame`, `JSONL`, `CSV`, and HuggingFace `datasets.Dataset`.
+  - Custom column selection for tabular and nested datasets.
+- **Reporting & Visualization**:
+  - `ContaminationReport` class with query helpers (`flagged()`, `contamination_rate`, method breakdown metrics).
+  - Modern, self-contained, responsive HTML reports with interactive filters, search, word-level diffs, and health indicators.
+  - Machine-readable JSON export (`to_json()`) and dictionary conversion (`to_dict()`).
+- **Command-Line Interface (CLI)**:
+  - `verascan check` command with multi-method selection, threshold configuration, and output path options.
+  - `--fail-above` CI gate flag to exit with non-zero status when contamination exceeds acceptable limits.
+- **Quiet & Clean Runtime**:
+  - Automatic environment configuration to silence noisy backend C++/oneDNN and framework deprecation warnings.
+  - Zero startup latency for core exact/fuzzy operations.

verascan-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Verascan Contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

verascan-0.1.0/PKG-INFO

@@ -0,0 +1,319 @@
+Metadata-Version: 2.5
+Name: verascan
+Version: 0.1.0
+Summary: Detect data contamination between training and evaluation sets — exact, fuzzy, and semantic matching.
+Project-URL: Homepage, https://github.com/balamuruganpg/verascan
+Project-URL: Documentation, https://github.com/balamuruganpg/verascan#readme
+Project-URL: Repository, https://github.com/balamuruganpg/verascan
+Project-URL: Issues, https://github.com/balamuruganpg/verascan/issues
+Author: Verascan Contributors
+License-Expression: MIT
+License-File: LICENSE
+Keywords: contamination,data-quality,deduplication,evaluation,llm,machine-learning,rag,synthetic-data
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Typing :: Typed
+Requires-Python: >=3.9
+Requires-Dist: datasketch>=1.6
+Requires-Dist: jinja2>=3.1
+Requires-Dist: pandas>=1.5
+Requires-Dist: tqdm>=4.60
+Requires-Dist: typer[all]>=0.9
+Provides-Extra: all
+Requires-Dist: datasets>=2.0; extra == 'all'
+Requires-Dist: faiss-cpu>=1.7; extra == 'all'
+Requires-Dist: sentence-transformers>=3.0; extra == 'all'
+Provides-Extra: dev
+Requires-Dist: mypy>=1.5; extra == 'dev'
+Requires-Dist: pandas-stubs; extra == 'dev'
+Requires-Dist: pytest-cov>=4.0; extra == 'dev'
+Requires-Dist: pytest>=7.0; extra == 'dev'
+Requires-Dist: ruff>=0.4; extra == 'dev'
+Provides-Extra: hf
+Requires-Dist: datasets>=2.0; extra == 'hf'
+Provides-Extra: semantic
+Requires-Dist: faiss-cpu>=1.7; extra == 'semantic'
+Requires-Dist: sentence-transformers>=3.0; extra == 'semantic'
+Description-Content-Type: text/markdown
+
+<div align="center">
+
+# Verascan
+
+**Data Contamination & Leakage Detection for AI / ML Workflows**
+
+[![PyPI Version](https://img.shields.io/badge/pypi-v0.1.0-blue.svg)](https://pypi.org/project/verascan/)
+[![Python Versions](https://img.shields.io/badge/python-3.9%20%7C%203.10%20%7C%203.11%20%7C%203.12-blue.svg)](https://pypi.org/project/verascan/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://github.com/balamuruganpg/verascan/blob/main/LICENSE)
+[![Tests](https://img.shields.io/badge/Tests-71%20passed-brightgreen.svg)](https://github.com/balamuruganpg/verascan/actions)
+[![Type Checking](https://img.shields.io/badge/Typing-Strict-blue.svg)](https://mypy-lang.org/)
+[![Code Style](https://img.shields.io/badge/Code%20Style-Ruff-000000.svg)](https://github.com/astral-sh/ruff)
+
+<br>
+
+<p align="center">
+  <strong>Detect exact, fuzzy, and semantic data leakage between training and evaluation datasets.</strong><br>
+  Built for LLM fine-tuning, benchmark validation, RAG pipelines, and synthetic data auditing.
+</p>
+
+</div>
+
+---
+
+## Overview
+
+Data contamination occurs when evaluation or benchmark examples leak into a model's training data. This compromises evaluation validity, inflates benchmark scores, and masks real-world model degradation.
+
+**Verascan** provides a multi-tier contamination detection pipeline:
+1. **Exact match** — $O(N)$ hash-based verbatim duplicate detection with normalisation.
+2. **Fuzzy match** — MinHash + Locality-Sensitive Hashing (LSH) for near-duplicates and minor edits.
+3. **Semantic match** — Dense embedding similarity search (`sentence-transformers` + FAISS) for paraphrased content.
+
+---
+
+## Features
+
+- **Multi-Tier Detection**: Run exact, fuzzy, and semantic algorithms independently or in a cascaded pipeline.
+- **Cross-Method Deduplication**: Matches identified by exact hashing are automatically excluded from fuzzy/semantic passes to prevent double-counting.
+- **Multi-Format Ingestion**: Natively accepts `pandas.DataFrame`, `JSONL`, `CSV`, Hugging Face `datasets.Dataset`, and Python `list[str]`.
+- **Interactive HTML Reports**: Generates self-contained, offline-ready HTML reports featuring search, method filtering, and word-level diffs.
+- **CI/CD Integration**: CLI includes `--fail-above` to fail builds if contamination exceeds an allowed threshold.
+- **Lightweight Core**: Installs cleanly with minimal dependencies; heavy ML dependencies (`sentence-transformers`, `faiss-cpu`) are optional extras.
+- **Noise-Free Execution**: Built-in log suppression prevents noisy C++/oneDNN and framework deprecation logs from polluting `stderr`.
+
+---
+
+## Detection Engines
+
+| Method | Algorithm | Complexity / Speed | Best For |
+|---|---|---|---|
+| **`exact`** | SHA-256 Content Hashing (normalised) | $O(N + M)$ &bull; *Microseconds* | Verbatim duplicates, casing/whitespace variations |
+| **`fuzzy`** | MinHash + LSH (`datasketch`) | $O(N + M)$ &bull; *Milliseconds* | Minor edits, word insertions/deletions, truncations |
+| **`semantic`** | Dense Vector Cosine Similarity (FAISS) | $O(M \cdot d)$ &bull; *Seconds* | Paraphrased sentences, reworded questions, synonyms |
+
+---
+
+## Installation
+
+```bash
+# Core installation (exact + fuzzy matching)
+pip install verascan
+
+# With semantic similarity matching (sentence-transformers + FAISS)
+pip install "verascan[semantic]"
+
+# With Hugging Face datasets support
+pip install "verascan[hf]"
+
+# Complete installation with all optional extras
+pip install "verascan[all]"
+```
+
+---
+
+## Quickstart
+
+### Python API
+
+```python
+import verascan
+
+# Run contamination audit across training and evaluation splits
+report = verascan.check(
+    train="data/train.jsonl",
+    eval="data/eval.jsonl",
+    methods=["exact", "fuzzy"],
+    threshold=0.85,
+)
+
+# Print terminal summary
+report.summary()
+
+# Inspect metrics
+print(f"Contamination Rate: {report.contamination_rate:.1%}")
+print(f"Flagged Pairs     : {report.total_matches}")
+
+# Query high-confidence matches
+for match in report.flagged(min_score=0.90):
+    print(
+        f"[{match.method}] Eval #{match.eval_index} <-> Train #{match.train_index} (Score: {match.score:.3f})"
+    )
+    print(f"  Eval : {match.eval_text}")
+    print(f"  Train: {match.train_text}")
+
+# Export reports
+report.to_html("contamination_report.html")
+report.to_json("contamination_report.json")
+```
+
+### Terminal Output
+
+```text
+===============================================
+  Verascan Contamination Report
+===============================================
+  Train size      : 50,000
+  Eval size       : 1,000
+  Methods         : exact, fuzzy
+  Threshold       : 0.85
+-----------------------------------------------
+  Total matches   : 14
+  Contaminated    : 12 / 1,000 eval samples (1.2%)
+    Exact matches : 4
+    Fuzzy matches : 10
+===============================================
+```
+
+---
+
+## Supported Input Formats
+
+Verascan normalises inputs into clean text sequences automatically:
+
+```python
+import pandas as pd
+import verascan
+
+# 1. Plain String Lists
+report = verascan.check(
+    train=["The quick brown fox.", "Artificial intelligence."],
+    eval=["The quick brown fox."],
+)
+
+# 2. File Paths (CSV or JSONL)
+report = verascan.check(
+    train="data/train.jsonl",
+    eval="data/eval.csv",
+    column="text",
+)
+
+# 3. Pandas DataFrames
+train_df = pd.DataFrame({"prompt": ["Translate to French...", "Summarize..."]})
+eval_df = pd.DataFrame({"prompt": ["Translate to French..."]})
+report = verascan.check(train=train_df, eval=eval_df, column="prompt")
+
+# 4. Hugging Face Datasets
+from datasets import load_dataset
+
+train_ds = load_dataset("imdb", split="train")
+eval_ds = load_dataset("imdb", split="test")
+report = verascan.check(train=train_ds, eval=eval_ds, column="text")
+```
+
+---
+
+## CLI Usage
+
+The `verascan` command-line interface enables automated checks in terminal workflows and CI/CD pipelines:
+
+```bash
+# Basic contamination check
+verascan check --train train.jsonl --eval eval.jsonl
+
+# Specify custom column, methods, and threshold
+verascan check \
+  --train data/train.csv \
+  --eval data/eval.csv \
+  --methods exact,fuzzy \
+  --threshold 0.80 \
+  --column instruction \
+  --output report.html
+
+# CI/CD Gate: Fail build if contamination rate exceeds 1%
+verascan check \
+  --train train.jsonl \
+  --eval eval.jsonl \
+  --fail-above 0.01
+```
+
+---
+
+## Interactive HTML Reports
+
+The HTML report generated via `report.to_html("report.html")` is **100% self-contained** (no external fonts, CDNs, or scripts required):
+
+- **Health Status Banner**: Visual indicator (`Clean`, `Low Risk`, `High Risk`) with contamination percentage and progress meter.
+- **Method Breakdown**: Color-coded badges for exact (Rose), fuzzy (Amber), and semantic (Indigo) detections.
+- **Live Search & Filtering**: Instant client-side search across text samples and index numbers.
+- **Word-Level Diffs**: Color-coded `<del>` and `<ins>` tags illustrating textual overlap.
+- **Responsive Layout**: Designed for seamless viewing across desktop monitors and mobile devices.
+
+---
+
+## ContaminationReport API
+
+```python
+report = verascan.check(train, eval)
+
+# Properties
+report.contamination_rate  # float: Fraction of eval examples found in train (0.0 to 1.0)
+report.total_matches  # int: Total flagged pairs
+report.exact_count  # int: Exact duplicate count
+report.fuzzy_count  # int: Fuzzy / near-duplicate count
+report.semantic_count  # int: Semantic match count
+report.train_size  # int: Size of training corpus
+report.eval_size  # int: Size of evaluation corpus
+
+# Methods
+report.flagged(min_score=0.9)  # Returns list of MatchRecord objects >= min_score
+report.summary()  # Prints ASCII summary to stdout
+report.to_dict()  # Serialises report to a Python dict
+report.to_json("report.json")  # Exports JSON file
+report.to_html("report.html")  # Exports self-contained interactive HTML report
+```
+
+### `MatchRecord` Structure
+
+Each match in `report.matches` contains:
+- `eval_index: int` — Index of the sample in the evaluation dataset.
+- `train_index: int` — Index of the sample in the training dataset.
+- `eval_text: str` — Evaluation sample text.
+- `train_text: str` — Matching training sample text.
+- `score: float` — Similarity metric (`1.0` for exact matches, Jaccard for fuzzy, cosine for semantic).
+- `method: str` — Engine that produced the match (`"exact"`, `"fuzzy"`, `"semantic"`).
+
+---
+
+## Limitations
+
+- **Large-Scale Semantic Search**: While FAISS provides fast approximate search, semantic matching encodes all samples using transformer models, which is compute-intensive on CPU for corpora with millions of rows. For very large datasets, start with `methods=["exact", "fuzzy"]`.
+- **Character N-Gram Sensitivity**: Fuzzy matching relies on character 5-grams by default. Very short texts (fewer than 5 characters) fall back to exact matching.
+- **Cross-Lingual Matching**: The default semantic model (`all-MiniLM-L6-v2`) is optimized for English text. For multilingual evaluation datasets, specify a multilingual model via `model_name="paraphrase-multilingual-MiniLM-L12-v2"`.
+
+---
+
+## Development
+
+```bash
+# Clone repository
+git clone https://github.com/balamuruganpg/verascan.git
+cd verascan
+
+# Install development dependencies
+pip install -e ".[all,dev]"
+
+# Run test suite
+pytest
+
+# Code formatting and linting
+ruff check .
+ruff format --check .
+
+# Type checking
+mypy src/
+```
+
+---
+
+## License
+
+Distributed under the [MIT License](https://github.com/balamuruganpg/verascan/blob/main/LICENSE).