dataruff 0.1.0__tar.gz

dataruff-0.1.0/PKG-INFO

@@ -0,0 +1,235 @@
+Metadata-Version: 2.4
+Name: dataruff
+Version: 0.1.0
+Summary: One-command dataset health diagnostics — the Ruff of datasets.
+Author: dataruff contributors
+License-Expression: MIT
+Keywords: data quality,pandas,csv,data validation,EDA,data science
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: pandas>=2.0
+Requires-Dist: numpy>=1.24
+Requires-Dist: scipy>=1.10
+Requires-Dist: scikit-learn>=1.3
+Requires-Dist: openpyxl>=3.1
+Requires-Dist: python-dateutil>=2.8
+Provides-Extra: rich
+Requires-Dist: rich>=13.0; extra == "rich"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.4; extra == "dev"
+Requires-Dist: pytest-cov>=4.1; extra == "dev"
+Requires-Dist: rich>=13.0; extra == "dev"
+
+# dataruff
+
+[![CI](https://github.com/AryanPatankar27/dataruff/actions/workflows/ci.yml/badge.svg)](https://github.com/AryanPatankar27/dataruff/actions/workflows/ci.yml)
+[![codecov](https://codecov.io/gh/AryanPatankar27/dataruff/branch/main/graph/badge.svg)](https://codecov.io/gh/AryanPatankar27/dataruff)
+[![PyPI version](https://img.shields.io/pypi/v/dataruff)](https://pypi.org/project/dataruff/)
+[![Python](https://img.shields.io/pypi/pyversions/dataruff)](https://pypi.org/project/dataruff/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+**The Ruff of datasets.** One command to discover, explain, score, and fix data quality problems in Pandas DataFrames and CSV/Excel files.
+
+```python
+from datadoctor import audit
+
+audit(df)
+```
+
+```
+Data Quality Score: 81/100
+
+Issues Found (5):
+  !      42 duplicate rows
+  ~      13 invalid email  (column: email)
+  !       3 empty columns
+  ~       7 outlier  (column: salary)
+  .       2 inconsistent date format  (column: created_at)
+
+Rows: 10,000 | Columns: 12
+```
+
+---
+
+## Install
+
+```bash
+pip install dataruff
+```
+
+Optionally install [rich](https://github.com/Textualize/rich) for prettier terminal output:
+
+```bash
+pip install dataruff[rich]
+```
+
+---
+
+## Quick start
+
+```python
+import pandas as pd
+from datadoctor import audit, fix, score, validate, detect_pii
+
+df = pd.read_csv("customers.csv")
+
+# Full health report
+audit(df)
+
+# Get numeric score
+s = score(df)
+print(s.overall)   # 81
+print(s.to_dict()) # {'overall': 81, 'completeness': 92, ...}
+
+# Auto-fix common issues
+clean_df = fix(df)
+
+# Validate against a schema
+result = validate(df, schema={
+    "email": "email",
+    "age":   "0-120",
+    "id":    "unique",
+})
+
+# PII detection
+report = detect_pii(df)
+print(report.columns_with_pii)
+# {'email': ['email'], 'phone': ['phone'], 'uid': ['aadhaar']}
+```
+
+---
+
+## API reference
+
+| Function | Description | Returns |
+|---|---|---|
+| `audit(df)` | Print full health report | `InvestigationReport` |
+| `investigate(df)` | Structured issue breakdown | `InvestigationReport` |
+| `score(df)` | Data quality score | `ScoreBreakdown` |
+| `fix(df)` | Auto-repair common issues | `pd.DataFrame` |
+| `validate(df, schema)` | Check schema constraints | `dict` |
+| `compare(old, new)` | Diff two datasets | `ComparisonReport` |
+| `detect_pii(df)` | Find PII columns | `PIIReport` |
+| `mask_pii(df)` | Redact PII values | `pd.DataFrame` |
+| `detect_drift(old, new)` | Distribution drift analysis | `DriftReport` |
+| `find_anomalies(df)` | Anomaly / outlier detection | `dict` |
+
+All functions accept a **DataFrame, CSV path, or XLSX path** as input.
+
+---
+
+## Scoring formula
+
+| Dimension | Weight | Measures |
+|---|---|---|
+| Completeness | 25% | Non-null ratio across all cells |
+| Validity | 25% | Format correctness (emails, dates, types) |
+| Consistency | 20% | Uniform types and formats per column |
+| Uniqueness | 20% | Absence of duplicate rows |
+| Schema compliance | 10% | Adherence to user-provided schema |
+
+---
+
+## `fix()` — what gets repaired
+
+| Issue | Fix applied |
+|---|---|
+| Duplicate rows | Removed |
+| Leading/trailing whitespace | Stripped |
+| Boolean strings (`yes/no/true/false`) | Converted to `bool` |
+| Mixed date formats | Normalized to `YYYY-MM-DD` |
+| Missing numeric values | Filled with column median |
+| Missing string values | Filled with column mode |
+
+---
+
+## `validate()` — schema rules
+
+```python
+validate(df, schema={
+    "email":   "email",          # valid email format
+    "age":     "0-120",          # numeric range
+    "user_id": "unique",         # no duplicates
+    "price":   "positive",       # > 0
+    "code":    "not_null",       # no missing values
+    "ref":     "regex:[A-Z]{3}", # custom regex
+})
+```
+
+---
+
+## `detect_pii()` — supported PII types
+
+| Type | Example |
+|---|---|
+| `email` | `alice@example.com` |
+| `phone` | `9876543210` |
+| `aadhaar` | `2345 6789 0123` |
+| `pan` | `ABCDE1234F` |
+| `ssn` | `123-45-6789` |
+| `credit_card` | `4111 1111 1111 1111` |
+
+---
+
+## CLI
+
+```bash
+# Audit a CSV file
+dataruff audit customers.csv
+
+# Output as JSON
+dataruff audit customers.csv --json
+
+# Fix issues and write cleaned file
+dataruff fix customers.csv
+# -> customers_clean.csv
+
+# Compare two datasets
+dataruff compare old.csv new.csv
+
+# Data quality score
+dataruff score customers.csv
+
+# PII detection
+dataruff detect-pii customers.csv
+
+# Mask PII
+dataruff mask-pii customers.csv
+# -> customers_masked.csv
+```
+
+---
+
+## Architecture
+
+```
+datadoctor/
+├── analyzers/       # DuplicateAnalyzer, NullAnalyzer, TypeAnalyzer,
+│                    # FormatAnalyzer, OutlierAnalyzer, PIIAnalyzer, DriftAnalyzer
+├── scoring/         # Weighted scoring engine
+├── fixing/          # Auto-remediation rules
+└── reporting/       # Terminal (rich + plain fallback) and JSON output
+```
+
+No LLMs. No API calls. Everything deterministic and offline.
+
+---
+
+## Requirements
+
+- Python 3.10+
+- pandas, numpy, scipy, scikit-learn, openpyxl, python-dateutil
+
+---
+
+## License
+
+MIT

dataruff-0.1.0/README.md

@@ -0,0 +1,205 @@
+# dataruff
+
+[![CI](https://github.com/AryanPatankar27/dataruff/actions/workflows/ci.yml/badge.svg)](https://github.com/AryanPatankar27/dataruff/actions/workflows/ci.yml)
+[![codecov](https://codecov.io/gh/AryanPatankar27/dataruff/branch/main/graph/badge.svg)](https://codecov.io/gh/AryanPatankar27/dataruff)
+[![PyPI version](https://img.shields.io/pypi/v/dataruff)](https://pypi.org/project/dataruff/)
+[![Python](https://img.shields.io/pypi/pyversions/dataruff)](https://pypi.org/project/dataruff/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+**The Ruff of datasets.** One command to discover, explain, score, and fix data quality problems in Pandas DataFrames and CSV/Excel files.
+
+```python
+from datadoctor import audit
+
+audit(df)
+```
+
+```
+Data Quality Score: 81/100
+
+Issues Found (5):
+  !      42 duplicate rows
+  ~      13 invalid email  (column: email)
+  !       3 empty columns
+  ~       7 outlier  (column: salary)
+  .       2 inconsistent date format  (column: created_at)
+
+Rows: 10,000 | Columns: 12
+```
+
+---
+
+## Install
+
+```bash
+pip install dataruff
+```
+
+Optionally install [rich](https://github.com/Textualize/rich) for prettier terminal output:
+
+```bash
+pip install dataruff[rich]
+```
+
+---
+
+## Quick start
+
+```python
+import pandas as pd
+from datadoctor import audit, fix, score, validate, detect_pii
+
+df = pd.read_csv("customers.csv")
+
+# Full health report
+audit(df)
+
+# Get numeric score
+s = score(df)
+print(s.overall)   # 81
+print(s.to_dict()) # {'overall': 81, 'completeness': 92, ...}
+
+# Auto-fix common issues
+clean_df = fix(df)
+
+# Validate against a schema
+result = validate(df, schema={
+    "email": "email",
+    "age":   "0-120",
+    "id":    "unique",
+})
+
+# PII detection
+report = detect_pii(df)
+print(report.columns_with_pii)
+# {'email': ['email'], 'phone': ['phone'], 'uid': ['aadhaar']}
+```
+
+---
+
+## API reference
+
+| Function | Description | Returns |
+|---|---|---|
+| `audit(df)` | Print full health report | `InvestigationReport` |
+| `investigate(df)` | Structured issue breakdown | `InvestigationReport` |
+| `score(df)` | Data quality score | `ScoreBreakdown` |
+| `fix(df)` | Auto-repair common issues | `pd.DataFrame` |
+| `validate(df, schema)` | Check schema constraints | `dict` |
+| `compare(old, new)` | Diff two datasets | `ComparisonReport` |
+| `detect_pii(df)` | Find PII columns | `PIIReport` |
+| `mask_pii(df)` | Redact PII values | `pd.DataFrame` |
+| `detect_drift(old, new)` | Distribution drift analysis | `DriftReport` |
+| `find_anomalies(df)` | Anomaly / outlier detection | `dict` |
+
+All functions accept a **DataFrame, CSV path, or XLSX path** as input.
+
+---
+
+## Scoring formula
+
+| Dimension | Weight | Measures |
+|---|---|---|
+| Completeness | 25% | Non-null ratio across all cells |
+| Validity | 25% | Format correctness (emails, dates, types) |
+| Consistency | 20% | Uniform types and formats per column |
+| Uniqueness | 20% | Absence of duplicate rows |
+| Schema compliance | 10% | Adherence to user-provided schema |
+
+---
+
+## `fix()` — what gets repaired
+
+| Issue | Fix applied |
+|---|---|
+| Duplicate rows | Removed |
+| Leading/trailing whitespace | Stripped |
+| Boolean strings (`yes/no/true/false`) | Converted to `bool` |
+| Mixed date formats | Normalized to `YYYY-MM-DD` |
+| Missing numeric values | Filled with column median |
+| Missing string values | Filled with column mode |
+
+---
+
+## `validate()` — schema rules
+
+```python
+validate(df, schema={
+    "email":   "email",          # valid email format
+    "age":     "0-120",          # numeric range
+    "user_id": "unique",         # no duplicates
+    "price":   "positive",       # > 0
+    "code":    "not_null",       # no missing values
+    "ref":     "regex:[A-Z]{3}", # custom regex
+})
+```
+
+---
+
+## `detect_pii()` — supported PII types
+
+| Type | Example |
+|---|---|
+| `email` | `alice@example.com` |
+| `phone` | `9876543210` |
+| `aadhaar` | `2345 6789 0123` |
+| `pan` | `ABCDE1234F` |
+| `ssn` | `123-45-6789` |
+| `credit_card` | `4111 1111 1111 1111` |
+
+---
+
+## CLI
+
+```bash
+# Audit a CSV file
+dataruff audit customers.csv
+
+# Output as JSON
+dataruff audit customers.csv --json
+
+# Fix issues and write cleaned file
+dataruff fix customers.csv
+# -> customers_clean.csv
+
+# Compare two datasets
+dataruff compare old.csv new.csv
+
+# Data quality score
+dataruff score customers.csv
+
+# PII detection
+dataruff detect-pii customers.csv
+
+# Mask PII
+dataruff mask-pii customers.csv
+# -> customers_masked.csv
+```
+
+---
+
+## Architecture
+
+```
+datadoctor/
+├── analyzers/       # DuplicateAnalyzer, NullAnalyzer, TypeAnalyzer,
+│                    # FormatAnalyzer, OutlierAnalyzer, PIIAnalyzer, DriftAnalyzer
+├── scoring/         # Weighted scoring engine
+├── fixing/          # Auto-remediation rules
+└── reporting/       # Terminal (rich + plain fallback) and JSON output
+```
+
+No LLMs. No API calls. Everything deterministic and offline.
+
+---
+
+## Requirements
+
+- Python 3.10+
+- pandas, numpy, scipy, scikit-learn, openpyxl, python-dateutil
+
+---
+
+## License
+
+MIT

dataruff-0.1.0/datadoctor/__init__.py

@@ -0,0 +1,36 @@
+"""
+dataruff — One-command dataset health diagnostics.
+
+Usage:
+    from datadoctor import audit, fix, score, detect_pii
+
+    audit(df)                    # Print quality report
+    fix(df)                      # Return cleaned DataFrame
+    score(df)                    # Return ScoreBreakdown
+    detect_pii(df)               # Return PIIReport
+"""
+
+from datadoctor.audit import audit
+from datadoctor.investigate import investigate
+from datadoctor.fix import fix
+from datadoctor.validate import validate
+from datadoctor.compare import compare
+from datadoctor.pii import detect_pii, mask_pii
+from datadoctor.drift import detect_drift
+from datadoctor.anomalies import find_anomalies
+from datadoctor.score import score
+
+__version__ = "0.1.0"
+
+__all__ = [
+    "audit",
+    "investigate",
+    "fix",
+    "validate",
+    "compare",
+    "detect_pii",
+    "mask_pii",
+    "detect_drift",
+    "find_anomalies",
+    "score",
+]

dataruff-0.1.0/datadoctor/_compat.py

@@ -0,0 +1,40 @@
+"""
+Pandas 2.x / 3.x dtype compatibility.
+
+In pandas 2.x string columns have dtype ``object`` (dtype.name == 'object').
+In pandas 3.x (infer_string=True by default) they have a ``StringDtype``
+instance whose repr shows as ``dtype: str`` (dtype.name may be 'str',
+'string', or 'string[python]' depending on the sub-release).
+
+The safest guard is ``isinstance(dtype, pd.StringDtype)`` — it covers every
+StringDtype variant without relying on the `.name` attribute.
+"""
+from __future__ import annotations
+
+import pandas as pd
+
+
+def is_str_col(series: pd.Series) -> bool:
+    """
+    True for any string-like column, pandas 2.x and 3.x compatible.
+
+    - pandas 2.x default:  dtype == object   (plain Python objects)
+    - pandas 3.x default:  isinstance(dtype, pd.StringDtype)
+                            (repr shows as ``dtype: str``)
+    """
+    dtype = series.dtype
+    # Fast path: classic object dtype used by pandas 2.x
+    if dtype == object:
+        return True
+    # All StringDtype variants (pd.StringDtype was added in pandas 1.0 and is
+    # the default in pandas 3.x regardless of storage backend)
+    if hasattr(pd, "StringDtype") and isinstance(dtype, pd.StringDtype):
+        return True
+    # Extra safety-net: catch any future/vendor string dtype by name
+    name = getattr(dtype, "name", "")
+    return name in ("str", "string", "large_string") or "string" in str(dtype).lower()
+
+
+def str_columns(df: pd.DataFrame) -> list[str]:
+    """Return names of all string-like columns in *df*."""
+    return [col for col in df.columns if is_str_col(df[col])]

dataruff-0.1.0/datadoctor/analyzers/__init__.py

@@ -0,0 +1,19 @@
+from datadoctor.analyzers import (
+    duplicate,
+    null_analyzer,
+    type_analyzer,
+    format_analyzer,
+    outlier,
+    pii_analyzer,
+    drift_analyzer,
+)
+
+__all__ = [
+    "duplicate",
+    "null_analyzer",
+    "type_analyzer",
+    "format_analyzer",
+    "outlier",
+    "pii_analyzer",
+    "drift_analyzer",
+]

dataruff-0.1.0/datadoctor/analyzers/drift_analyzer.py

@@ -0,0 +1,68 @@
+from __future__ import annotations
+
+from typing import Any
+
+import numpy as np
+import pandas as pd
+from scipy import stats
+
+from datadoctor._compat import is_str_col
+
+_KS_SIGNIFICANCE = 0.05
+_CATEGORY_DRIFT_THRESHOLD = 0.05
+
+
+def analyze(old_df: pd.DataFrame, new_df: pd.DataFrame) -> dict[str, Any]:
+    distribution_drift: dict[str, float] = {}
+    category_drift: dict[str, dict[str, Any]] = {}
+    missing_value_drift: dict[str, float] = {}
+    drifted: set[str] = set()
+
+    common_cols = set(old_df.columns) & set(new_df.columns)
+
+    for col in sorted(common_cols):
+        old_s = old_df[col]
+        new_s = new_df[col]
+
+        # Missing-value drift
+        old_null = old_s.isna().mean()
+        new_null = new_s.isna().mean()
+        mv_change = round(abs(new_null - old_null) * 100, 2)
+        missing_value_drift[col] = mv_change
+        if mv_change > 5.0:
+            drifted.add(col)
+
+        # Numeric distribution drift (KS test)
+        if pd.api.types.is_numeric_dtype(old_s) and pd.api.types.is_numeric_dtype(new_s):
+            old_clean = old_s.dropna().astype(float)
+            new_clean = new_s.dropna().astype(float)
+            if len(old_clean) > 1 and len(new_clean) > 1:
+                stat, p_value = stats.ks_2samp(old_clean, new_clean)
+                distribution_drift[col] = round(float(stat), 4)
+                if p_value < _KS_SIGNIFICANCE:
+                    drifted.add(col)
+
+        # Categorical distribution drift
+        elif is_str_col(old_s) and is_str_col(new_s):
+            old_freq = old_s.value_counts(normalize=True)
+            new_freq = new_s.value_counts(normalize=True)
+            all_cats = set(old_freq.index) | set(new_freq.index)
+            changes: dict[str, Any] = {}
+            for cat in all_cats:
+                old_p = float(old_freq.get(cat, 0.0))
+                new_p = float(new_freq.get(cat, 0.0))
+                if abs(new_p - old_p) > _CATEGORY_DRIFT_THRESHOLD:
+                    changes[str(cat)] = {
+                        "old_pct": round(old_p * 100, 2),
+                        "new_pct": round(new_p * 100, 2),
+                    }
+            if changes:
+                category_drift[col] = changes
+                drifted.add(col)
+
+    return {
+        "distribution_drift": distribution_drift,
+        "category_drift": category_drift,
+        "missing_value_drift": missing_value_drift,
+        "drifted_columns": sorted(drifted),
+    }

dataruff-0.1.0/datadoctor/analyzers/duplicate.py

@@ -0,0 +1,33 @@
+from __future__ import annotations
+
+import pandas as pd
+
+from datadoctor.models import Issue
+
+_HIGH_THRESHOLD = 0.10  # >10% duplicates → high severity
+
+
+def analyze(df: pd.DataFrame) -> list[Issue]:
+    if df.empty:
+        return []
+
+    mask = df.duplicated()
+    dup_count = int(mask.sum())
+
+    if dup_count == 0:
+        return []
+
+    pct = dup_count / len(df)
+    severity = "high" if pct > _HIGH_THRESHOLD else "medium"
+
+    return [
+        Issue(
+            type="duplicate_rows",
+            severity=severity,
+            count=dup_count,
+            details={
+                "percentage": round(pct * 100, 2),
+                "duplicate_indices": df[mask].index.tolist(),
+            },
+        )
+    ]