universal-data-cleaner 2.2.0__tar.gz → 2.3.0__tar.gz

{universal_data_cleaner-2.2.0/src/universal_data_cleaner.egg-info → universal_data_cleaner-2.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: universal-data-cleaner
-Version: 2.2.0
+Version: 2.3.0
 Summary: Production-ready, scikit-learn-compatible data cleaning transformer for pandas.
 Author: Haseeb Khan
 License-Expression: MIT
@@ -241,6 +241,61 @@ restored = DataContract.from_json(payload)
 # restored = DataContract.from_yaml(contract.to_yaml())
 ```
 
+## Audit trail, manifests, and fingerprints
+
+Every learned or configured decision is recorded during `fit()` and exposed as
+a tidy DataFrame:
+
+```python
+log = cleaner.decision_log()
+# columns: column, action, reason, source, before, after
+log[log["action"] == "cap_outliers"]
+```
+
+`source` says whether a rule was `learned` from the training data, supplied via
+`config`, declared in a `contract`, or `inferred` by the library. Imputation
+values, outlier bounds, datetime expansion, missing indicators, dropped
+constants, and identifier handling are all explained instead of hidden.
+
+Build a JSON-serializable manifest for logs or a model registry:
+
+```python
+import json
+
+manifest = cleaner.to_manifest(X_train, include_fingerprint=True)
+# library_version, artifact_version, environment, config, contract,
+# input_schema, output_columns, fitted, decisions, fingerprint
+
+json.dumps(manifest)  # safe to ship to a registry or a log line
+```
+
+`save()` embeds the manifest and the training fingerprint in the artifact, and
+`load()` restores them:
+
+```python
+cleaner.save("cleaner.joblib")
+restored = DataCleaner.load("cleaner.joblib")
+restored.artifact_manifest()
+```
+
+Fingerprints are stable SHA-256 hashes of the schema plus summary statistics.
+They never hash raw values, so they are safe to log:
+
+```python
+from data_cleaner import (
+    assert_schema_stable,
+    compare_fingerprints,
+    fingerprint_dataframe,
+)
+
+expected = fingerprint_dataframe(X_train)
+same = compare_fingerprints(expected, fingerprint_dataframe(X_new))
+assert_schema_stable(X_new, expected)  # raises ValueError on schema/stat drift
+```
+
+Set `CleanerConfig(fingerprint_on_fit=True)` to compute the training fingerprint
+once during `fit()`, so `to_manifest(include_fingerprint=True)` needs no data.
+
 ## scikit-learn Pipeline
 
 ```python
@@ -279,11 +334,19 @@ predictions = pipeline.predict(X_test)
 | --- | --- |
 | `CleanerConfig` | Validated cleaning configuration |
 | `DataCleaner` | Fit/transform cleaner with save/load and drift reporting |
+| `CleaningDecision` | One auditable learned or configured decision |
 | `ColumnSpec` | Declarative expectations and semantics for one column |
 | `DataContract` | A set of column specs, with inference and (de)serialization |
 | `DriftReport` | Findings from a contract check, with a Markdown summary |
 | `DataCleaner.drift_report()` | Drift findings from the last `transform` |
 | `DataCleaner.fitted_contract()` | The contract used by the fitted cleaner |
+| `DataCleaner.decision_log()` | Tidy DataFrame of cleaning decisions |
+| `DataCleaner.to_manifest()` | JSON-serializable artifact manifest |
+| `DataCleaner.artifact_manifest()` | Manifest stored in a loaded artifact |
+| `DataCleaner.fingerprint()` | Stable schema + statistics fingerprint |
+| `fingerprint_dataframe` | Fingerprint any DataFrame for lineage |
+| `compare_fingerprints` | Compare two fingerprints |
+| `assert_schema_stable` | Raise when a frame's fingerprint changed |
 | `profile_dataframe` | Column-level missing/type/cardinality profile |
 | `dataset_summary` | Row/column/missing/duplicate summary |
 | `drop_duplicate_rows` | Explicit duplicate removal helper |
@@ -305,6 +368,10 @@ predictions = pipeline.predict(X_test)
   represented by the generated feature NaNs plus an optional missing indicator.
 - IQR capping is skipped automatically when the IQR is zero, so constant or
   near-constant numeric columns are not accidentally destroyed.
+- Persist `cleaner.to_manifest(X_train, include_fingerprint=True)` alongside the
+  model so you can audit exactly which cleaning rules produced an artifact.
+- `decision_log()` explains every learned or configured rule; ship it to your
+  experiment tracker when a model's preprocessing changes.
 
 ## Project layout
 

{universal_data_cleaner-2.2.0 → universal_data_cleaner-2.3.0}/README.md

@@ -208,6 +208,61 @@ restored = DataContract.from_json(payload)
 # restored = DataContract.from_yaml(contract.to_yaml())
 ```
 
+## Audit trail, manifests, and fingerprints
+
+Every learned or configured decision is recorded during `fit()` and exposed as
+a tidy DataFrame:
+
+```python
+log = cleaner.decision_log()
+# columns: column, action, reason, source, before, after
+log[log["action"] == "cap_outliers"]
+```
+
+`source` says whether a rule was `learned` from the training data, supplied via
+`config`, declared in a `contract`, or `inferred` by the library. Imputation
+values, outlier bounds, datetime expansion, missing indicators, dropped
+constants, and identifier handling are all explained instead of hidden.
+
+Build a JSON-serializable manifest for logs or a model registry:
+
+```python
+import json
+
+manifest = cleaner.to_manifest(X_train, include_fingerprint=True)
+# library_version, artifact_version, environment, config, contract,
+# input_schema, output_columns, fitted, decisions, fingerprint
+
+json.dumps(manifest)  # safe to ship to a registry or a log line
+```
+
+`save()` embeds the manifest and the training fingerprint in the artifact, and
+`load()` restores them:
+
+```python
+cleaner.save("cleaner.joblib")
+restored = DataCleaner.load("cleaner.joblib")
+restored.artifact_manifest()
+```
+
+Fingerprints are stable SHA-256 hashes of the schema plus summary statistics.
+They never hash raw values, so they are safe to log:
+
+```python
+from data_cleaner import (
+    assert_schema_stable,
+    compare_fingerprints,
+    fingerprint_dataframe,
+)
+
+expected = fingerprint_dataframe(X_train)
+same = compare_fingerprints(expected, fingerprint_dataframe(X_new))
+assert_schema_stable(X_new, expected)  # raises ValueError on schema/stat drift
+```
+
+Set `CleanerConfig(fingerprint_on_fit=True)` to compute the training fingerprint
+once during `fit()`, so `to_manifest(include_fingerprint=True)` needs no data.
+
 ## scikit-learn Pipeline
 
 ```python
@@ -246,11 +301,19 @@ predictions = pipeline.predict(X_test)
 | --- | --- |
 | `CleanerConfig` | Validated cleaning configuration |
 | `DataCleaner` | Fit/transform cleaner with save/load and drift reporting |
+| `CleaningDecision` | One auditable learned or configured decision |
 | `ColumnSpec` | Declarative expectations and semantics for one column |
 | `DataContract` | A set of column specs, with inference and (de)serialization |
 | `DriftReport` | Findings from a contract check, with a Markdown summary |
 | `DataCleaner.drift_report()` | Drift findings from the last `transform` |
 | `DataCleaner.fitted_contract()` | The contract used by the fitted cleaner |
+| `DataCleaner.decision_log()` | Tidy DataFrame of cleaning decisions |
+| `DataCleaner.to_manifest()` | JSON-serializable artifact manifest |
+| `DataCleaner.artifact_manifest()` | Manifest stored in a loaded artifact |
+| `DataCleaner.fingerprint()` | Stable schema + statistics fingerprint |
+| `fingerprint_dataframe` | Fingerprint any DataFrame for lineage |
+| `compare_fingerprints` | Compare two fingerprints |
+| `assert_schema_stable` | Raise when a frame's fingerprint changed |
 | `profile_dataframe` | Column-level missing/type/cardinality profile |
 | `dataset_summary` | Row/column/missing/duplicate summary |
 | `drop_duplicate_rows` | Explicit duplicate removal helper |
@@ -272,6 +335,10 @@ predictions = pipeline.predict(X_test)
   represented by the generated feature NaNs plus an optional missing indicator.
 - IQR capping is skipped automatically when the IQR is zero, so constant or
   near-constant numeric columns are not accidentally destroyed.
+- Persist `cleaner.to_manifest(X_train, include_fingerprint=True)` alongside the
+  model so you can audit exactly which cleaning rules produced an artifact.
+- `decision_log()` explains every learned or configured rule; ship it to your
+  experiment tracker when a model's preprocessing changes.
 
 ## Project layout
 

{universal_data_cleaner-2.2.0 → universal_data_cleaner-2.3.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "universal-data-cleaner"
-version = "2.2.0"
+version = "2.3.0"
 description = "Production-ready, scikit-learn-compatible data cleaning transformer for pandas."
 readme = "README.md"
 license = "MIT"

{universal_data_cleaner-2.2.0 → universal_data_cleaner-2.3.0}/src/data_cleaner/__init__.py

@@ -13,9 +13,15 @@ exactly as it did when the library shipped as a single module.
 from __future__ import annotations
 
 from ._version import __version__
+from .artifacts import (
+    assert_schema_stable,
+    compare_fingerprints,
+    fingerprint_dataframe,
+)
 from .cleaner import DataCleaner
 from .config import CleanerConfig
 from .contract import ColumnSpec, DataContract, DriftReport
+from .decisions import CleaningDecision
 from .profiling import (
     assert_frame_clean,
     dataset_summary,
@@ -25,13 +31,17 @@ from .profiling import (
 
 __all__ = [
     "CleanerConfig",
+    "CleaningDecision",
     "ColumnSpec",
     "DataCleaner",
     "DataContract",
     "DriftReport",
     "__version__",
     "assert_frame_clean",
+    "assert_schema_stable",
+    "compare_fingerprints",
     "dataset_summary",
     "drop_duplicate_rows",
+    "fingerprint_dataframe",
     "profile_dataframe",
 ]

{universal_data_cleaner-2.2.0 → universal_data_cleaner-2.3.0}/src/data_cleaner/_version.py

@@ -1,4 +1,4 @@
 """Single source of truth for the package version."""
 
-__version__ = "2.2.0"
-_ARTIFACT_VERSION = 1
+__version__ = "2.3.0"
+_ARTIFACT_VERSION = 2

universal_data_cleaner-2.3.0/src/data_cleaner/artifacts.py

@@ -0,0 +1,171 @@
+"""Reproducibility helpers: data fingerprints and environment metadata."""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import math
+import platform
+from collections.abc import Sequence
+from typing import Any
+
+import numpy as np
+import pandas as pd
+import sklearn
+
+from ._roles import _series_kind
+from ._version import __version__
+
+
+def package_versions() -> dict[str, str]:
+    """Return the versions of the runtime that produced an artifact."""
+    return {
+        "python": platform.python_version(),
+        "pandas": pd.__version__,
+        "numpy": np.__version__,
+        "scikit_learn": sklearn.__version__,
+        "universal_data_cleaner": __version__,
+    }
+
+
+def _json_default(value: object) -> str:
+    """Render unknown scalar types (timestamps, decimals) for JSON encoding."""
+    return str(value)
+
+
+def _canonical_number(value: Any) -> float | str | None:
+    """Normalize a number so JSON hashing is stable and NaN-safe."""
+    if value is None:
+        return None
+    try:
+        number = float(value)
+    except (TypeError, ValueError):
+        return str(value)
+    if math.isnan(number):
+        return "nan"
+    if math.isinf(number):
+        return "inf" if number > 0 else "-inf"
+    return round(number, 10)
+
+
+def _column_stats(series: pd.Series) -> dict[str, object]:
+    kind = _series_kind(series)
+    stats: dict[str, object] = {
+        "name": str(series.name),
+        "kind": kind,
+        "nulls": int(series.isna().sum()),
+    }
+
+    if kind == "numeric":
+        numeric = pd.to_numeric(series, errors="coerce")
+        numeric = numeric.replace([np.inf, -np.inf], np.nan).dropna()
+        stats.update(
+            {
+                "count": int(numeric.size),
+                "mean": _canonical_number(numeric.mean()) if numeric.size else None,
+                "std": _canonical_number(numeric.std()) if numeric.size > 1 else None,
+                "min": _canonical_number(numeric.min()) if numeric.size else None,
+                "max": _canonical_number(numeric.max()) if numeric.size else None,
+            }
+        )
+    elif kind == "boolean":
+        valid = series.dropna()
+        flags = valid.astype(bool)
+        stats.update(
+            {
+                "count": int(valid.size),
+                "true": int(flags.sum()),
+                "false": int((~flags).sum()),
+            }
+        )
+    elif kind == "datetime":
+        valid = series.dropna()
+        stats.update(
+            {
+                "count": int(valid.size),
+                "min": str(valid.min()) if not valid.empty else None,
+                "max": str(valid.max()) if not valid.empty else None,
+            }
+        )
+    else:
+        try:
+            unique: int | None = int(series.nunique(dropna=True))
+        except TypeError:
+            unique = None
+        stats.update({"count": int(series.notna().sum()), "unique": unique})
+
+    return stats
+
+
+def fingerprint_dataframe(
+    frame: pd.DataFrame, columns: Sequence[str] | None = None
+) -> str:
+    """Return a stable SHA-256 fingerprint of a frame's schema and summary stats.
+
+    The hash deliberately excludes raw values: it is a lineage fingerprint, not
+    a row-level checksum. Column order participates in the hash, and summary
+    statistics are rounded so the result is stable across platforms.
+
+    Parameters
+    ----------
+    frame:
+        DataFrame to fingerprint.
+    columns:
+        Optional subset/order of columns. Defaults to all columns in frame
+        order.
+
+    Raises
+    ------
+    TypeError
+        If ``frame`` is not a pandas DataFrame.
+    ValueError
+        If ``frame`` has duplicate column names.
+    KeyError
+        If a requested column is missing from ``frame``.
+    """
+    if not isinstance(frame, pd.DataFrame):
+        raise TypeError("frame must be a pandas DataFrame")
+    if frame.columns.has_duplicates:
+        duplicates = frame.columns[frame.columns.duplicated()].tolist()
+        raise ValueError(f"Duplicate column names are not allowed: {duplicates}")
+
+    if columns is None:
+        selected = list(frame.columns)
+    else:
+        selected = [str(column) for column in columns]
+        missing = [column for column in selected if column not in frame.columns]
+        if missing:
+            raise KeyError(f"Columns not found in frame: {missing}")
+
+    payload = {
+        "n_rows": len(frame),
+        "n_columns": len(selected),
+        "columns": [_column_stats(frame[column]) for column in selected],
+    }
+    encoded = json.dumps(
+        payload,
+        sort_keys=True,
+        separators=(",", ":"),
+        default=_json_default,
+    )
+    return hashlib.sha256(encoded.encode("utf-8")).hexdigest()
+
+
+def compare_fingerprints(left: str, right: str) -> bool:
+    """Return ``True`` when two fingerprints are identical."""
+    return left == right
+
+
+def assert_schema_stable(
+    frame: pd.DataFrame, expected: str, *, name: str = "frame"
+) -> None:
+    """Raise ``ValueError`` when ``frame`` does not match ``expected``.
+
+    ``expected`` is a fingerprint produced by :func:`fingerprint_dataframe`.
+    """
+    actual = fingerprint_dataframe(frame)
+    if not compare_fingerprints(actual, expected):
+        raise ValueError(
+            f"{name} fingerprint {actual} does not match expected {expected}. "
+            "The schema or summary statistics changed."
+        )