cleanalytix 0.1.0__tar.gz

cleanalytix-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Probot-DATA contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

cleanalytix-0.1.0/PKG-INFO

@@ -0,0 +1,235 @@
+Metadata-Version: 2.4
+Name: cleanalytix
+Version: 0.1.0
+Summary: Cleanalytix is a modular Python library for profiling, scoring, and cleaning tabular datasets.
+Author: Probot-DATA contributors
+License: MIT License
+        
+        Copyright (c) 2026 Probot-DATA contributors
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+        
+Project-URL: Homepage, https://github.com/Probot-DATA/Cleanalytix_Repo
+Project-URL: Repository, https://github.com/Probot-DATA/Cleanalytix_Repo
+Project-URL: Issues, https://github.com/Probot-DATA/Cleanalytix_Repo/issues
+Keywords: data quality,data cleaning,data profiling,EDA,machine learning,pandas
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pandas>=1.5.0
+Requires-Dist: numpy>=1.23.0
+Requires-Dist: scikit-learn>=1.1.0
+Requires-Dist: nltk>=3.8.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0; extra == "dev"
+Requires-Dist: ruff>=0.4.0; extra == "dev"
+Dynamic: license-file
+
+# Cleanalytix
+
+`Cleanalytix` is a Python library for profiling, scoring, cleaning, and monitoring the quality of tabular datasets with a single pipeline.
+
+It is designed for pandas-first workflows and supports:
+
+- baseline dataset profiling and scoring
+- optional cleaning recommendations and automatic cleaning
+- optional production/new-dataset monitoring
+- optional business rules, thresholds, weights, and type inference for new data
+
+## Installation
+
+From a source checkout:
+
+```bash
+git clone https://github.com/Probot-DATA/Cleanalytix_Repo
+cd Cleanalytix_Repo
+pip install -e ".[dev]"
+```
+
+Once this project is published to PyPI, the install command will be:
+
+```bash
+pip install cleanalytix
+```
+
+Runtime requirements:
+
+- Python 3.9+
+- pandas
+- numpy
+- scikit-learn
+- nltk
+
+## Quick Start
+
+```python
+import pandas as pd
+from cleanalytix import Run_DQ_Pipeline
+
+df = pd.read_csv("my_data.csv")
+
+result = Run_DQ_Pipeline(
+    dataset_names=["my_dataset"],
+    dataset_list=[df],
+)
+
+print(result["base_data"]["dirty_scores"])
+print(result["base_data"]["meta_before_cleaning"])
+print(result["base_data"]["recommendations"])
+```
+
+## Production / Monitoring Example
+
+```python
+import pandas as pd
+from cleanalytix import Run_DQ_Pipeline
+
+train_df = pd.read_csv("train.csv")
+prod_df = pd.read_csv("production.csv")
+
+rules = {
+    "age": lambda value: pd.isna(value) or 0 <= float(value) <= 120,
+}
+
+result = Run_DQ_Pipeline(
+    dataset_names=["customers"],
+    dataset_list=[train_df],
+    new_dataset_list=[prod_df],
+    rules=rules,
+    cleaning=True,
+    interactive=False,
+    score_mode="exponential",
+)
+
+print(result["base_data"]["dirty_scores"])
+print(result["base_data"]["cleaned_scores"])
+print(result["prod_data"]["dirty_scores"])
+print(result["prod_data"]["change_log"])
+```
+
+## Public API
+
+The primary entrypoint is:
+
+```python
+from cleanalytix import Run_DQ_Pipeline
+```
+
+Additional building blocks are also exported:
+
+```python
+from cleanalytix import (
+    Compute_DQ_Score,
+    DEFAULT_THRESHOLDS,
+    generate_meta,
+    cleaning_recommendations,
+    get_cleaned_data,
+    get_table_for_DQ_computation,
+    summarize_dataset_health,
+    learn_reference_profile,
+    adjust_prod_meta_with_reference,
+    infer_and_fix_types,
+)
+```
+
+## Pipeline Output Structure
+
+`Run_DQ_Pipeline` returns:
+
+```python
+{
+    "base_data": {...},
+    "prod_data": {...},
+}
+```
+
+Each block preserves the same keys:
+
+- `dirty_scores`
+- `cleaned_scores`
+- `cleaned_datasets`
+- `meta_before_cleaning`
+- `meta_after_cleaning`
+- `recommendations`
+- `change_log`
+- `summarized_before`
+- `summarized_after`
+- `main_metrics_before`
+- `main_metrics_after`
+
+## Examples
+
+Runnable examples live in [examples](./examples):
+
+```bash
+python examples/simple_usage.py
+python examples/production_usage.py
+```
+
+These examples assume the package has already been installed in the active environment.
+
+## Validation
+
+The [validation](./validation) folder contains a portable real-world validation workflow.
+
+- Large raw datasets are intentionally not committed to the repository.
+- Put the expected files under `validation/datasets/` by following
+  [validation/datasets/README.md](./validation/datasets/README.md).
+- Run the validation script:
+
+```bash
+python validation/run_validation.py
+```
+
+- The script saves non-empty outputs to `validation/outputs/<dataset_name>/`.
+- The notebook [validation/main.ipynb](./validation/main.ipynb) uses the same relative-path workflow.
+
+## Repository Layout
+
+- `cleanalytix/` - installable library package
+- `examples/` - small runnable examples
+- `tests/` - smoke tests and lightweight sample fixtures
+- `validation/` - public-friendly validation workflow and output folder
+- `archive/legacy/` - historical prototype notebook/code kept for reference, not for active use
+
+## Known Limitations
+
+- Validation datasets are not bundled with the repository.
+- The yellow taxi validation workflow samples the first `20,000` rows from each configured monthly file to match the original project workflow and to keep validation practical.
+- Interactive cleaning is intended for notebook/CLI use and will prompt for input when `interactive=True`.
+
+## Contributing
+
+See [CONTRIBUTING.md](./CONTRIBUTING.md).
+
+## License
+
+[MIT](./LICENSE) (c) 2026 Probot-DATA contributors

cleanalytix-0.1.0/README.md

@@ -0,0 +1,180 @@
+# Cleanalytix
+
+`Cleanalytix` is a Python library for profiling, scoring, cleaning, and monitoring the quality of tabular datasets with a single pipeline.
+
+It is designed for pandas-first workflows and supports:
+
+- baseline dataset profiling and scoring
+- optional cleaning recommendations and automatic cleaning
+- optional production/new-dataset monitoring
+- optional business rules, thresholds, weights, and type inference for new data
+
+## Installation
+
+From a source checkout:
+
+```bash
+git clone https://github.com/Probot-DATA/Cleanalytix_Repo
+cd Cleanalytix_Repo
+pip install -e ".[dev]"
+```
+
+Once this project is published to PyPI, the install command will be:
+
+```bash
+pip install cleanalytix
+```
+
+Runtime requirements:
+
+- Python 3.9+
+- pandas
+- numpy
+- scikit-learn
+- nltk
+
+## Quick Start
+
+```python
+import pandas as pd
+from cleanalytix import Run_DQ_Pipeline
+
+df = pd.read_csv("my_data.csv")
+
+result = Run_DQ_Pipeline(
+    dataset_names=["my_dataset"],
+    dataset_list=[df],
+)
+
+print(result["base_data"]["dirty_scores"])
+print(result["base_data"]["meta_before_cleaning"])
+print(result["base_data"]["recommendations"])
+```
+
+## Production / Monitoring Example
+
+```python
+import pandas as pd
+from cleanalytix import Run_DQ_Pipeline
+
+train_df = pd.read_csv("train.csv")
+prod_df = pd.read_csv("production.csv")
+
+rules = {
+    "age": lambda value: pd.isna(value) or 0 <= float(value) <= 120,
+}
+
+result = Run_DQ_Pipeline(
+    dataset_names=["customers"],
+    dataset_list=[train_df],
+    new_dataset_list=[prod_df],
+    rules=rules,
+    cleaning=True,
+    interactive=False,
+    score_mode="exponential",
+)
+
+print(result["base_data"]["dirty_scores"])
+print(result["base_data"]["cleaned_scores"])
+print(result["prod_data"]["dirty_scores"])
+print(result["prod_data"]["change_log"])
+```
+
+## Public API
+
+The primary entrypoint is:
+
+```python
+from cleanalytix import Run_DQ_Pipeline
+```
+
+Additional building blocks are also exported:
+
+```python
+from cleanalytix import (
+    Compute_DQ_Score,
+    DEFAULT_THRESHOLDS,
+    generate_meta,
+    cleaning_recommendations,
+    get_cleaned_data,
+    get_table_for_DQ_computation,
+    summarize_dataset_health,
+    learn_reference_profile,
+    adjust_prod_meta_with_reference,
+    infer_and_fix_types,
+)
+```
+
+## Pipeline Output Structure
+
+`Run_DQ_Pipeline` returns:
+
+```python
+{
+    "base_data": {...},
+    "prod_data": {...},
+}
+```
+
+Each block preserves the same keys:
+
+- `dirty_scores`
+- `cleaned_scores`
+- `cleaned_datasets`
+- `meta_before_cleaning`
+- `meta_after_cleaning`
+- `recommendations`
+- `change_log`
+- `summarized_before`
+- `summarized_after`
+- `main_metrics_before`
+- `main_metrics_after`
+
+## Examples
+
+Runnable examples live in [examples](./examples):
+
+```bash
+python examples/simple_usage.py
+python examples/production_usage.py
+```
+
+These examples assume the package has already been installed in the active environment.
+
+## Validation
+
+The [validation](./validation) folder contains a portable real-world validation workflow.
+
+- Large raw datasets are intentionally not committed to the repository.
+- Put the expected files under `validation/datasets/` by following
+  [validation/datasets/README.md](./validation/datasets/README.md).
+- Run the validation script:
+
+```bash
+python validation/run_validation.py
+```
+
+- The script saves non-empty outputs to `validation/outputs/<dataset_name>/`.
+- The notebook [validation/main.ipynb](./validation/main.ipynb) uses the same relative-path workflow.
+
+## Repository Layout
+
+- `cleanalytix/` - installable library package
+- `examples/` - small runnable examples
+- `tests/` - smoke tests and lightweight sample fixtures
+- `validation/` - public-friendly validation workflow and output folder
+- `archive/legacy/` - historical prototype notebook/code kept for reference, not for active use
+
+## Known Limitations
+
+- Validation datasets are not bundled with the repository.
+- The yellow taxi validation workflow samples the first `20,000` rows from each configured monthly file to match the original project workflow and to keep validation practical.
+- Interactive cleaning is intended for notebook/CLI use and will prompt for input when `interactive=True`.
+
+## Contributing
+
+See [CONTRIBUTING.md](./CONTRIBUTING.md).
+
+## License
+
+[MIT](./LICENSE) (c) 2026 Probot-DATA contributors

cleanalytix-0.1.0/cleanalytix/__init__.py

@@ -0,0 +1,39 @@
+"""
+Cleanalytix - Data Quality Framework.
+
+Quick start
+-----------
+>>> from cleanalytix import Run_DQ_Pipeline
+>>> result = Run_DQ_Pipeline(["my_dataset"], [df])
+>>> print(result["base_data"]["dirty_scores"])
+"""
+
+from .version import __author__, __version__
+from .pipeline import Run_DQ_Pipeline
+from .compute_dq_score import Compute_DQ_Score, DEFAULT_THRESHOLDS
+from .generate_meta import generate_meta
+from .cleaning_recommendations import cleaning_recommendations
+from .get_cleaned_data import get_cleaned_data
+from .get_table_for_DQ_computation import get_table_for_DQ_computation
+from .summarize_dataset_health import summarize_dataset_health
+from .adjust_prod_meta import learn_reference_profile, adjust_prod_meta_with_reference
+from .preprocess_types import infer_and_fix_types
+
+run_dq_pipeline = Run_DQ_Pipeline
+
+__all__ = [
+    "Run_DQ_Pipeline",
+    "run_dq_pipeline",
+    "Compute_DQ_Score",
+    "DEFAULT_THRESHOLDS",
+    "__author__",
+    "__version__",
+    "generate_meta",
+    "cleaning_recommendations",
+    "get_cleaned_data",
+    "get_table_for_DQ_computation",
+    "summarize_dataset_health",
+    "learn_reference_profile",
+    "adjust_prod_meta_with_reference",
+    "infer_and_fix_types",
+]

cleanalytix-0.1.0/cleanalytix/adjust_prod_meta.py

@@ -0,0 +1,135 @@
+# cleanalytix/adjust_prod_meta.py
+import pandas as pd
+import numpy as np
+
+EPS = 1e-9
+
+def learn_reference_profile(dataset_names, dataset_list, rare_pct_threshold=5.0):
+    """
+    Learn per-column numeric bounds and training-popular categories from reference datasets.
+    Returns nested dict: profile[dataset_name][column] = {type, lower, upper, popular, skewness_level}
+    """
+    profile = {}
+    for ds_name, df in zip(dataset_names, dataset_list):
+        profile[ds_name] = {}
+        for col in df.columns:
+            ser = df[col].dropna()
+            if ser.empty:
+                profile[ds_name][col] = {"type": "empty"}
+                continue
+
+            if pd.api.types.is_numeric_dtype(ser):
+                skew = ser.skew() if len(ser) > 2 else 0.0
+                abs_skew = abs(skew)
+                if abs_skew < 0.5:
+                    mean = ser.mean()
+                    std = ser.std(ddof=0) if ser.size > 1 else 0.0
+                    if std == 0 or np.isnan(std):
+                        lower, upper = -np.inf, np.inf
+                    else:
+                        lower, upper = mean - 3.0 * std, mean + 3.0 * std
+                    method = "z"
+                elif abs_skew < 1:
+                    Q1, Q3 = ser.quantile(0.25), ser.quantile(0.75)
+                    IQR = Q3 - Q1
+                    if IQR == 0 or np.isnan(IQR):
+                        lower, upper = -np.inf, np.inf
+                    else:
+                        lower, upper = Q1 - 1.5 * IQR, Q3 + 1.5 * IQR
+                    method = "iqr"
+                else:
+                    median = ser.median()
+                    MAD = np.median(np.abs(ser - median))
+                    if MAD == 0 or np.isnan(MAD):
+                        lower, upper = -np.inf, np.inf
+                    else:
+                        factor = 3.5 / 0.6745
+                        delta = factor * MAD
+                        lower, upper = median - delta, median + delta
+                    method = "mad"
+
+                profile[ds_name][col] = {
+                    "type": "numeric",
+                    "lower": float(lower) if np.isfinite(lower) else np.nan,
+                    "upper": float(upper) if np.isfinite(upper) else np.nan,
+                    "method": method,
+                    "skewness": float(skew)
+                }
+            else:
+                vals = ser.astype(str)
+                freq = vals.value_counts(normalize=True) * 100
+                popular = freq[freq > rare_pct_threshold].index.tolist()
+                profile[ds_name][col] = {
+                    "type": "categorical",
+                    "popular": popular
+                }
+    return profile
+
+
+def adjust_prod_meta_with_reference(profile, reference_meta, prod_meta, prod_dataset_names, prod_dataset_list):
+    """
+    Adjust production meta statistics using the learned reference profile.
+
+    Returns a deep copy of ``prod_meta`` with two fields overwritten:
+
+    - ``outlier_count``: recomputed using the numeric bounds (mean±3σ / IQR / MAD)
+      derived from the reference (training) dataset, so production outliers are
+      judged against the training distribution rather than their own.
+    - ``rare_category_percent``: recomputed as the fraction of production values
+      that do not appear in the training dataset's popular category list.
+
+    Parameters
+    ----------
+    profile : dict
+        Output of ``learn_reference_profile``.
+    reference_meta : pd.DataFrame
+        Meta table for the reference (training) split — accepted but not
+        currently consumed; reserved for future drift-metric alignment.
+    prod_meta : pd.DataFrame
+        Meta table for the production dataset (output of ``generate_meta``).
+    prod_dataset_names : list of str
+    prod_dataset_list : list of pd.DataFrame
+    """
+    prod_meta = prod_meta.copy(deep=True)
+
+    for ds_name, df in zip(prod_dataset_names, prod_dataset_list):
+        for col in df.columns:
+            mask = (prod_meta["dataset_name"] == ds_name) & (prod_meta["column_name"] == col)
+            if not mask.any():
+                continue
+
+            info = profile.get(ds_name, {}).get(col, None)
+            series = df[col].dropna()
+
+            # numeric outliers using reference bounds
+            if info and info.get("type") == "numeric" and len(series) > 0:
+                lb, ub = info.get("lower", np.nan), info.get("upper", np.nan)
+                outliers = 0
+                if pd.notna(lb) and pd.notna(ub):
+                    # count finite bounds only
+                    try:
+                        outliers = int(((series < lb) | (series > ub)).sum())
+                    except Exception:
+                        outliers = 0
+                else:
+                    # fallback: keep existing value in prod_meta (or recompute if desired)
+                    # we choose to recompute with the same logic as generate_meta fallback
+                    outliers = int(prod_meta.loc[mask, "outlier_count"].fillna(0).values[0])
+
+                prod_meta.loc[mask, "outlier_count"] = int(outliers)
+
+            # categorical rare percent relative to training popular categories
+            elif info and info.get("type") == "categorical" and len(series) > 0:
+                popular = info.get("popular", [])
+                vals = series.astype(str)
+                if len(popular) == 0:
+                    # no popular categories in training -> set rare percent to 0 (or keep existing)
+                    rare_percent = float(prod_meta.loc[mask, "rare_category_percent"].fillna(0).values[0])
+                else:
+                    rare_percent = float((~vals.isin(popular)).mean() * 100)
+                prod_meta.loc[mask, "rare_category_percent"] = rare_percent
+
+
+    # ensure NaT replaced
+    prod_meta.replace({pd.NaT: np.nan}, inplace=True)
+    return prod_meta