tput 0.1.0__tar.gz

tput-0.1.0/PKG-INFO

@@ -0,0 +1,238 @@
+Metadata-Version: 2.4
+Name: tput
+Version: 0.1.0
+Summary: Automated DataFrame audit library for ML pipelines
+License: MIT
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: pandas>=1.5
+Requires-Dist: scipy>=1.10
+Requires-Dist: scikit-learn>=1.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+
+# tput — Automated DataFrame Audit for ML
+
+`tput` is a Python library for fast, opinionated data auditing before ML pipelines. Pass it a DataFrame and get a structured report that surfaces everything that matters before you start modelling: missing values, outliers, skewness, type issues, categorical noise, correlations, multicollinearity, and more.
+
+```python
+from tput import quick_report
+
+report = quick_report(df, target="SalePrice")
+report.show()       # full column-by-column report
+report.summary()    # condensed overview for large datasets
+```
+
+---
+
+## Installation
+
+```bash
+# Not yet on PyPI — clone and import locally
+git clone https://github.com/yourname/tput
+cd tput
+pip install -e .
+```
+
+Dependencies: `pandas>=1.5`, `scipy>=1.10`, `scikit-learn>=1.0`
+
+---
+
+## Quickstart
+
+```python
+import pandas as pd
+from tput import quick_report
+
+df = pd.read_csv("titanic.csv")
+
+# Basic audit
+report = quick_report(df)
+report.show()
+
+# With target column — unlocks classification/regression analysis
+report = quick_report(df, target="Survived")
+report.show()
+
+# Condensed view — useful for wide datasets (50+ columns)
+report.summary()
+
+# Programmatic access
+report.get("nan_analysis")
+report.get("target_analysis")
+report.warnings
+```
+
+---
+
+## What it detects
+
+### Data quality
+| Step | What it catches |
+|---|---|
+| `overview` | Shape, dtypes |
+| `visible_missing` | NaN counts and percentages per column |
+| `duplicates` | Fully duplicated rows |
+| `categorical_profile` | Cardinality, mode, top/bottom values, hidden missing (`""`, `"   "`), case collisions (`"Paris"` vs `"paris"`) |
+| `type_issues` | String columns that should be numeric or datetime |
+| `row_analysis` | Rows with too many missing fields (default: ≥50%) — flagged for removal |
+
+### Statistical analysis
+| Step | What it catches |
+|---|---|
+| `numeric_profile` | mean, std, Q1/Q3, min/max, mode |
+| `skewness` | Asymmetry level (symmetric / moderate / high) + recommended transform (log1p, sqrt, reflect + log1p, yeo-johnson) |
+| `outliers` | IQR on symmetric columns, MAD on skewed ones — auto-selected per column |
+| `nan_analysis` | MCAR / MAR / MNAR classification + imputation strategy (mean, median, mode, knn_or_regression). High missing rate (>70%) flagged as potential semantic NaN (e.g. "no pool", "no garage") |
+| `feature_quality` | quasi-constant columns, low-cardinality numerics, suspected ID columns |
+
+### Multicollinearity & associations
+| Step | What it catches |
+|---|---|
+| `correlations` | Pearson pairs above threshold (default: 0.85) + Chi² / Cramér's V (bias-corrected) for categorical pairs |
+| `vif` | Variance Inflation Factor per continuous column — flags multicollinearity for linear models |
+
+### Target analysis (`target=` parameter)
+When a target column is specified, `tput` automatically detects classification vs regression and runs:
+
+**Classification:**
+- Class balance and imbalance detection (minority class < 20% → warning)
+- Feature correlation ranking (Cramér's V for categorical, point-biserial for numeric)
+- Leakage detection (correlation > 0.95 with target)
+
+**Regression:**
+- Target skewness + recommended transform
+- Outliers in the target (direct impact on loss function)
+- Pearson correlation ranking for all numeric features
+- Leakage detection
+
+---
+
+## Display modes
+
+### `report.show()`
+Column-by-column view — every column gets a block with all its properties stacked:
+
+```
+--- COLUMN: Age  [float64] ---
+  nulls          : 177 (19.87%)
+  mean / median  : 29.70 / 28.0  |  std: 14.53
+  skewness       : 0.389 (symmetric)
+  outliers       : 11 (1.54%)  [method=IQR  bounds=-6.69, 64.81]
+  nan_analysis   : MAR -> impute (knn_or_regression)
+    correlated_with: Pclass (r=0.173), Parch (r=-0.124)
+  target_corr    : point_biserial_r=0.077 with 'Survived'
+```
+
+### `report.summary()`
+Condensed grouped view — designed for datasets with 50+ columns where `show()` would be overwhelming:
+
+```
+=== TPUT SUMMARY ===
+Shape : 1460 rows x 81 columns
+
+ISSUES DETECTED:
+  missing values   : 19 columns affected (6 proposed drop, 13 impute, 11 MAR)
+  skewness         : 34 high, 12 moderate
+  outliers         : 28 columns, 1842 values total
+  feature quality  : 9 quasi-constant, 18 low_cardinality, 1 potential_id
+  vif (redundant)  : 8 columns
+  correlations     : 1 numeric pairs, 182 categorical pairs (showing top 10 in warnings)
+
+Total warnings : 156  (use report.show() for full detail)
+```
+
+---
+
+## Parameters
+
+```python
+quick_report(
+    df,
+
+    # Steps — toggle any on/off
+    overview=True,
+    visible_missing=True,
+    duplicates=True,
+    categorical_profile=True,
+    type_issues=True,
+    numeric_profile=True,
+    skewness=True,
+    outliers=True,
+    nan_analysis=True,
+    correlations=True,
+    feature_quality=True,
+    row_analysis=True,
+    vif=True,
+
+    # Target column — enables target_analysis
+    target=None,                      # e.g. target="SalePrice" or target="Survived"
+
+    # Thresholds
+    nan_drop_threshold=0.30,          # drop column if missing rate > 30%
+    correlation_threshold=0.85,       # Pearson |r| threshold for numeric pairs
+    cramers_v_threshold=0.25,         # Cramér's V threshold for categorical pairs
+    max_correlation_warnings=10,      # cap categorical warnings, show top N by V
+    row_drop_threshold=0.50,          # flag rows with >= 50% NaN
+    apply_row_filter=True,            # run outliers/nan/correlations on filtered df
+    quasi_constant_threshold=0.95,    # flag column if dominant value > 95%
+    low_cardinality_max_unique=10,    # numeric columns with <= N values → categorical
+    vif_threshold=10.0,               # VIF threshold for multicollinearity warning
+
+    # Display
+    feature_display=True,             # True = column-by-column view, False = step-by-step
+)
+```
+
+---
+
+## Programmatic access
+
+```python
+# Access any step result directly
+report.get("outliers")
+report.get("nan_analysis")
+report.get("skewness")
+report.get("correlations")       # includes high_pairs and categorical_associations
+report.get("target_analysis")    # classification or regression breakdown
+
+# All warnings as a list
+report.warnings
+
+# Drop rows flagged by row_analysis
+drop_idx = report.get("row_analysis")["rows_to_drop_idx"]
+df_clean = df.drop(drop_idx)
+
+# Full categorical association list (bypasses max_correlation_warnings cap)
+all_cat_pairs = report.get("correlations")["categorical_associations"]
+
+# Feature ranking by correlation with target
+feature_corr = report.get("target_analysis")["feature_correlations"]
+```
+
+---
+
+## Design decisions
+
+**IQR vs MAD** — outlier method is chosen per column based on skewness. IQR is appropriate for symmetric distributions; MAD (Median Absolute Deviation) is more robust on skewed data where extreme values distort the mean and the IQR bounds.
+
+**Bias-corrected Cramér's V** — categorical associations use the Bergsma (2013) bias correction, which removes the positive bias of the standard formula on small samples and low-cardinality columns. High-cardinality columns (>50% unique values) are excluded from Chi² computation as they produce artificially inflated V values.
+
+**VIF via sklearn** — implemented without `statsmodels` using `LinearRegression` from sklearn. Only continuous columns (nunique > 10) are included. The target column is automatically excluded when `target=` is specified.
+
+**Row filter** — `row_analysis` identifies sparse rows, then `outliers`, `nan_analysis`, `correlations`, and `vif` all run on the filtered DataFrame. This prevents sparse rows from distorting column-level statistics. The raw DataFrame is always used for the upstream steps (`overview`, `visible_missing`, `duplicates`, `categorical_profile`, `type_issues`, `numeric_profile`, `skewness`, `row_analysis` itself).
+
+**Semantic NaN detection** — columns with >70% missing rate that are proposed for dropping receive an additional note: the high missing rate may encode absence of a feature (e.g. `PoolQC` missing = no pool) rather than truly missing data. Replacing NaN with a `"None"` category is often preferable to dropping.
+
+**Target exclusion** — when `target=` is specified, the target column is automatically excluded from inter-feature correlation computation (both Pearson and Cramér's V). Feature-target relationships are computed separately in `target_analysis`.
+
+---
+
+## Known limitations
+
+- **No domain knowledge.** A latitude of -78° or a weight of 1100 kg may be perfectly valid. All flagging is statistical.
+- **Skewness on small samples.** Columns with <30 non-null values produce unreliable skewness estimates.
+- **Outlier detection assumes unimodal distributions.** IQR and MAD both fail on bimodal distributions (e.g. a mixed-species weight column).
+- **MNAR is a hypothesis.** The library flags it when missingness is high but unexplained — proving MNAR requires domain knowledge.
+- **VIF assumes no perfect multicollinearity.** If the matrix is singular (e.g. `BsmtFinSF1 + BsmtFinSF2 + BsmtUnfSF = TotalBsmtSF` exactly), VIF is capped at 10000 to avoid division-by-zero. This is a signal, not a bug.

tput-0.1.0/README.md

@@ -0,0 +1,224 @@
+# tput — Automated DataFrame Audit for ML
+
+`tput` is a Python library for fast, opinionated data auditing before ML pipelines. Pass it a DataFrame and get a structured report that surfaces everything that matters before you start modelling: missing values, outliers, skewness, type issues, categorical noise, correlations, multicollinearity, and more.
+
+```python
+from tput import quick_report
+
+report = quick_report(df, target="SalePrice")
+report.show()       # full column-by-column report
+report.summary()    # condensed overview for large datasets
+```
+
+---
+
+## Installation
+
+```bash
+# Not yet on PyPI — clone and import locally
+git clone https://github.com/yourname/tput
+cd tput
+pip install -e .
+```
+
+Dependencies: `pandas>=1.5`, `scipy>=1.10`, `scikit-learn>=1.0`
+
+---
+
+## Quickstart
+
+```python
+import pandas as pd
+from tput import quick_report
+
+df = pd.read_csv("titanic.csv")
+
+# Basic audit
+report = quick_report(df)
+report.show()
+
+# With target column — unlocks classification/regression analysis
+report = quick_report(df, target="Survived")
+report.show()
+
+# Condensed view — useful for wide datasets (50+ columns)
+report.summary()
+
+# Programmatic access
+report.get("nan_analysis")
+report.get("target_analysis")
+report.warnings
+```
+
+---
+
+## What it detects
+
+### Data quality
+| Step | What it catches |
+|---|---|
+| `overview` | Shape, dtypes |
+| `visible_missing` | NaN counts and percentages per column |
+| `duplicates` | Fully duplicated rows |
+| `categorical_profile` | Cardinality, mode, top/bottom values, hidden missing (`""`, `"   "`), case collisions (`"Paris"` vs `"paris"`) |
+| `type_issues` | String columns that should be numeric or datetime |
+| `row_analysis` | Rows with too many missing fields (default: ≥50%) — flagged for removal |
+
+### Statistical analysis
+| Step | What it catches |
+|---|---|
+| `numeric_profile` | mean, std, Q1/Q3, min/max, mode |
+| `skewness` | Asymmetry level (symmetric / moderate / high) + recommended transform (log1p, sqrt, reflect + log1p, yeo-johnson) |
+| `outliers` | IQR on symmetric columns, MAD on skewed ones — auto-selected per column |
+| `nan_analysis` | MCAR / MAR / MNAR classification + imputation strategy (mean, median, mode, knn_or_regression). High missing rate (>70%) flagged as potential semantic NaN (e.g. "no pool", "no garage") |
+| `feature_quality` | quasi-constant columns, low-cardinality numerics, suspected ID columns |
+
+### Multicollinearity & associations
+| Step | What it catches |
+|---|---|
+| `correlations` | Pearson pairs above threshold (default: 0.85) + Chi² / Cramér's V (bias-corrected) for categorical pairs |
+| `vif` | Variance Inflation Factor per continuous column — flags multicollinearity for linear models |
+
+### Target analysis (`target=` parameter)
+When a target column is specified, `tput` automatically detects classification vs regression and runs:
+
+**Classification:**
+- Class balance and imbalance detection (minority class < 20% → warning)
+- Feature correlation ranking (Cramér's V for categorical, point-biserial for numeric)
+- Leakage detection (correlation > 0.95 with target)
+
+**Regression:**
+- Target skewness + recommended transform
+- Outliers in the target (direct impact on loss function)
+- Pearson correlation ranking for all numeric features
+- Leakage detection
+
+---
+
+## Display modes
+
+### `report.show()`
+Column-by-column view — every column gets a block with all its properties stacked:
+
+```
+--- COLUMN: Age  [float64] ---
+  nulls          : 177 (19.87%)
+  mean / median  : 29.70 / 28.0  |  std: 14.53
+  skewness       : 0.389 (symmetric)
+  outliers       : 11 (1.54%)  [method=IQR  bounds=-6.69, 64.81]
+  nan_analysis   : MAR -> impute (knn_or_regression)
+    correlated_with: Pclass (r=0.173), Parch (r=-0.124)
+  target_corr    : point_biserial_r=0.077 with 'Survived'
+```
+
+### `report.summary()`
+Condensed grouped view — designed for datasets with 50+ columns where `show()` would be overwhelming:
+
+```
+=== TPUT SUMMARY ===
+Shape : 1460 rows x 81 columns
+
+ISSUES DETECTED:
+  missing values   : 19 columns affected (6 proposed drop, 13 impute, 11 MAR)
+  skewness         : 34 high, 12 moderate
+  outliers         : 28 columns, 1842 values total
+  feature quality  : 9 quasi-constant, 18 low_cardinality, 1 potential_id
+  vif (redundant)  : 8 columns
+  correlations     : 1 numeric pairs, 182 categorical pairs (showing top 10 in warnings)
+
+Total warnings : 156  (use report.show() for full detail)
+```
+
+---
+
+## Parameters
+
+```python
+quick_report(
+    df,
+
+    # Steps — toggle any on/off
+    overview=True,
+    visible_missing=True,
+    duplicates=True,
+    categorical_profile=True,
+    type_issues=True,
+    numeric_profile=True,
+    skewness=True,
+    outliers=True,
+    nan_analysis=True,
+    correlations=True,
+    feature_quality=True,
+    row_analysis=True,
+    vif=True,
+
+    # Target column — enables target_analysis
+    target=None,                      # e.g. target="SalePrice" or target="Survived"
+
+    # Thresholds
+    nan_drop_threshold=0.30,          # drop column if missing rate > 30%
+    correlation_threshold=0.85,       # Pearson |r| threshold for numeric pairs
+    cramers_v_threshold=0.25,         # Cramér's V threshold for categorical pairs
+    max_correlation_warnings=10,      # cap categorical warnings, show top N by V
+    row_drop_threshold=0.50,          # flag rows with >= 50% NaN
+    apply_row_filter=True,            # run outliers/nan/correlations on filtered df
+    quasi_constant_threshold=0.95,    # flag column if dominant value > 95%
+    low_cardinality_max_unique=10,    # numeric columns with <= N values → categorical
+    vif_threshold=10.0,               # VIF threshold for multicollinearity warning
+
+    # Display
+    feature_display=True,             # True = column-by-column view, False = step-by-step
+)
+```
+
+---
+
+## Programmatic access
+
+```python
+# Access any step result directly
+report.get("outliers")
+report.get("nan_analysis")
+report.get("skewness")
+report.get("correlations")       # includes high_pairs and categorical_associations
+report.get("target_analysis")    # classification or regression breakdown
+
+# All warnings as a list
+report.warnings
+
+# Drop rows flagged by row_analysis
+drop_idx = report.get("row_analysis")["rows_to_drop_idx"]
+df_clean = df.drop(drop_idx)
+
+# Full categorical association list (bypasses max_correlation_warnings cap)
+all_cat_pairs = report.get("correlations")["categorical_associations"]
+
+# Feature ranking by correlation with target
+feature_corr = report.get("target_analysis")["feature_correlations"]
+```
+
+---
+
+## Design decisions
+
+**IQR vs MAD** — outlier method is chosen per column based on skewness. IQR is appropriate for symmetric distributions; MAD (Median Absolute Deviation) is more robust on skewed data where extreme values distort the mean and the IQR bounds.
+
+**Bias-corrected Cramér's V** — categorical associations use the Bergsma (2013) bias correction, which removes the positive bias of the standard formula on small samples and low-cardinality columns. High-cardinality columns (>50% unique values) are excluded from Chi² computation as they produce artificially inflated V values.
+
+**VIF via sklearn** — implemented without `statsmodels` using `LinearRegression` from sklearn. Only continuous columns (nunique > 10) are included. The target column is automatically excluded when `target=` is specified.
+
+**Row filter** — `row_analysis` identifies sparse rows, then `outliers`, `nan_analysis`, `correlations`, and `vif` all run on the filtered DataFrame. This prevents sparse rows from distorting column-level statistics. The raw DataFrame is always used for the upstream steps (`overview`, `visible_missing`, `duplicates`, `categorical_profile`, `type_issues`, `numeric_profile`, `skewness`, `row_analysis` itself).
+
+**Semantic NaN detection** — columns with >70% missing rate that are proposed for dropping receive an additional note: the high missing rate may encode absence of a feature (e.g. `PoolQC` missing = no pool) rather than truly missing data. Replacing NaN with a `"None"` category is often preferable to dropping.
+
+**Target exclusion** — when `target=` is specified, the target column is automatically excluded from inter-feature correlation computation (both Pearson and Cramér's V). Feature-target relationships are computed separately in `target_analysis`.
+
+---
+
+## Known limitations
+
+- **No domain knowledge.** A latitude of -78° or a weight of 1100 kg may be perfectly valid. All flagging is statistical.
+- **Skewness on small samples.** Columns with <30 non-null values produce unreliable skewness estimates.
+- **Outlier detection assumes unimodal distributions.** IQR and MAD both fail on bimodal distributions (e.g. a mixed-species weight column).
+- **MNAR is a hypothesis.** The library flags it when missingness is high but unexplained — proving MNAR requires domain knowledge.
+- **VIF assumes no perfect multicollinearity.** If the matrix is singular (e.g. `BsmtFinSF1 + BsmtFinSF2 + BsmtUnfSF = TotalBsmtSF` exactly), VIF is capped at 10000 to avoid division-by-zero. This is a signal, not a bug.

tput-0.1.0/pyproject.toml

@@ -0,0 +1,27 @@
+[build-system]
+requires = ["setuptools>=68", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "tput"
+version = "0.1.0"
+description = "Automated DataFrame audit library for ML pipelines"
+readme = "README.md"
+license = { text = "MIT" }
+requires-python = ">=3.9"
+
+dependencies = [
+    "pandas>=1.5",
+    "scipy>=1.10",
+    "scikit-learn>=1.0",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=7.0",
+    "pytest-cov",
+]
+
+[tool.setuptools.packages.find]
+where = ["."]
+include = ["tput*"]

tput-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

tput-0.1.0/tput/__init__.py

@@ -0,0 +1 @@
+from .core import quick_report