cleanflow-kit 1.2.1__tar.gz → 1.2.3__tar.gz

cleanflow_kit-1.2.3/PKG-INFO

@@ -0,0 +1,622 @@
+Metadata-Version: 2.4
+Name: cleanflow-kit
+Version: 1.2.3
+Summary: A comprehensive, modular Python tool for automated data cleaning, EDA, and feature engineering.
+Author: CleanFlow Contributors
+License: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pandas>=2.0.0
+Requires-Dist: numpy>=1.20.0
+Requires-Dist: scikit-learn>=1.0.0
+Requires-Dist: matplotlib>=3.5.0
+Requires-Dist: joblib>=1.1.0
+Provides-Extra: smote
+Requires-Dist: imbalanced-learn>=0.9.0; extra == "smote"
+Provides-Extra: drift
+Requires-Dist: scipy>=1.7.0; extra == "drift"
+Provides-Extra: all
+Requires-Dist: imbalanced-learn>=0.9.0; extra == "all"
+Requires-Dist: scipy>=1.7.0; extra == "all"
+Provides-Extra: web
+Requires-Dist: Flask; extra == "web"
+Requires-Dist: Flask-SQLAlchemy; extra == "web"
+Requires-Dist: PyJWT; extra == "web"
+Requires-Dist: gunicorn; extra == "web"
+Requires-Dist: openpyxl; extra == "web"
+Provides-Extra: api
+Requires-Dist: FastAPI; extra == "api"
+Requires-Dist: uvicorn; extra == "api"
+Requires-Dist: pydantic; extra == "api"
+Dynamic: license-file
+
+# CleanFlow (`cleanflow-kit`)
+
+A comprehensive, intelligent Python package for automated **data cleaning**, **exploratory data analysis (EDA)**, **feature engineering**, **AutoML model training**, **data drift detection**, **synthetic data generation**, and **HTML/Markdown report generation**.
+
+CleanFlow eliminates the repetitive 80% of data science work — replacing hundreds of lines of boilerplate Pandas/Scikit-learn code with a clean, chainable API.
+
+---
+
+## 📦 Installation
+
+```bash
+pip install cleanflow-kit
+```
+
+Install with optional extras:
+
+```bash
+# SMOTE for class imbalance handling
+pip install "cleanflow-kit[smote]"
+
+# Drift detection (requires scipy)
+pip install "cleanflow-kit[drift]"
+
+# Everything
+pip install "cleanflow-kit[all]"
+
+# Web application (Flask UI)
+pip install "cleanflow-kit[web,api]"
+```
+
+**Requirements:** Python 3.8+, pandas ≥ 2.0, numpy ≥ 1.20, scikit-learn ≥ 1.0, matplotlib ≥ 3.5
+
+---
+
+## 🚀 Quick Start (3 Lines)
+
+```python
+from cleanflow import DataPipeline
+
+pipeline = DataPipeline()
+pipeline.load("your_data.csv")
+
+cleaned_df, final_df = pipeline.run_full_pipeline(
+    target_col="price",
+    problem_type="regression"   # or "classification" — auto-detected if omitted
+)
+
+pipeline.save_data("./output")
+```
+
+That's it. Your data is cleaned, analyzed, feature-engineered, and saved.
+
+---
+
+## 📖 Complete Usage Guide
+
+### 1. One-Click Full Pipeline
+
+The simplest way to use CleanFlow. Runs validation → cleaning → EDA → feature engineering automatically.
+
+```python
+from cleanflow import DataPipeline
+
+pipeline = DataPipeline()
+pipeline.load("housing_prices.csv")
+
+# Run everything
+cleaned_df, final_df = pipeline.run_full_pipeline(
+    target_col="price",
+    problem_type="regression",
+    show_eda_plots=True,
+    cleaning_config={                        # Optional overrides
+        "missing_numeric_strategy": "mean",
+        "outlier_method": "zscore"
+    },
+    feature_config={                         # Optional overrides
+        "scale_method": "minmax",
+        "create_polynomial_features": True,
+        "auto_evolve_features": True
+    }
+)
+
+# Access reports
+print(pipeline.pipeline_report['validation'])
+print(pipeline.pipeline_report['cleaning'])
+print(pipeline.pipeline_report['eda'])
+print(pipeline.pipeline_report['feature_engineering'])
+
+# Save outputs
+pipeline.save_data("./output", format="csv")       # or "parquet"
+```
+
+---
+
+### 2. Step-by-Step Pipeline (Full Control)
+
+Call each stage individually for granular control over every parameter.
+
+```python
+from cleanflow import DataPipeline
+
+pipeline = DataPipeline()
+pipeline.load("customer_churn.csv")
+
+# ── Step 1: Validate ──
+report = pipeline.validate()
+# Returns: shape, dtypes, missing values, duplicates, constant columns, potential ID columns
+
+# ── Step 2: Clean ──
+pipeline.clean(
+    remove_duplicates=True,
+    handle_missing=True,
+    missing_numeric_strategy='median',       # 'mean', 'median', 'zero'
+    missing_categorical_strategy='mode',     # 'mode', 'unknown'
+    missing_drop_threshold=0.4,              # Drop columns with >40% missing
+    fix_types=True,                          # Auto-infer numeric, datetime, category types
+    handle_outliers=True,
+    outlier_method='iqr',                    # 'iqr' or 'zscore'
+    outlier_action='clip',                   # 'clip', 'remove', or 'nan'
+    clean_categorical=True,                  # Strip whitespace, lowercase
+    drop_constant=True                       # Remove constant/single-value columns
+)
+
+# ── Step 3: Exploratory Data Analysis ──
+eda_results = pipeline.analyze(
+    target_col="churn",
+    show_plots=True,
+    save_plots=True,
+    output_dir="./eda_plots"
+)
+# Generates: distributions.png, boxplots.png, correlation.png, categorical.png, target_relationships.png
+
+# ── Step 4: Feature Engineering ──
+pipeline.engineer_features(
+    target_col="churn",
+    problem_type="classification",
+    encode_categorical=True,                 # Auto: OneHot for low-cardinality, Label for high
+    scale_features=True,
+    scale_method='standard',                 # 'standard' or 'minmax'
+    create_datetime_features=True,           # Extracts year, month, day, dayofweek, is_weekend
+    create_polynomial_features=False,        # Generates degree-2 polynomial terms
+    polynomial_degree=2,
+    drop_low_variance=True,                  # Removes near-zero variance features
+    drop_high_correlation=True,              # Removes one of each highly correlated pair
+    correlation_threshold=0.95,
+    handle_imbalance=True,                   # Applies SMOTE for imbalanced classes
+    imbalance_method='smote',                # 'smote', 'oversample', 'undersample'
+    auto_evolve_features=False               # Auto-generates & selects best interaction features
+)
+
+# ── Step 5: Train Models (AutoML) ──
+model_results = pipeline.train_model(
+    target_col="churn",
+    problem_type="classification",
+    export_path="./best_model.pkl"           # Optional: export trained model
+)
+
+print("Best Model:", model_results['best_model']['name'])
+print("Metrics:", model_results['best_model']['metrics'])
+print("Reliability:", model_results['reliability'])
+print("Feature Importance:", model_results['feature_importance'])
+
+# ── Step 6: Generate Reports ──
+pipeline.generate_html_report("./report.html")
+pipeline.generate_markdown_report("./report.md")
+
+# ── Access Data at Any Stage ──
+raw_df     = pipeline.get_raw_data()
+cleaned_df = pipeline.get_cleaned_data()
+final_df   = pipeline.get_final_data()
+full_report = pipeline.get_report()
+```
+
+---
+
+### 3. Using Individual Modules Directly
+
+For maximum flexibility, use the standalone classes without the pipeline orchestrator.
+
+```python
+from cleanflow import DataLoader, CleanFlower, EDAAnalyzer, FeatureEngineer, ModelTrainer
+```
+
+#### DataLoader
+
+```python
+loader = DataLoader()
+
+# Load from file (CSV, Excel, JSON, Parquet)
+df = loader.load("data.csv")
+df = loader.load("data.xlsx")
+df = loader.load("data.json")
+df = loader.load("data.parquet")
+
+# Load from existing DataFrame
+df = loader.load(my_dataframe)
+
+# Validate and print profile
+report = loader.validate()
+loader.print_summary()
+# Reports: shape, memory usage, column types, missing values, duplicates,
+#          constant columns, near-constant columns, potential ID columns
+```
+
+#### CleanFlower (Data Cleaner)
+
+```python
+cleaner = CleanFlower(df)
+
+# Chain operations fluently
+cleaner.remove_duplicates()
+
+cleaner.handle_missing_values(
+    numeric_strategy='median',               # 'mean', 'median', 'zero'
+    categorical_strategy='mode',             # 'mode', 'unknown'
+    drop_threshold=0.4                       # Drop cols with >40% missing
+)
+
+cleaner.fix_data_types()                     # Auto-infer numeric, datetime, category
+
+cleaner.handle_outliers(
+    method='iqr',                            # 'iqr' or 'zscore'
+    threshold=1.5,                           # IQR multiplier or Z-score threshold
+    action='clip'                            # 'clip', 'remove', 'nan'
+)
+
+cleaner.clean_categorical_values(
+    lowercase=True,
+    strip_whitespace=True,
+    replace_mapping={"city": {"nyc": "new york", "sf": "san francisco"}}
+)
+
+cleaner.clean_numeric_text()                 # Converts "$1,200" → 1200, "1.5k" → 1500
+
+cleaner.normalize_features(method='standard') # 'standard', 'minmax', 'robust'
+
+cleaner.encode_categorical(method='onehot')  # 'onehot' or 'label'
+
+cleaner.rename_columns({"old_name": "new_name"})
+
+cleaner.extract_regex_feature(
+    source_col="description",
+    pattern=r'ID: (\d+)',
+    new_col_name="extracted_id"
+)
+
+cleaner.drop_columns(
+    columns=["unwanted_col"],
+    drop_constant=True,                      # Remove single-value columns
+    drop_id_like=True                        # Remove columns where every value is unique
+)
+
+# Data quality scoring (0–100)
+quality = cleaner.validate_quality()
+# Returns: score, grade (A/B/C/D/F), missing_values, duplicate_rows, constant_columns, warnings
+
+# AI-like cleaning suggestions
+suggestions = cleaner.generate_suggestions()
+# Returns list of: {column, issue, action, reason} for every detected problem
+
+# Get results
+cleaned_df = cleaner.get_cleaned_data()
+summary = cleaner.get_cleaning_summary()
+# Summary includes: original_shape, final_shape, operations log, row-level change audit trail
+cleaner.print_summary()
+```
+
+#### EDAAnalyzer
+
+```python
+eda = EDAAnalyzer(df, target_col="price")
+
+# Individual analyses
+stats = eda.summary_statistics()             # Descriptive stats + skewness + kurtosis
+cat_summary = eda.categorical_summary()      # Value counts & percentages per categorical col
+corr_matrix = eda.correlation_analysis(
+    method='pearson',                        # 'pearson', 'kendall', 'spearman'
+    threshold=0.7                            # Flag pairs above this correlation
+)
+distributions = eda.distribution_analysis()  # Mean, median, std, skewness, kurtosis, IQR, range
+target_info = eda.target_analysis()          # Stats for continuous, class distribution for categorical
+
+# Generate visualizations
+eda.plot_distributions(save_path="dist.png")
+eda.plot_boxplots(save_path="box.png")
+eda.plot_correlation_heatmap(save_path="corr.png")
+eda.plot_categorical(save_path="cat.png")
+eda.plot_target_relationships(save_path="target.png")
+
+# Or run everything at once
+results = eda.run_full_analysis(show_plots=True, save_plots=True, output_dir="./plots")
+
+# Get auto-generated insights
+insights = eda.get_insights()
+# Examples: "Highly skewed features detected: [col1, col2]"
+#           "High cardinality in 'city': 342 unique values"
+#           "Class imbalance detected in target: ratio = 0.12"
+```
+
+#### FeatureEngineer
+
+```python
+engineer = FeatureEngineer(df, target_col="churn", problem_type="classification")
+
+# Encoding
+engineer.encode_categorical(
+    method='auto',                           # 'auto', 'onehot', 'label'
+    max_categories=10,                       # OneHot threshold; Label above this
+    drop_first=True
+)
+
+# Scaling
+engineer.scale_features(method='standard')   # 'standard' or 'minmax'
+
+# Datetime extraction
+engineer.create_datetime_features(
+    features=['year', 'month', 'day', 'dayofweek', 'quarter', 'hour', 'is_weekend']
+)
+
+# Polynomial & interaction features
+engineer.create_polynomial_features(degree=2, interaction_only=False)
+
+# Binned features
+engineer.create_binned_features(n_bins=5, strategy='quantile')  # 'quantile' or 'uniform'
+
+# Feature selection
+engineer.compute_feature_importance(n_features=20)  # Uses mutual information
+engineer.drop_low_importance_features(threshold=0.01)
+engineer.drop_low_variance_features(threshold=0.01)
+engineer.drop_highly_correlated(threshold=0.95)
+
+# Class imbalance handling
+engineer.handle_class_imbalance(
+    method='smote',                          # 'smote', 'oversample', 'undersample'
+    sampling_strategy='auto'
+)
+
+# Auto-evolve: automatically generates interaction features, evaluates them
+# with mutual information, and keeps only the best ones
+engineer.auto_evolve(max_new_features=10)
+
+# Get results
+final_df = engineer.get_transformed_data()
+summary = engineer.get_summary()
+engineer.print_summary()
+```
+
+#### ModelTrainer
+
+```python
+trainer = ModelTrainer(df, target_col="price", problem_type="regression")
+
+# Full pipeline
+results = trainer.run()
+# Internally runs: detect_target → detect_problem_type → prepare_features →
+#                  train_all_models → get_best_model → get_metrics_dashboard
+
+# Or step by step:
+trainer.detect_target()                      # Auto-detects target using heuristics
+trainer.detect_problem_type()                # Classification vs regression by cardinality
+trainer.prepare_features()                   # Encode, scale, impute, drop non-numeric
+trainer.train_all_models()                   # CV + RandomizedSearchCV for all models
+best = trainer.get_best_model()              # Select winner by F1 (clf) or R² (reg)
+importance = trainer.get_feature_importance(top_n=10)
+reliability = trainer.get_reliability_score() # 0–100 score + grade (A/B/C/D)
+dashboard = trainer.get_metrics_dashboard()  # Complete results dict
+
+# Export model as .pkl
+trainer.export_model("./model.pkl")
+# Saved: model, scaler, label_encoder, feature_names, metrics, reliability
+
+# Models trained:
+# Classification: Logistic Regression, Random Forest (with class_weight='balanced' if imbalanced)
+# Regression:     Ridge Regression, Random Forest
+# All with hyperparameter tuning via RandomizedSearchCV and 3-fold Stratified/KFold CV
+```
+
+---
+
+### 4. Data Drift Detection
+
+Compare a baseline dataset against new production data to detect schema and statistical drift.
+
+```python
+from cleanflow.drift_detector import DriftDetector
+
+detector = DriftDetector(baseline_df=train_df, current_df=production_df)
+report = detector.run()
+
+print(report['schema_drift'])         # Missing or new columns
+print(report['statistical_drift'])    # Per-column KS-test p-values
+print(report['drifted_features'])     # List of columns with drift (p < 0.05)
+print(report['drift_detected'])       # True/False
+print(report['score'])                # Health score 0–100
+```
+
+Uses the **Kolmogorov-Smirnov test** on shared numeric columns. Penalizes schema changes and statistical deviations to produce a single health score.
+
+---
+
+### 5. Synthetic Data Generation
+
+Generate privacy-safe statistical clones of your dataset using Kernel Density Estimation (KDE).
+
+```python
+from cleanflow.synthetic_generator import SyntheticGenerator
+
+gen = SyntheticGenerator(df)
+gen.fit()                                    # Learn distributions
+synthetic_df = gen.generate(n_rows=1000)     # Generate 1000 synthetic rows
+
+# Numeric columns: sampled from KDE (Gaussian kernel)
+# Categorical columns: sampled from observed probability distribution
+# Constant columns: replicated as-is
+```
+
+---
+
+### 6. Working with DataFrames Directly
+
+No file needed — pass a DataFrame directly.
+
+```python
+import pandas as pd
+from cleanflow import DataPipeline
+
+df = pd.DataFrame({
+    'feature1': [1.0, 2.5, None, 4.0, 100.0],
+    'category': ['A', 'B', 'A', None, 'B'],
+    'target':   [0, 1, 0, 1, 0]
+})
+
+pipeline = DataPipeline()
+pipeline.load(df)
+cleaned_df, final_df = pipeline.run_full_pipeline(
+    target_col='target',
+    problem_type='classification'
+)
+```
+
+---
+
+### 7. Flask Web Application
+
+An optional browser-based UI with drag-and-drop uploads, user authentication, and an iOS 26-inspired glassmorphism design.
+
+```bash
+pip install "cleanflow-kit[web,api]"
+python app.py
+```
+
+---
+
+## 📋 Complete Feature Reference
+
+### Data Loading & Validation
+| Feature | Details |
+|---------|---------|
+| Multi-format support | CSV, Excel (.xlsx/.xls), JSON, Parquet, DataFrame |
+| Smart profiling | Auto-detects dtypes, missing values, duplicates |
+| Noise detection | Flags constant columns, near-constant columns, potential ID columns |
+| Memory reporting | Reports dataset memory usage in MB |
+
+### Data Cleaning (CleanFlower)
+| Feature | Details |
+|---------|---------|
+| Duplicate removal | Exact row deduplication |
+| Missing value imputation | mean / median / zero (numeric), mode / "Unknown" (categorical) |
+| Threshold-based column dropping | Drops columns exceeding a configurable missing % (default 40%) |
+| Auto type inference | Converts object → numeric, object → datetime, low-cardinality → category |
+| Outlier handling | IQR or Z-score detection with clip / remove / NaN actions |
+| Categorical normalization | Strip whitespace, lowercase, custom value mapping |
+| Numeric text parsing | Converts "$1,200" → 1200, "1.5k" → 1500, "2M" → 2000000 |
+| Regex feature extraction | Extract structured data from text using capture groups |
+| Column renaming | Batch rename via dictionary mapping |
+| Column dropping | Drop by name, drop constants, drop ID-like columns |
+| Feature scaling | Standard (Z-score), MinMax, Robust scalers |
+| Categorical encoding | One-Hot or Label encoding |
+| Quality scoring | 0–100 data quality score (A/B/C/D/F grade) based on completeness, uniqueness, consistency |
+| Cleaning suggestions | Auto-generated per-column recommendations with issue, action, and reason |
+| Row-level audit trail | Logs every individual cell change (old value → new value, operation, reason) |
+
+### Exploratory Data Analysis (EDA)
+| Feature | Details |
+|---------|---------|
+| Summary statistics | Count, mean, std, min/max, quartiles, skewness, kurtosis |
+| Categorical profiling | Value counts and percentages per category |
+| Correlation analysis | Pearson/Kendall/Spearman with high-correlation pair detection |
+| Distribution analysis | Mean, median, std, skewness, kurtosis, IQR, range per feature |
+| Target analysis | Continuous stats or class distribution + imbalance detection |
+| Auto-generated insights | Flags skewed features, high-cardinality columns, class imbalance |
+| Visualizations | Histograms, Boxplots, Correlation Heatmaps, Category Bar Charts, Feature-Target Scatters |
+
+### Feature Engineering
+| Feature | Details |
+|---------|---------|
+| Smart encoding | Auto-selects OneHot (≤ threshold) or Label (above threshold) per column |
+| Feature scaling | StandardScaler or MinMaxScaler on numeric features |
+| Datetime extraction | year, month, day, dayofweek, quarter, hour, minute, is_weekend |
+| Polynomial features | Degree-N polynomial and/or interaction-only terms |
+| Binned features | Quantile or uniform binning of numeric columns |
+| Mutual information importance | Ranks features by mutual information with the target |
+| Low-importance dropping | Removes features below an importance threshold |
+| Low-variance dropping | Removes near-zero variance features via VarianceThreshold |
+| High-correlation dropping | Removes one of each pair exceeding a correlation threshold |
+| Class imbalance handling | SMOTE, random oversampling, or random undersampling |
+| Auto-evolve | Automatically generates interaction features, evaluates with MI, keeps the best |
+
+### AutoML Model Training
+| Feature | Details |
+|---------|---------|
+| Auto target detection | Heuristic scan for columns named "target", "label", "price", "churn", etc. |
+| Auto problem type detection | Classification (categorical/few unique) vs Regression (continuous) |
+| Classification models | Logistic Regression, Random Forest (with balanced class weights if needed) |
+| Regression models | Ridge Regression, Random Forest |
+| Hyperparameter tuning | RandomizedSearchCV with configurable iteration budget |
+| Cross-validation | 3-fold StratifiedKFold (classification) or KFold (regression) |
+| Multi-metric evaluation | Accuracy, Precision, Recall, F1, ROC-AUC (clf) / R², MAE, RMSE (reg) |
+| Imbalance detection | Flags majority class > 70% and auto-applies SMOTE if available |
+| Reliability score | Custom 0–100 score based on dataset size, CV variance, metric quality (grade A/B/C/D) |
+| Feature importance | Extracts from tree `feature_importances_` or linear `coef_` |
+| Baseline comparison | Trains a naive Decision Tree on raw data to quantify cleaning improvement |
+| Model export | Saves model + scaler + encoder + feature names + metrics as `.pkl` |
+
+### Drift Detection
+| Feature | Details |
+|---------|---------|
+| Schema drift | Detects missing or new columns between baseline and current data |
+| Statistical drift | Kolmogorov-Smirnov test on shared numeric columns (p < 0.05 = drift) |
+| Health score | 0–100 composite score penalizing schema and statistical drift |
+
+### Synthetic Data Generation
+| Feature | Details |
+|---------|---------|
+| Numeric columns | Kernel Density Estimation (Gaussian) with automatic bandwidth selection |
+| Categorical columns | Probabilistic sampling from observed frequency distributions |
+| Integer preservation | Rounds generated values for originally integer-typed columns |
+
+### Report Generation
+| Feature | Details |
+|---------|---------|
+| HTML report | Publication-quality report with executive summary, methodology, data profiling, model architecture, performance evaluation, and future recommendations |
+| Markdown report | Lightweight report with data transformation summary, model performance table, and key observations |
+
+---
+
+## 🗂️ Package Structure
+
+```
+cleanflow/
+├── __init__.py              # Package exports (DataPipeline, DataLoader, CleanFlower, etc.)
+├── pipeline.py              # DataPipeline orchestrator
+├── data_loader.py           # Multi-format data loading & validation
+├── data_cleaner.py          # CleanFlower — 30+ cleaning operations
+├── eda.py                   # EDAAnalyzer — statistics & visualizations
+├── feature_engineer.py      # FeatureEngineer — encoding, scaling, selection
+├── model_trainer.py         # ModelTrainer — AutoML with CV, tuning, reliability
+├── drift_detector.py        # DriftDetector — schema & statistical drift
+├── synthetic_generator.py   # SyntheticGenerator — KDE-based data generation
+├── report_generator.py      # ReportGenerator — HTML/Markdown reports
+└── _compat.py               # String normalization compatibility layer
+```
+
+---
+
+## 🔌 API Quick Reference
+
+```python
+from cleanflow import DataPipeline
+
+pipeline = DataPipeline()
+pipeline.load(source)                        # str path or pd.DataFrame
+pipeline.validate()                          # → dict
+pipeline.clean(...)                          # → DataPipeline (chainable)
+pipeline.analyze(...)                        # → dict
+pipeline.engineer_features(...)              # → DataPipeline (chainable)
+pipeline.run_full_pipeline(...)              # → (cleaned_df, final_df)
+pipeline.train_model(...)                    # → dict (metrics dashboard)
+pipeline.generate_html_report(path)          # → str (file path)
+pipeline.generate_markdown_report(path)      # → str (file path)
+pipeline.save_data(dir, format='csv')        # saves cleaned_data + final_data
+pipeline.get_raw_data()                      # → pd.DataFrame
+pipeline.get_cleaned_data()                  # → pd.DataFrame
+pipeline.get_final_data()                    # → pd.DataFrame
+pipeline.get_report()                        # → dict (full pipeline report)
+```