datapilot-polars 0.3.0__tar.gz

datapilot_polars-0.3.0/PKG-INFO

@@ -0,0 +1,618 @@
+Metadata-Version: 2.4
+Name: datapilot-polars
+Version: 0.3.0
+Summary: Your Data Science Copilot — Polars-native EDA, auto-cleaning, drift detection, and multi-provider AI insights
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: polars>=1.0.0
+Requires-Dist: pyarrow>=14.0.0
+Requires-Dist: pandas>=2.0.0
+Requires-Dist: ollama>=0.2.0
+Requires-Dist: matplotlib>=3.5.0
+Requires-Dist: seaborn>=0.12.0
+Requires-Dist: scikit-learn>=1.0.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0; extra == "dev"
+Provides-Extra: openai
+Requires-Dist: openai>=1.0.0; extra == "openai"
+Provides-Extra: gemini
+Requires-Dist: google-generativeai>=0.7.0; extra == "gemini"
+Provides-Extra: claude
+Requires-Dist: anthropic>=0.25.0; extra == "claude"
+Provides-Extra: groq
+Requires-Dist: groq>=0.9.0; extra == "groq"
+Provides-Extra: all-ai
+Requires-Dist: openai>=1.0.0; extra == "all-ai"
+Requires-Dist: google-generativeai>=0.7.0; extra == "all-ai"
+Requires-Dist: anthropic>=0.25.0; extra == "all-ai"
+Requires-Dist: groq>=0.9.0; extra == "all-ai"
+
+# 📖 DataPilot — Your Data Science Copilot
+
+[![Python Version](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/)
+[![Version](https://img.shields.io/badge/version-0.3.0-brightgreen.svg)](https://github.com/yourusername/datapilot)
+[![Core Engine](https://img.shields.io/badge/Powered%20By-Polars%20%7C%20Apache%20Arrow-orange.svg)](https://pola.rs/)
+[![AI Providers](https://img.shields.io/badge/AI-Ollama%20%7C%20OpenAI%20%7C%20Gemini%20%7C%20Claude%20%7C%20Groq-purple.svg)](https://github.com/yourusername/datapilot)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Tests](https://img.shields.io/badge/tests-30%20passed-success.svg)](https://github.com/yourusername/datapilot)
+
+> **The only EDA library that is Polars-native, multi-provider AI-powered, and actually cleans your data.**
+
+DataPilot is an open-source Python library that automates Exploratory Data Analysis, detects data quality issues, cleans datasets, benchmarks performance, and generates intelligent AI recommendations — via Ollama (local), OpenAI, Google Gemini, Anthropic Claude, or Groq — all with minimal code.
+
+---
+
+## 🚀 Why DataPilot?
+
+| | ydata-profiling | sweetviz | dtale | **DataPilot** |
+|---|:-:|:-:|:-:|:-:|
+| Polars-native (10x faster) | ❌ | ❌ | ❌ | ✅ |
+| Local AI (Ollama, private) | ❌ | ❌ | ❌ | ✅ |
+| Cloud AI (OpenAI/Gemini/Claude/Groq) | ❌ | ❌ | ❌ | ✅ |
+| Auto data cleaning | ❌ | ❌ | ❌ | ✅ |
+| Train/test drift detection | ✅ | ✅ | ❌ | ✅ |
+| Outlier detection | ⚠️ | ❌ | ✅ | ✅ |
+| Smart column suggestions | ❌ | ❌ | ❌ | ✅ |
+| Offline HTML dashboard | ✅ | ✅ | ❌ | ✅ |
+| Regression diagnostics | ❌ | ❌ | ❌ | ✅ |
+| ML model diagnostics | ❌ | ❌ | ❌ | ✅ |
+
+---
+
+## 📑 Table of Contents
+1. [Installation](#installation)
+2. [Quick Start](#quick-start)
+3. [Module 1: Auto EDA Pipeline](#module-1-auto-eda-pipeline)
+4. [Module 2: Smart Column Suggestions](#module-2-smart-column-suggestions)
+5. [Module 3: Dataset Analysis API](#module-3-dataset-analysis-api)
+6. [Module 4: Outlier Detection](#module-4-outlier-detection)
+7. [Module 5: Auto Data Cleaning](#module-5-auto-data-cleaning)
+8. [Module 6: Train/Test Drift Detection](#module-6-traintest-drift-detection)
+9. [Module 7: Visualization Engine](#module-7-visualization-engine)
+10. [Module 8: Machine Learning Diagnostics](#module-8-machine-learning-diagnostics)
+11. [Module 9: Standalone HTML Dashboard](#module-9-standalone-html-dashboard)
+12. [Module 10: Performance Benchmark](#module-10-performance-benchmark)
+13. [Module 11: AI Copilot (5 Providers)](#module-11-ai-copilot--5-providers)
+14. [Installation & Setup](#installation--setup)
+15. [Troubleshooting](#troubleshooting)
+
+---
+
+## Installation
+
+```bash
+# Clone and install in editable mode
+git clone https://github.com/yourusername/datapilot.git
+cd datapilot
+
+uv venv && source .venv/bin/activate
+uv pip install -e .
+
+# For development / running tests
+uv pip install -e .[dev]
+```
+
+> **System requirement:** Python 3.9+. Ollama is only required if you use `use_ai=True`.
+
+---
+
+## Quick Start
+
+```python
+import pandas as pd
+import datapilot as dp
+
+df = pd.read_csv("your_dataset.csv")
+
+# Full automated EDA in one line
+dp.analyze(df)
+
+# Smart suggestions on what to fix
+dp.suggest(df)
+
+# Auto-clean the dataset
+clean_df, log = dp.auto_clean(df)
+
+# Export a full HTML report
+dp.dashboard(df, "report.html")
+```
+
+---
+
+## Module 1: Auto EDA Pipeline
+
+### `dp.analyze(df, use_ai=False, ai_model="llama3")`
+
+The primary entry point. Runs all structural checks simultaneously and prints a clean console report.
+
+```python
+# Standard rule-based analysis
+dp.analyze(df)
+
+# With local AI copilot (requires Ollama)
+dp.analyze(df, use_ai=True, ai_model="llama3")
+```
+
+**Console Output:**
+```
+==================================================
+         DATAPILOT AUTOMATED EDA REPORT         
+==================================================
+📊 Dataset Shape: 891 rows × 12 columns
+💾 Memory Usage:  0.0815 MB
+👥 Duplicate Rows: 0 (0.0%)
+--------------------------------------------------
+
+🔍 Missing Value Profile:
+   • Cabin: 687 missing (77.1%)
+   • Age: 177 missing (19.87%)
+
+🔗 Strong Linear Correlations (|r| >= 0.6):
+   • Fare ↔ Survived (pos: 0.697)
+==================================================
+```
+
+---
+
+## Module 2: Smart Column Suggestions
+
+### `dp.suggest(df)`
+
+Analyses every column and returns actionable, rule-based preprocessing recommendations — no AI or internet required.
+
+```python
+suggestions = dp.suggest(df)
+```
+
+**Detects:**
+- 🔴 ID/key columns (100% unique values — no signal, should be dropped)
+- 🔴 Constant columns (zero variance — useless for ML)
+- 🔴 Columns with >60% missing data
+- 🟡 Columns with 20–60% missing data
+- 🟡 Unencoded categorical strings (should be One-Hot or Target encoded)
+- 🟡 High-cardinality strings (need embedding or target encoding)
+- 🟡 Date strings that should be parsed to datetime
+- 🟡 Imbalanced binary targets (class ratio < 20%)
+
+**Console Output:**
+```
+========================================================
+       DATAPILOT SMART COLUMN SUGGESTIONS        
+========================================================
+
+[1] 🔴 HIGH  —  PassengerId
+     Issue:      100% unique values — likely an ID or key column
+     Fix:        Drop 'PassengerId' — unique identifiers leak no signal.
+
+[2] 🟡 MEDIUM  —  Cabin
+     Issue:      Very high missing rate (77.1%)
+     Fix:        Consider dropping 'Cabin' — over 60% nulls will degrade model quality.
+
+[3] 🟡 MEDIUM  —  Sex
+     Issue:      Unencoded categorical string (2 unique values)
+     Fix:        Encode 'Sex' using One-Hot Encoding before modelling.
+========================================================
+```
+
+**Returns:** `List[Dict]` with keys `column`, `issue`, `severity`, `suggestion`.
+
+---
+
+## Module 3: Dataset Analysis API
+
+All functions accept both `pandas.DataFrame` and `polars.DataFrame`.
+
+### `dp.summary(df)`
+
+```python
+meta = dp.summary(df)
+# Returns: {'engine_detected': 'pandas', 'rows': 891, 'columns': 12,
+#           'datatypes': {...}, 'memory_usage_mb': 0.08, 'total_missing_values': 866}
+```
+
+### `dp.missing(df)`
+
+```python
+report = dp.missing(df)
+```
+
+| column | missing_count | missing_percentage |
+|--------|:---:|:---:|
+| Cabin | 687 | 77.10% |
+| Age | 177 | 19.87% |
+
+### `dp.duplicates(df)`
+
+```python
+dups = dp.duplicates(df)
+# Returns: {'duplicate_count': 5, 'duplicate_percentage': 0.56}
+```
+
+### `dp.correlation(df, threshold=0.6)`
+
+```python
+corr = dp.correlation(df, threshold=0.6)
+print(corr['strong_positive'])  # [('Fare ↔ Survived', 0.697)]
+print(corr['strong_negative'])  # [('Pclass ↔ Fare', -0.549)]
+```
+
+---
+
+## Module 4: Outlier Detection
+
+### `dp.outliers(df, method="both", z_threshold=3.0, iqr_multiplier=1.5)`
+
+Detects outliers across all numeric columns using **IQR fencing** and/or **Z-score** — whichever flags more values when `method="both"`.
+
+```python
+result = dp.outliers(df)
+result = dp.outliers(df, method="iqr")          # IQR only
+result = dp.outliers(df, method="zscore")        # Z-score only
+result = dp.outliers(df, iqr_multiplier=3.0)     # extreme outliers only
+```
+
+**Console Output:**
+```
+========================================================
+        DATAPILOT OUTLIER DETECTION REPORT        
+  Method: BOTH | Z-threshold: 3.0 | IQR×1.5
+========================================================
+
+  🔴 HIGH  —  Fare
+     Outlier count:  12 (6.7% of non-null values)
+     IQR fences:     [-26.18, 65.63]
+     Extreme values: [512.3, 263.0, 211.3]
+```
+
+**Returns:** `Dict[column → {count, percentage, severity, values, lower_fence, upper_fence}]`
+
+---
+
+## Module 5: Auto Data Cleaning
+
+### `dp.auto_clean(df, drop_null_threshold=0.6, impute_strategy="auto", drop_id_columns=True, drop_constant_columns=True)`
+
+The killer feature. Automatically fixes your dataset and tells you exactly what it changed.
+
+```python
+clean_df, change_log = dp.auto_clean(df)
+```
+
+**What it does (in order):**
+1. Drops columns where **all values are the same** (zero variance)
+2. Drops **ID/key columns** where every row is unique
+3. Drops columns with **≥60% null values**
+4. **Imputes** remaining numeric nulls with **median**
+5. **Imputes** remaining categorical nulls with **mode**
+
+**Console Output:**
+```
+==========================================================
+         DATAPILOT AUTO-CLEAN ENGINE              
+==========================================================
+  Input:  891 rows × 12 columns
+
+  🗑️  DROP   'PassengerId'  →  ID-like column (all 891 values unique)
+  🗑️  DROP   'Cabin'        →  High null rate (77.1%) exceeds threshold (60%)
+  🔧  IMPUTE 'Age'          →  filled 177 null(s) with median=28.0
+  🔧  IMPUTE 'Embarked'     →  filled 2 null(s) with mode='S'
+
+----------------------------------------------------------
+  Output: 891 rows × 10 columns
+  ✅ 2 column(s) dropped  |  2 column(s) imputed
+==========================================================
+```
+
+**Returns:** `Tuple[cleaned_DataFrame, List[change_log_dicts]]`
+
+---
+
+## Module 6: Train/Test Drift Detection
+
+### `dp.compare(df_train, df_test, threshold=0.1)`
+
+Detects distribution shift between training and production data — the root cause of silent model degradation.
+
+```python
+flags = dp.compare(df_train, df_test)
+flags = dp.compare(df_train, df_test, threshold=0.15)  # stricter
+```
+
+**How it works:**
+- **Numeric columns:** Flags if mean shifts >15% or std shifts >25%
+- **Categorical columns:** Computes Jensen-Shannon divergence (0 = identical, 1 = completely different)
+
+**Console Output:**
+```
+============================================================
+       DATAPILOT TRAIN vs TEST DRIFT REPORT       
+  Train shape: 712 rows × 11 cols
+  Test  shape: 179 rows × 11 cols
+  Shared columns analysed: 11
+------------------------------------------------------------
+
+  ⚠️  Age  [🟡 MEDIUM]  (numeric drift)
+      Mean:  train=29.64  →  test=30.27  (2.1% shift)
+      Std:   train=14.52  →  test=15.11
+
+  ⚠️  Embarked  [🔴 HIGH]  (categorical drift)
+      JS Divergence: 0.341 (threshold: 0.1)
+      Train top value: 'S'  |  Test top value: 'C'
+```
+
+**Returns:** `List[Dict]` of flagged columns with drift statistics.
+
+---
+
+## Module 7: Visualization Engine
+
+One-liner plots, no matplotlib syntax required.
+
+```python
+dp.hist(df, "Age", bins=15)      # distribution with KDE overlay
+dp.box(df, "Fare")               # quartiles and outliers
+dp.heatmap(df)                   # annotated Pearson correlation grid
+```
+
+---
+
+## Module 8: Machine Learning Diagnostics
+
+### `dp.classification_report(y_true, y_pred, average="auto")`
+
+Auto-detects binary vs multi-class and picks the right averaging strategy.
+
+```python
+metrics = dp.classification_report(y_test, predictions)
+# Returns: {'accuracy': 0.85, 'precision': 0.82, 'recall': 0.88, 'f1_score': 0.85}
+```
+
+### `dp.regression_report(y_true, y_pred)`
+
+```python
+metrics = dp.regression_report(y_test, predictions)
+# Returns: {'mae': 2.14, 'mse': 7.32, 'rmse': 2.71, 'r2': 0.94, 'mape': 4.8, 'max_error': 8.3}
+```
+
+**Console Output:**
+```
+==================================================
+     DATAPILOT REGRESSION METRICS REPORT     
+==================================================
+  MAE        (Mean Absolute Error)   : 2.1400
+  MSE        (Mean Squared Error)    : 7.3200
+  RMSE       (Root Mean Sq. Error)   : 2.7055
+  R²         (Coefficient of Det.)   : 0.9400
+  MAPE       (Mean Abs. % Error)     : 4.80%
+  Max Error  (Worst single pred.)    : 8.3000
+--------------------------------------------------
+  Verdict: ✅ EXCELLENT — model explains ≥90% of variance.
+==================================================
+```
+
+### `dp.diagnose(train_score, test_score, metric_name="Accuracy")`
+
+```python
+dp.diagnose(train_score=0.98, test_score=0.72)
+# → 🚨 OVERFITTING DETECTED: Add regularization, gather more data.
+
+dp.diagnose(train_score=0.50, test_score=0.48)
+# → ⚠️ UNDERFITTING DETECTED: Increase model complexity.
+```
+
+---
+
+## Module 9: Standalone HTML Dashboard
+
+### `dp.dashboard(df, output_path="datapilot_report.html")`
+
+Generates a complete, **offline-ready** HTML report with:
+- Summary metric cards (rows, columns, memory, duplicates, missing values)
+- Data type overview table
+- Missing values table + inline bar chart
+- Pearson correlation pairs table
+- Inline correlation heatmap image (base64 encoded — no external dependencies)
+
+```python
+dp.dashboard(df, output_path="my_project_report.html")
+# → 🎉 Standalone HTML Dashboard successfully exported to: my_project_report.html
+```
+
+> ✅ **No internet required.** No backend server. Just open the `.html` file in any browser.
+
+---
+
+## Module 10: Performance Benchmark
+
+### `dp.benchmark(df)`
+
+Proves DataPilot's Polars-native speed advantage against equivalent Pandas operations.
+
+```python
+dp.benchmark(df)
+```
+
+**Console Output:**
+```
+==================================================================
+          DATAPILOT BENCHMARK  —  Polars vs Pandas          
+  Dataset: 891 rows × 12 columns
+==================================================================
+  Operation                   DataPilot       Pandas    Speedup
+------------------------------------------------------------------
+  Null Count                     0.09ms      0.48ms      5.3× ▓▓▓▓▓
+  Duplicate Detection            0.11ms      0.91ms      8.3× ▓▓▓▓▓▓▓▓
+  Describe / Summary Stats       0.22ms      1.47ms      6.7× ▓▓▓▓▓▓
+  Correlation Matrix             0.31ms      2.18ms      7.0× ▓▓▓▓▓▓▓
+  Group-By Mean                  0.08ms      0.54ms      6.8× ▓▓▓▓▓▓
+------------------------------------------------------------------
+  Average Speedup                                         6.8×
+
+  ⚡ DataPilot is on average 6.8× faster than Pandas on this dataset.
+  📈 Speedup grows significantly on larger datasets (1M+ rows).
+==================================================================
+```
+
+---
+
+## Module 11: AI Copilot — 5 Providers
+
+DataPilot uses a **Metadata-Only AI Pattern** — raw data rows are **never** transmitted to any provider. Only lightweight statistical summaries (e.g. `"Age has 19.8% nulls"`) are sent.
+
+### Supported Providers
+
+| Provider | Type | Default Model | Requires |
+|----------|------|--------------|----------|
+| `ollama` | 🔒 Local / Private | `llama3` | Ollama daemon running locally |
+| `openai` | ☁️ Cloud | `gpt-4o-mini` | `pip install datapilot-polars[openai]` + API key |
+| `gemini` | ☁️ Cloud | `gemini-1.5-flash` | `pip install datapilot-polars[gemini]` + API key |
+| `claude` | ☁️ Cloud | `claude-3-haiku-20240307` | `pip install datapilot-polars[claude]` + API key |
+| `groq`   | ☁️ Cloud (free tier) | `llama3-70b-8192` | `pip install datapilot-polars[groq]` + API key |
+
+
+### Usage
+
+```python
+# 🔒 Local — fully private, no API key needed (default)
+dp.analyze(df, use_ai=True)
+dp.analyze(df, use_ai=True, ai_provider="ollama", ai_model="mistral")
+
+# ☁️ OpenAI — GPT-4o, GPT-4, GPT-3.5
+dp.analyze(df, use_ai=True, ai_provider="openai", api_key="sk-...")
+dp.analyze(df, use_ai=True, ai_provider="openai", ai_model="gpt-4o", api_key="sk-...")
+
+# ☁️ Google Gemini — Gemini 1.5 Pro / Flash
+dp.analyze(df, use_ai=True, ai_provider="gemini", api_key="AIza...")
+dp.analyze(df, use_ai=True, ai_provider="gemini", ai_model="gemini-1.5-pro", api_key="AIza...")
+
+# ☁️ Anthropic Claude — Claude 3.5 Sonnet / Haiku
+dp.analyze(df, use_ai=True, ai_provider="claude", api_key="sk-ant-...")
+dp.analyze(df, use_ai=True, ai_provider="claude", ai_model="claude-3-5-sonnet-20241022", api_key="sk-ant-...")
+
+# ☁️ Groq — Ultra-fast free-tier inference (Llama3-70b, Mixtral, Gemma2)
+dp.analyze(df, use_ai=True, ai_provider="groq", api_key="gsk_...")
+dp.analyze(df, use_ai=True, ai_provider="groq", ai_model="mixtral-8x7b-32768", api_key="gsk_...")
+```
+
+### Install Only What You Need
+
+```bash
+pip install datapilot-polars[openai]    # OpenAI only
+pip install datapilot-polars[gemini]    # Google Gemini only
+pip install datapilot-polars[claude]    # Anthropic Claude only
+pip install datapilot-polars[groq]      # Groq only (free tier)
+pip install datapilot-polars[all-ai]    # All cloud providers at once
+```
+
+### AI Output Example
+
+```
+🤖 AI Copilot Insights  [GROQ]:
+• The Age column has 19.8% missing values — impute with median before
+  training to avoid biased estimates in tree models.
+• Sex and Embarked are unencoded strings — apply One-Hot Encoding;
+  Sex has only 2 values making it ideal for binary encoding.
+• The strong positive correlation between Fare and Survived (r=0.697)
+  suggests Fare is a strong predictor — include it and check for
+  outliers before normalising.
+```
+
+> 💡 **Groq is recommended for getting started** — it has a generous free tier,
+> runs Llama3-70b at extremely low latency, and requires just a free account at
+> [console.groq.com](https://console.groq.com).
+
+---
+
+## Installation & Setup
+
+### Basic Setup
+
+```bash
+git clone https://github.com/yourusername/datapilot.git
+cd datapilot
+
+uv venv
+source .venv/bin/activate
+uv pip install -e .
+```
+
+### With Cloud AI Providers
+
+```bash
+uv pip install -e .[openai]     # Add OpenAI support
+uv pip install -e .[groq]       # Add Groq support (free)
+uv pip install -e .[all-ai]     # Add all cloud AI providers
+```
+
+### Development Setup (with tests)
+
+```bash
+uv pip install -e .[dev]
+pytest
+```
+
+### Initializing Local AI (Ollama)
+
+Only needed if you use `ai_provider="ollama"` (the default):
+
+```bash
+# Start the Ollama daemon
+ollama serve
+
+# Pull a model (first time only)
+ollama pull llama3
+```
+
+---
+
+## Troubleshooting
+
+### `ModuleNotFoundError` after editing files
+```bash
+uv pip install -e . --force-reinstall
+```
+
+### AI: Ollama Connection Refused
+```bash
+ollama serve   # make sure the Ollama daemon is running
+```
+
+### AI: Missing cloud provider package
+```bash
+# Install the extra for your chosen provider
+pip install datapilot-polars[openai]   # or gemini / claude / groq / all-ai
+```
+
+### AI: Invalid or missing API key
+```python
+# Always pass the key explicitly for cloud providers
+dp.analyze(df, use_ai=True, ai_provider="openai", api_key="sk-...")
+```
+
+### Running Tests
+```bash
+pytest -v
+pytest --cov=datapilot   # with coverage report
+```
+
+---
+
+## Contributing
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup, code style guidelines, and the pull request process.
+
+---
+
+## Changelog
+
+See [CHANGELOG.md](CHANGELOG.md) for a full version history.
+
+---
+
+## License
+
+MIT License — see [LICENSE](LICENSE) for details.