datadoc-cli 0.1.0__tar.gz → 0.2.0__tar.gz

{datadoc_cli-0.1.0 → datadoc_cli-0.2.0}/.gitignore

@@ -31,3 +31,7 @@ report_*.md
 .pytest_cache/
 htmlcov/
 .coverage
+
+# Environment & Secrets
+.env
+.env.*

datadoc_cli-0.2.0/FULL_DOCUMENTATION.md

@@ -0,0 +1,144 @@
+# DATADOC: The Open Source OS for Dataset Engineering
+
+## Overview
+**DATADOC** is a blazingly fast, deterministic, and highly extensible framework designed to automate the most tedious part of Machine Learning: Data Preparation. Built on top of **Polars**, it eliminates the 80% of time Data Scientists spend manually writing Pandas scripts to clean missing values, handle outliers, encode variables, and scale features.
+
+DATADOC operates in two modes:
+1. **Rule-Based Engine:** A deterministic engine that automatically detects dataset flaws and applies best-practice transformations sequentially.
+2. **AI Planner Engine (Phase 2):** A semantic engine powered by **LiteLLM** that accepts natural language goals (e.g., "Clean this for a time-series forecast") and dynamically orchestrates the perfect pipeline—without ever sending your private, raw data rows to an external API.
+
+---
+
+## Key Features
+* **Blazing Fast Backend:** Powered by Polars, processing millions of rows in milliseconds using multi-threaded Rust execution.
+* **Intelligent Plugins:** Modular transformations (Missing Values, Outliers, Encoders, Scalers, Datetimes) that automatically trigger only when needed.
+* **Terminal Dashboard:** A massive, rich CLI UI providing health reports, before/after diffs, and visual histograms right in your terminal.
+* **Zero-Lock-in Export:** Hate black boxes? DATADOC can export the exact pipeline it generated into a standalone, highly-readable Python script.
+* **Privacy-First AI:** Our AI Planner only extracts a high-level statistical metadata schema (column names, types, null counts) to prompt the LLM, keeping your actual rows secure on your local machine.
+* **Bring Your Own Model:** Because we use LiteLLM, you can use any API key: Gemini, OpenAI, Claude, or local Ollama models.
+
+---
+
+## Installation
+
+```bash
+pip install datadoc-cli
+```
+*(Requires Python 3.9+)*
+
+---
+
+## The CLI Reference
+
+DATADOC ships with a beautiful, Typer-based CLI.
+
+### 1. `datadoc analyze <file.csv>`
+Scans your dataset and prints a comprehensive "Health Report", checking for nulls, categorical imbalances, skewed numericals, and more.
+
+### 2. `datadoc recommend <file.csv>`
+Does not touch your data. Instead, it outputs a numbered list of suggested actions it believes should be applied based on the health scan.
+
+### 3. `datadoc engineer <file.csv>`
+The core command. It runs the dataset through the engine, applying transformations and outputs a `clean_file.csv`.
+* **The AI Flag:** `datadoc engineer <file.csv> --ai --goal "Predict churn" --model "gemini/gemini-1.5-flash"`
+  * Activates the AI Planner to dynamically select plugins instead of the default rule sequence.
+
+### 4. `datadoc compare <file.csv>`
+Runs the engineering pipeline in memory and prints a git-style diff comparing the "Before" vs "After" state (rows changed, columns added, missing values resolved).
+
+### 5. `datadoc visualize <file.csv>`
+Generates a massive, interactive terminal dashboard containing ASCII bar charts and histograms comparing the numerical distributions before and after cleaning.
+
+### 6. `datadoc pipeline <file.csv>`
+Exports the generated pipeline as a standalone `.py` script so you can deploy it to production, Airflow, or dbt without taking DATADOC as a dependency.
+
+### 7. `datadoc plugin`
+Lists all registered plugins, their current version, priority order, and explanations of how they work.
+
+---
+
+## Python SDK Reference
+
+You can use DATADOC programmatically inside your Jupyter Notebooks or Python backends.
+
+```python
+import polars as pl
+from datadoc.core.engine import DATADOC
+
+# 1. Load the dataset
+doc = DATADOC("data.csv")
+
+# 2. Get the health report
+report = doc.analyze()
+print(report["rows"], report["cols"])
+
+# 3. Engineer the data (Rule-Based)
+clean_df = doc.engineer()
+
+# 4. Engineer the data (AI Planner)
+# Requires python-dotenv and a .env file with your API Key (e.g. GEMINI_API_KEY)
+clean_df_ai = doc.ai_engineer(
+    model="gemini/gemini-1.5-flash",
+    goal="Prepare this data for XGBoost Classification"
+)
+
+# 5. Export Python Script
+script = doc.pipeline()
+with open("my_pipeline.py", "w") as f:
+    f.write(script)
+```
+
+---
+
+## The Plugin Architecture
+
+DATADOC is built heavily on the **Strategy Pattern**. The core engine does almost nothing on its own; it simply orchestrates a list of `BasePlugin` objects.
+
+### Current Built-in Plugins
+1. **MissingValuePlugin:** Fills numeric nulls with the median and categorical nulls with the mode.
+2. **OutlierPlugin:** Uses the Interquartile Range (IQR) method to cap extreme values at the 5th and 95th percentiles.
+3. **DatetimePlugin:** Automatically detects string dates, converts them to Polars Datetime objects, and extracts features like `year`, `month`, and `day`.
+4. **CategoricalEncoderPlugin:** Automatically detects low-cardinality string columns and performs one-hot encoding (dummy variables).
+5. **ScalingPlugin:** Uses Min-Max scaling on numeric columns if their maximum values vary by more than 100x, ensuring algorithms like SVMs and Neural Networks converge quickly.
+
+### Writing a Custom Plugin
+Creating a new rule is as simple as inheriting from `BasePlugin`.
+
+```python
+import polars as pl
+from datadoc.plugins.base import BasePlugin
+
+class MyCustomPlugin(BasePlugin):
+    name = "MyCustomPlugin"
+    priority = 10 
+
+    def analyze(self, df: pl.DataFrame) -> dict:
+        # Return a dictionary of findings. 
+        # If any key starts with 'has_', the engine will trigger this plugin.
+        return {"has_work": True}
+
+    def recommend(self, analysis: dict) -> list[str]:
+        return ["I will apply custom logic to this dataframe."]
+
+    def apply(self, df: pl.DataFrame) -> pl.DataFrame:
+        # Return the transformed dataframe
+        return df.with_columns(pl.col("A") * 2)
+```
+
+Once defined, simply append it to `doc.plugins` inside `engine.py`.
+
+---
+
+## Privacy & The AI Planner
+Security is a massive concern in enterprise data. When you run `datadoc engineer --ai`, the LLM prompt looks like this:
+
+```json
+{
+  "rows": 100000,
+  "columns": 5,
+  "schema": {"age": "Int64", "name": "String"},
+  "null_counts": {"age": 40},
+  "available_plugins": ["MissingValuePlugin", "ScalingPlugin"]
+}
+```
+**No PII or actual rows are sent over the wire.** The LLM simply replies with a strict JSON array instructing DATADOC on which local plugins to execute to achieve the user's goal.

datadoc_cli-0.2.0/PKG-INFO

@@ -0,0 +1,206 @@
+Metadata-Version: 2.4
+Name: datadoc-cli
+Version: 0.2.0
+Summary: The Open Source Operating System for Dataset Engineering.
+Project-URL: Homepage, https://github.com/narain-karti/DATADOC
+Project-URL: Repository, https://github.com/narain-karti/DATADOC
+Project-URL: Issues, https://github.com/narain-karti/DATADOC/issues
+Author: narain-karti
+License-Expression: MIT
+License-File: LICENSE
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering
+Requires-Python: >=3.9
+Requires-Dist: litellm>=1.40.0
+Requires-Dist: numpy>=1.24.0
+Requires-Dist: plotext>=5.2.8
+Requires-Dist: polars>=0.20.0
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: python-dotenv>=1.0.0
+Requires-Dist: rich>=13.7.0
+Requires-Dist: typer>=0.12.0
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0.0; extra == 'dev'
+Requires-Dist: ruff>=0.4.0; extra == 'dev'
+Description-Content-Type: text/markdown
+
+<p align="center">
+  <pre>
+ ____    _  _____  _    ____   ___   ____
+|  _ \  / \|_   _|/ \  |  _ \ / _ \ / ___|
+| | | |/ _ \ | | / _ \ | | | | | | | |
+| |_| / ___ \| |/ ___ \| |_| | |_| | |___
+|____/_/   \_\_/_/   \_\____/ \___/ \____|
+  </pre>
+</p>
+
+<h3 align="center">The Open Source Operating System for Dataset Engineering.</h3>
+
+<p align="center">
+  <a href="https://pypi.org/project/datadoc-cli/"><img alt="PyPI version" src="https://img.shields.io/pypi/v/datadoc-cli.svg"></a>
+  <a href="https://pypi.org/project/datadoc-cli/"><img alt="Python Versions" src="https://img.shields.io/pypi/pyversions/datadoc-cli.svg"></a>
+  <a href="https://github.com/narain-karti/DATADOC/blob/main/LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/License-MIT-yellow.svg"></a>
+</p>
+
+<p align="center">
+  <a href="#installation">Install</a> |
+  <a href="#why-datadoc">Why DATADOC?</a> |
+  <a href="#quick-start">Quick Start</a> |
+  <a href="#cli-commands">CLI Commands</a> |
+  <a href="#architecture--plugins">Architecture</a>
+</p>
+
+---
+
+## 🚀 What is DATADOC?
+
+**DATADOC** is an intelligent, blazing-fast Command Line Interface (CLI) and Python Library designed to completely automate the most tedious part of Machine Learning: **Dataset Engineering and Data Cleaning**. 
+
+Powered by a high-performance **Polars** backend, DATADOC analyzes your raw CSV files, diagnoses missing values, outliers, and schema issues, and **automatically engineers a machine-learning-ready dataset in milliseconds**.
+
+**DATADOC is NOT just another EDA (Exploratory Data Analysis) tool.** It doesn't just show you charts. It **fixes your data** and hands you a portable, deterministic Python script to replicate the pipeline anywhere.
+
+### ⚡ The Impact: Why Industry Professionals Use DATADOC
+
+Data Scientists and ML Engineers spend **80% of their time cleaning data** and only 20% training models. 
+DATADOC eliminates the 80%.
+
+- **Save Hundreds of Hours:** Stop writing boilerplate code to impute nulls, one-hot encode categorical variables, or clip outliers. DATADOC does it in one command.
+- **Zero Black-Box AI:** Every transformation is strictly mathematical (IQR, medians, mode). It is 100% deterministic, explainable, and safe for enterprise production environments.
+- **Lightning Fast:** By utilizing `polars` (written in Rust) instead of `pandas`, DATADOC processes millions of rows with minimal memory overhead.
+- **Agentic AI Integration:** DATADOC features a built-in AI Planner and an interactive Chat Assistant that can autonomously analyze your dataset, generate engineering plans, and execute plugins using tool-calling!
+- **Avoid Data Leakage:** Built-in safeguards ensure that data scaling and imputation are handled correctly.
+
+---
+
+## 📦 Installation
+
+DATADOC is published on PyPI. You can install it globally via `pip` or `uv`:
+
+```bash
+pip install datadoc-cli
+```
+
+*(Requires Python 3.9+)*
+
+---
+
+## 🛠️ Quick Start (CLI)
+
+You don't need to write a single line of Python to clean your data. Just use the CLI.
+
+```bash
+# 1. Analyze your dataset's health (shows a beautiful terminal report)
+datadoc analyze raw_data.csv
+
+# 2. Get recommendations (DATADOC tells you exactly what is wrong)
+datadoc recommend raw_data.csv
+
+# 3. AUTO-ENGINEER! (Fixes everything and saves clean_raw_data.csv)
+datadoc engineer raw_data.csv
+
+# 4. Compare the before vs. after visually in your terminal
+datadoc compare raw_data.csv clean_raw_data.csv
+
+# 5. Export a standalone Python script to automate this in the future
+datadoc pipeline raw_data.csv
+
+# 6. Have an interactive AI session where the LLM engineers your data via chat!
+datadoc chat raw_data.csv
+
+*(Pro Tip: Add `--ai` to `analyze`, `recommend`, or `engineer` for AI-driven insights and orchestration!)*
+```
+
+---
+
+## 🐍 Python SDK (Library Usage)
+
+DATADOC is also a powerful Python library. You can import the engine directly into your Jupyter Notebooks or backend servers:
+
+```python
+from datadoc.core.engine import DATADOC
+
+# Initialize the blazing-fast Polars engine
+doc = DATADOC("raw_data.csv")
+
+# Generate a diagnostic report
+report = doc.analyze()
+print(report)
+
+# Automatically engineer the dataset
+clean_df = doc.engineer()
+clean_df.write_csv("clean_data.csv")
+
+# Export the generated pipeline script
+with open("my_pipeline.py", "w") as f:
+    f.write(doc.pipeline())
+```
+
+---
+
+## 💻 CLI Commands Reference
+
+| Command | Description |
+|---------|-------------|
+| `datadoc analyze <file>` | Scans dataset and shows a health report with status indicators |
+| `datadoc recommend <file>` | Lists suggested engineering steps without modifying data |
+| `datadoc engineer <file>` | Automatically applies all recommended transformations |
+| `datadoc chat <file>` | Starts an interactive AI session with autonomous tool-calling |
+| `datadoc compare <file>` | Shows a before/after diff of the raw vs engineered dataset |
+| `datadoc pipeline <file>` | Exports a standalone `.py` script with the exact Polars code |
+| `datadoc visualize <file>` | Renders stunning terminal-based charts for numeric distributions |
+| `datadoc plugin` | Lists all registered plugins with priority and descriptions |
+| `datadoc version` | Displays the DATADOC version |
+
+*(Note: Use the `--ai` flag on `analyze`, `recommend`, or `engineer` for LLM-powered execution!)*
+
+---
+
+## 🧩 Architecture & Plugins
+
+DATADOC operates as an orchestrator. It passes your dataset through an isolated chain of plugins in a strict priority order.
+
+| Priority | Plugin | Action Performed |
+|----------|--------|-------------|
+| 10 | **MissingValuePlugin** | Imputes missing numeric values with median, categorical with mode |
+| 20 | **OutlierPlugin** | Detects outliers via IQR and clips them dynamically |
+| 30 | **DatetimePlugin** | Detects date strings and extracts year, month, day, day_of_week |
+| 40 | **CategoricalEncoderPlugin** | One-Hot Encodes categorical columns (< 10 unique values) |
+| 45 | **ScalingPlugin** | Standard scales numeric columns when scale ratio exceeds 10x |
+
+Every plugin implements a strict `BasePlugin` interface ensuring it can `analyze()`, `apply()`, `rollback()`, and `generate_code()`. 
+
+Want to build your own? See [CONTRIBUTING.md](CONTRIBUTING.md) to learn how to create and register custom plugins!
+
+---
+
+## 🗺️ Roadmap
+
+- [x] Core Engine with plugin orchestration
+- [x] 5 Built-in deterministic plugins
+- [x] Stunning Rich Terminal UI
+- [x] Pipeline export capability
+- [x] **Polars Backend Migration (100x Performance Boost)**
+- [x] PyPI Release (`pip install datadoc-cli`)
+- [x] Phase 2: Agentic AI Planner (LLM Orchestration)
+- [x] Interactive AI Chat with Tool-Calling capabilities
+- [ ] Export targets for `dbt` and Apache Airflow
+- [ ] REST API (FastAPI) wrapper
+
+---
+
+## ⚖️ License
+
+This project is licensed under the MIT License. See [LICENSE](LICENSE) for details.
+
+## 🤝 Contributing
+
+We welcome contributions from the community! If you'd like to add a new plugin or improve the core engine, please see [CONTRIBUTING.md](CONTRIBUTING.md).

datadoc_cli-0.2.0/README.md

@@ -0,0 +1,172 @@
+<p align="center">
+  <pre>
+ ____    _  _____  _    ____   ___   ____
+|  _ \  / \|_   _|/ \  |  _ \ / _ \ / ___|
+| | | |/ _ \ | | / _ \ | | | | | | | |
+| |_| / ___ \| |/ ___ \| |_| | |_| | |___
+|____/_/   \_\_/_/   \_\____/ \___/ \____|
+  </pre>
+</p>
+
+<h3 align="center">The Open Source Operating System for Dataset Engineering.</h3>
+
+<p align="center">
+  <a href="https://pypi.org/project/datadoc-cli/"><img alt="PyPI version" src="https://img.shields.io/pypi/v/datadoc-cli.svg"></a>
+  <a href="https://pypi.org/project/datadoc-cli/"><img alt="Python Versions" src="https://img.shields.io/pypi/pyversions/datadoc-cli.svg"></a>
+  <a href="https://github.com/narain-karti/DATADOC/blob/main/LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/License-MIT-yellow.svg"></a>
+</p>
+
+<p align="center">
+  <a href="#installation">Install</a> |
+  <a href="#why-datadoc">Why DATADOC?</a> |
+  <a href="#quick-start">Quick Start</a> |
+  <a href="#cli-commands">CLI Commands</a> |
+  <a href="#architecture--plugins">Architecture</a>
+</p>
+
+---
+
+## 🚀 What is DATADOC?
+
+**DATADOC** is an intelligent, blazing-fast Command Line Interface (CLI) and Python Library designed to completely automate the most tedious part of Machine Learning: **Dataset Engineering and Data Cleaning**. 
+
+Powered by a high-performance **Polars** backend, DATADOC analyzes your raw CSV files, diagnoses missing values, outliers, and schema issues, and **automatically engineers a machine-learning-ready dataset in milliseconds**.
+
+**DATADOC is NOT just another EDA (Exploratory Data Analysis) tool.** It doesn't just show you charts. It **fixes your data** and hands you a portable, deterministic Python script to replicate the pipeline anywhere.
+
+### ⚡ The Impact: Why Industry Professionals Use DATADOC
+
+Data Scientists and ML Engineers spend **80% of their time cleaning data** and only 20% training models. 
+DATADOC eliminates the 80%.
+
+- **Save Hundreds of Hours:** Stop writing boilerplate code to impute nulls, one-hot encode categorical variables, or clip outliers. DATADOC does it in one command.
+- **Zero Black-Box AI:** Every transformation is strictly mathematical (IQR, medians, mode). It is 100% deterministic, explainable, and safe for enterprise production environments.
+- **Lightning Fast:** By utilizing `polars` (written in Rust) instead of `pandas`, DATADOC processes millions of rows with minimal memory overhead.
+- **Agentic AI Integration:** DATADOC features a built-in AI Planner and an interactive Chat Assistant that can autonomously analyze your dataset, generate engineering plans, and execute plugins using tool-calling!
+- **Avoid Data Leakage:** Built-in safeguards ensure that data scaling and imputation are handled correctly.
+
+---
+
+## 📦 Installation
+
+DATADOC is published on PyPI. You can install it globally via `pip` or `uv`:
+
+```bash
+pip install datadoc-cli
+```
+
+*(Requires Python 3.9+)*
+
+---
+
+## 🛠️ Quick Start (CLI)
+
+You don't need to write a single line of Python to clean your data. Just use the CLI.
+
+```bash
+# 1. Analyze your dataset's health (shows a beautiful terminal report)
+datadoc analyze raw_data.csv
+
+# 2. Get recommendations (DATADOC tells you exactly what is wrong)
+datadoc recommend raw_data.csv
+
+# 3. AUTO-ENGINEER! (Fixes everything and saves clean_raw_data.csv)
+datadoc engineer raw_data.csv
+
+# 4. Compare the before vs. after visually in your terminal
+datadoc compare raw_data.csv clean_raw_data.csv
+
+# 5. Export a standalone Python script to automate this in the future
+datadoc pipeline raw_data.csv
+
+# 6. Have an interactive AI session where the LLM engineers your data via chat!
+datadoc chat raw_data.csv
+
+*(Pro Tip: Add `--ai` to `analyze`, `recommend`, or `engineer` for AI-driven insights and orchestration!)*
+```
+
+---
+
+## 🐍 Python SDK (Library Usage)
+
+DATADOC is also a powerful Python library. You can import the engine directly into your Jupyter Notebooks or backend servers:
+
+```python
+from datadoc.core.engine import DATADOC
+
+# Initialize the blazing-fast Polars engine
+doc = DATADOC("raw_data.csv")
+
+# Generate a diagnostic report
+report = doc.analyze()
+print(report)
+
+# Automatically engineer the dataset
+clean_df = doc.engineer()
+clean_df.write_csv("clean_data.csv")
+
+# Export the generated pipeline script
+with open("my_pipeline.py", "w") as f:
+    f.write(doc.pipeline())
+```
+
+---
+
+## 💻 CLI Commands Reference
+
+| Command | Description |
+|---------|-------------|
+| `datadoc analyze <file>` | Scans dataset and shows a health report with status indicators |
+| `datadoc recommend <file>` | Lists suggested engineering steps without modifying data |
+| `datadoc engineer <file>` | Automatically applies all recommended transformations |
+| `datadoc chat <file>` | Starts an interactive AI session with autonomous tool-calling |
+| `datadoc compare <file>` | Shows a before/after diff of the raw vs engineered dataset |
+| `datadoc pipeline <file>` | Exports a standalone `.py` script with the exact Polars code |
+| `datadoc visualize <file>` | Renders stunning terminal-based charts for numeric distributions |
+| `datadoc plugin` | Lists all registered plugins with priority and descriptions |
+| `datadoc version` | Displays the DATADOC version |
+
+*(Note: Use the `--ai` flag on `analyze`, `recommend`, or `engineer` for LLM-powered execution!)*
+
+---
+
+## 🧩 Architecture & Plugins
+
+DATADOC operates as an orchestrator. It passes your dataset through an isolated chain of plugins in a strict priority order.
+
+| Priority | Plugin | Action Performed |
+|----------|--------|-------------|
+| 10 | **MissingValuePlugin** | Imputes missing numeric values with median, categorical with mode |
+| 20 | **OutlierPlugin** | Detects outliers via IQR and clips them dynamically |
+| 30 | **DatetimePlugin** | Detects date strings and extracts year, month, day, day_of_week |
+| 40 | **CategoricalEncoderPlugin** | One-Hot Encodes categorical columns (< 10 unique values) |
+| 45 | **ScalingPlugin** | Standard scales numeric columns when scale ratio exceeds 10x |
+
+Every plugin implements a strict `BasePlugin` interface ensuring it can `analyze()`, `apply()`, `rollback()`, and `generate_code()`. 
+
+Want to build your own? See [CONTRIBUTING.md](CONTRIBUTING.md) to learn how to create and register custom plugins!
+
+---
+
+## 🗺️ Roadmap
+
+- [x] Core Engine with plugin orchestration
+- [x] 5 Built-in deterministic plugins
+- [x] Stunning Rich Terminal UI
+- [x] Pipeline export capability
+- [x] **Polars Backend Migration (100x Performance Boost)**
+- [x] PyPI Release (`pip install datadoc-cli`)
+- [x] Phase 2: Agentic AI Planner (LLM Orchestration)
+- [x] Interactive AI Chat with Tool-Calling capabilities
+- [ ] Export targets for `dbt` and Apache Airflow
+- [ ] REST API (FastAPI) wrapper
+
+---
+
+## ⚖️ License
+
+This project is licensed under the MIT License. See [LICENSE](LICENSE) for details.
+
+## 🤝 Contributing
+
+We welcome contributions from the community! If you'd like to add a new plugin or improve the core engine, please see [CONTRIBUTING.md](CONTRIBUTING.md).