datadoc-cli 0.1.0__tar.gz → 0.3.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (44) hide show
  1. {datadoc_cli-0.1.0 → datadoc_cli-0.3.0}/.gitignore +4 -0
  2. datadoc_cli-0.3.0/FULL_DOCUMENTATION.md +144 -0
  3. {datadoc_cli-0.1.0 → datadoc_cli-0.3.0}/LICENSE +1 -1
  4. datadoc_cli-0.3.0/PKG-INFO +210 -0
  5. datadoc_cli-0.3.0/README.md +176 -0
  6. datadoc_cli-0.3.0/comprehensive_test.csv +11 -0
  7. {datadoc_cli-0.1.0 → datadoc_cli-0.3.0}/datadoc/cli/app.py +298 -32
  8. datadoc_cli-0.3.0/datadoc/core/engine.py +353 -0
  9. {datadoc_cli-0.1.0 → datadoc_cli-0.3.0}/datadoc/plugins/base.py +16 -18
  10. {datadoc_cli-0.1.0 → datadoc_cli-0.3.0}/datadoc/plugins/datetime_feat.py +48 -22
  11. datadoc_cli-0.3.0/datadoc/plugins/encoders.py +95 -0
  12. {datadoc_cli-0.1.0 → datadoc_cli-0.3.0}/datadoc/plugins/missing_values.py +2 -5
  13. {datadoc_cli-0.1.0 → datadoc_cli-0.3.0}/datadoc/plugins/outliers.py +28 -26
  14. {datadoc_cli-0.1.0 → datadoc_cli-0.3.0}/datadoc/plugins/scaling.py +38 -27
  15. datadoc_cli-0.3.0/diagnose.py +97 -0
  16. datadoc_cli-0.3.0/diagnose_output.txt +130 -0
  17. datadoc_cli-0.3.0/diagnose_output2.txt +76 -0
  18. datadoc_cli-0.3.0/docs/api-docs.html +140 -0
  19. datadoc_cli-0.3.0/docs/contribute.html +128 -0
  20. datadoc_cli-0.3.0/docs/css/style.css +522 -0
  21. datadoc_cli-0.3.0/docs/how-it-works.html +127 -0
  22. datadoc_cli-0.3.0/docs/index.html +259 -0
  23. datadoc_cli-0.3.0/docs/license.html +106 -0
  24. datadoc_cli-0.3.0/docs/sdk.html +137 -0
  25. datadoc_cli-0.3.0/docs/setup.html +160 -0
  26. datadoc_cli-0.3.0/generate_heavy_test.py +113 -0
  27. datadoc_cli-0.3.0/heavy_test.csv +10001 -0
  28. datadoc_cli-0.3.0/heavy_test_output.txt +52 -0
  29. {datadoc_cli-0.1.0 → datadoc_cli-0.3.0}/pyproject.toml +4 -1
  30. datadoc_cli-0.3.0/test_output.txt +7 -0
  31. {datadoc_cli-0.1.0 → datadoc_cli-0.3.0}/tests/test_core.py +60 -5
  32. datadoc_cli-0.3.0/uv.lock +3940 -0
  33. datadoc_cli-0.1.0/PKG-INFO +0 -254
  34. datadoc_cli-0.1.0/README.md +0 -223
  35. datadoc_cli-0.1.0/datadoc/core/engine.py +0 -142
  36. datadoc_cli-0.1.0/datadoc/plugins/encoders.py +0 -73
  37. {datadoc_cli-0.1.0 → datadoc_cli-0.3.0}/.github/workflows/ci.yml +0 -0
  38. {datadoc_cli-0.1.0 → datadoc_cli-0.3.0}/CONTRIBUTING.md +0 -0
  39. {datadoc_cli-0.1.0 → datadoc_cli-0.3.0}/datadoc/__init__.py +0 -0
  40. {datadoc_cli-0.1.0 → datadoc_cli-0.3.0}/datadoc/cli/__init__.py +0 -0
  41. {datadoc_cli-0.1.0 → datadoc_cli-0.3.0}/datadoc/core/__init__.py +0 -0
  42. {datadoc_cli-0.1.0 → datadoc_cli-0.3.0}/datadoc/plugins/__init__.py +0 -0
  43. {datadoc_cli-0.1.0 → datadoc_cli-0.3.0}/test.csv +0 -0
  44. {datadoc_cli-0.1.0 → datadoc_cli-0.3.0}/tests/__init__.py +0 -0
@@ -31,3 +31,7 @@ report_*.md
31
31
  .pytest_cache/
32
32
  htmlcov/
33
33
  .coverage
34
+
35
+ # Environment & Secrets
36
+ .env
37
+ .env.*
@@ -0,0 +1,144 @@
1
+ # DATADOC: The Open Source OS for Dataset Engineering
2
+
3
+ ## Overview
4
+ **DATADOC** is a blazingly fast, deterministic, and highly extensible framework designed to automate the most tedious part of Machine Learning: Data Preparation. Built on top of **Polars**, it eliminates the 80% of time Data Scientists spend manually writing Pandas scripts to clean missing values, handle outliers, encode variables, and scale features.
5
+
6
+ DATADOC operates in two modes:
7
+ 1. **Rule-Based Engine:** A deterministic engine that automatically detects dataset flaws and applies best-practice transformations sequentially.
8
+ 2. **AI Planner Engine (Phase 2):** A semantic engine powered by **LiteLLM** that accepts natural language goals (e.g., "Clean this for a time-series forecast") and dynamically orchestrates the perfect pipeline—without ever sending your private, raw data rows to an external API.
9
+
10
+ ---
11
+
12
+ ## Key Features
13
+ * **Blazing Fast Backend:** Powered by Polars, processing millions of rows in milliseconds using multi-threaded Rust execution.
14
+ * **Intelligent Plugins:** Modular transformations (Missing Values, Outliers, Encoders, Scalers, Datetimes) that automatically trigger only when needed.
15
+ * **Terminal Dashboard:** A massive, rich CLI UI providing health reports, before/after diffs, and visual histograms right in your terminal.
16
+ * **Zero-Lock-in Export:** Hate black boxes? DATADOC can export the exact pipeline it generated into a standalone, highly-readable Python script.
17
+ * **Privacy-First AI:** Our AI Planner only extracts a high-level statistical metadata schema (column names, types, null counts) to prompt the LLM, keeping your actual rows secure on your local machine.
18
+ * **Bring Your Own Model:** Because we use LiteLLM, you can use any API key: Gemini, OpenAI, Claude, or local Ollama models.
19
+
20
+ ---
21
+
22
+ ## Installation
23
+
24
+ ```bash
25
+ pip install datadoc-cli
26
+ ```
27
+ *(Requires Python 3.9+)*
28
+
29
+ ---
30
+
31
+ ## The CLI Reference
32
+
33
+ DATADOC ships with a beautiful, Typer-based CLI.
34
+
35
+ ### 1. `datadoc analyze <file.csv>`
36
+ Scans your dataset and prints a comprehensive "Health Report", checking for nulls, categorical imbalances, skewed numericals, and more.
37
+
38
+ ### 2. `datadoc recommend <file.csv>`
39
+ Does not touch your data. Instead, it outputs a numbered list of suggested actions it believes should be applied based on the health scan.
40
+
41
+ ### 3. `datadoc engineer <file.csv>`
42
+ The core command. It runs the dataset through the engine, applying transformations and outputs a `clean_file.csv`.
43
+ * **The AI Flag:** `datadoc engineer <file.csv> --ai --goal "Predict churn" --model "gemini/gemini-1.5-flash"`
44
+ * Activates the AI Planner to dynamically select plugins instead of the default rule sequence.
45
+
46
+ ### 4. `datadoc compare <file.csv>`
47
+ Runs the engineering pipeline in memory and prints a git-style diff comparing the "Before" vs "After" state (rows changed, columns added, missing values resolved).
48
+
49
+ ### 5. `datadoc visualize <file.csv>`
50
+ Generates a massive, interactive terminal dashboard containing ASCII bar charts and histograms comparing the numerical distributions before and after cleaning.
51
+
52
+ ### 6. `datadoc pipeline <file.csv>`
53
+ Exports the generated pipeline as a standalone `.py` script so you can deploy it to production, Airflow, or dbt without taking DATADOC as a dependency.
54
+
55
+ ### 7. `datadoc plugin`
56
+ Lists all registered plugins, their current version, priority order, and explanations of how they work.
57
+
58
+ ---
59
+
60
+ ## Python SDK Reference
61
+
62
+ You can use DATADOC programmatically inside your Jupyter Notebooks or Python backends.
63
+
64
+ ```python
65
+ import polars as pl
66
+ from datadoc.core.engine import DATADOC
67
+
68
+ # 1. Load the dataset
69
+ doc = DATADOC("data.csv")
70
+
71
+ # 2. Get the health report
72
+ report = doc.analyze()
73
+ print(report["rows"], report["cols"])
74
+
75
+ # 3. Engineer the data (Rule-Based)
76
+ clean_df = doc.engineer()
77
+
78
+ # 4. Engineer the data (AI Planner)
79
+ # Requires python-dotenv and a .env file with your API Key (e.g. GEMINI_API_KEY)
80
+ clean_df_ai = doc.ai_engineer(
81
+ model="gemini/gemini-1.5-flash",
82
+ goal="Prepare this data for XGBoost Classification"
83
+ )
84
+
85
+ # 5. Export Python Script
86
+ script = doc.pipeline()
87
+ with open("my_pipeline.py", "w") as f:
88
+ f.write(script)
89
+ ```
90
+
91
+ ---
92
+
93
+ ## The Plugin Architecture
94
+
95
+ DATADOC is built heavily on the **Strategy Pattern**. The core engine does almost nothing on its own; it simply orchestrates a list of `BasePlugin` objects.
96
+
97
+ ### Current Built-in Plugins
98
+ 1. **MissingValuePlugin:** Fills numeric nulls with the median and categorical nulls with the mode.
99
+ 2. **OutlierPlugin:** Uses the Interquartile Range (IQR) method to cap extreme values at the 5th and 95th percentiles.
100
+ 3. **DatetimePlugin:** Automatically detects string dates, converts them to Polars Datetime objects, and extracts features like `year`, `month`, and `day`.
101
+ 4. **CategoricalEncoderPlugin:** Automatically detects low-cardinality string columns and performs one-hot encoding (dummy variables).
102
+ 5. **ScalingPlugin:** Uses Min-Max scaling on numeric columns if their maximum values vary by more than 100x, ensuring algorithms like SVMs and Neural Networks converge quickly.
103
+
104
+ ### Writing a Custom Plugin
105
+ Creating a new rule is as simple as inheriting from `BasePlugin`.
106
+
107
+ ```python
108
+ import polars as pl
109
+ from datadoc.plugins.base import BasePlugin
110
+
111
+ class MyCustomPlugin(BasePlugin):
112
+ name = "MyCustomPlugin"
113
+ priority = 10
114
+
115
+ def analyze(self, df: pl.DataFrame) -> dict:
116
+ # Return a dictionary of findings.
117
+ # If any key starts with 'has_', the engine will trigger this plugin.
118
+ return {"has_work": True}
119
+
120
+ def recommend(self, analysis: dict) -> list[str]:
121
+ return ["I will apply custom logic to this dataframe."]
122
+
123
+ def apply(self, df: pl.DataFrame) -> pl.DataFrame:
124
+ # Return the transformed dataframe
125
+ return df.with_columns(pl.col("A") * 2)
126
+ ```
127
+
128
+ Once defined, simply append it to `doc.plugins` inside `engine.py`.
129
+
130
+ ---
131
+
132
+ ## Privacy & The AI Planner
133
+ Security is a massive concern in enterprise data. When you run `datadoc engineer --ai`, the LLM prompt looks like this:
134
+
135
+ ```json
136
+ {
137
+ "rows": 100000,
138
+ "columns": 5,
139
+ "schema": {"age": "Int64", "name": "String"},
140
+ "null_counts": {"age": 40},
141
+ "available_plugins": ["MissingValuePlugin", "ScalingPlugin"]
142
+ }
143
+ ```
144
+ **No PII or actual rows are sent over the wire.** The LLM simply replies with a strict JSON array instructing DATADOC on which local plugins to execute to achieve the user's goal.
@@ -1,6 +1,6 @@
1
1
  MIT License
2
2
 
3
- Copyright (c) 2024 narain-karti
3
+ Copyright (c) 2026 narain-karti
4
4
 
5
5
  Permission is hereby granted, free of charge, to any person obtaining a copy
6
6
  of this software and associated documentation files (the "Software"), to deal
@@ -0,0 +1,210 @@
1
+ Metadata-Version: 2.4
2
+ Name: datadoc-cli
3
+ Version: 0.3.0
4
+ Summary: The Open Source Operating System for Dataset Engineering.
5
+ Project-URL: Homepage, https://github.com/narain-karti/DATADOC
6
+ Project-URL: Repository, https://github.com/narain-karti/DATADOC
7
+ Project-URL: Issues, https://github.com/narain-karti/DATADOC/issues
8
+ Author: narain-karti
9
+ License-Expression: MIT
10
+ License-File: LICENSE
11
+ Classifier: Development Status :: 3 - Alpha
12
+ Classifier: Intended Audience :: Developers
13
+ Classifier: Intended Audience :: Science/Research
14
+ Classifier: License :: OSI Approved :: MIT License
15
+ Classifier: Programming Language :: Python :: 3
16
+ Classifier: Programming Language :: Python :: 3.9
17
+ Classifier: Programming Language :: Python :: 3.10
18
+ Classifier: Programming Language :: Python :: 3.11
19
+ Classifier: Programming Language :: Python :: 3.12
20
+ Classifier: Topic :: Scientific/Engineering
21
+ Requires-Python: >=3.9
22
+ Requires-Dist: litellm>=1.40.0
23
+ Requires-Dist: numpy>=1.24.0
24
+ Requires-Dist: plotext>=5.2.8
25
+ Requires-Dist: polars>=0.20.0
26
+ Requires-Dist: pydantic>=2.0.0
27
+ Requires-Dist: python-dotenv>=1.0.0
28
+ Requires-Dist: rich>=13.7.0
29
+ Requires-Dist: typer>=0.12.0
30
+ Provides-Extra: dev
31
+ Requires-Dist: pytest>=8.0.0; extra == 'dev'
32
+ Requires-Dist: ruff>=0.4.0; extra == 'dev'
33
+ Description-Content-Type: text/markdown
34
+
35
+ <p align="center">
36
+ <pre align="center">
37
+ ____ _ _____ _ ____ ___ ____
38
+ | _ \ / \|_ _|/ \ | _ \ / _ \ / ___|
39
+ | | | |/ _ \ | | / _ \ | | | | | | | |
40
+ | |_| / ___ \| |/ ___ \| |_| | |_| | |___
41
+ |____/_/ \_\_/_/ \_\____/ \___/ \____|
42
+ </pre>
43
+ </p>
44
+
45
+ <h3 align="center">The Open Source Operating System for Dataset Engineering.</h3>
46
+
47
+ <p align="center">
48
+ <a href="https://narain-karti.github.io/DATADOC/"><b>&#x1F4D6; View Official Documentation Website</b></a>
49
+ </p>
50
+
51
+ <p align="center">
52
+ <a href="https://pypi.org/project/datadoc-cli/"><img alt="PyPI version" src="https://img.shields.io/pypi/v/datadoc-cli.svg"></a>
53
+ <a href="https://pypi.org/project/datadoc-cli/"><img alt="Python Versions" src="https://img.shields.io/pypi/pyversions/datadoc-cli.svg"></a>
54
+ <a href="https://github.com/narain-karti/DATADOC/blob/main/LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/License-MIT-yellow.svg"></a>
55
+ </p>
56
+
57
+ <p align="center">
58
+ <a href="#installation"><b>Install</b></a> &bull;
59
+ <a href="#why-datadoc"><b>Why DATADOC?</b></a> &bull;
60
+ <a href="#quick-start"><b>Quick Start</b></a> &bull;
61
+ <a href="#cli-commands"><b>CLI Commands</b></a> &bull;
62
+ <a href="#architecture--plugins"><b>Architecture</b></a>
63
+ </p>
64
+
65
+ <hr>
66
+
67
+ ## 🚀 What is DATADOC?
68
+
69
+ **DATADOC** is an intelligent, blazing-fast Command Line Interface (CLI) and Python Library designed to completely automate the most tedious part of Machine Learning: **Dataset Engineering and Data Cleaning**.
70
+
71
+ Powered by a high-performance **Polars** backend, DATADOC analyzes your raw CSV files, diagnoses missing values, outliers, and schema issues, and **automatically engineers a machine-learning-ready dataset in milliseconds**.
72
+
73
+ **DATADOC is NOT just another EDA (Exploratory Data Analysis) tool.** It doesn't just show you charts. It **fixes your data** and hands you a portable, deterministic Python script to replicate the pipeline anywhere.
74
+
75
+ ### ⚡ The Impact: Why Industry Professionals Use DATADOC
76
+
77
+ Data Scientists and ML Engineers spend **80% of their time cleaning data** and only 20% training models.
78
+ DATADOC eliminates the 80%.
79
+
80
+ - **Save Hundreds of Hours:** Stop writing boilerplate code to impute nulls, one-hot encode categorical variables, or clip outliers. DATADOC does it in one command.
81
+ - **Zero Black-Box AI:** Every transformation is strictly mathematical (IQR, medians, mode). It is 100% deterministic, explainable, and safe for enterprise production environments.
82
+ - **Lightning Fast:** By utilizing `polars` (written in Rust) instead of `pandas`, DATADOC processes millions of rows with minimal memory overhead.
83
+ - **Agentic AI Integration:** DATADOC features a built-in AI Planner and an interactive Chat Assistant that can autonomously analyze your dataset, generate engineering plans, and execute plugins using tool-calling!
84
+ - **Avoid Data Leakage:** Built-in safeguards ensure that data scaling and imputation are handled correctly.
85
+
86
+ ---
87
+
88
+ ## 📦 Installation
89
+
90
+ DATADOC is published on PyPI. You can install it globally via `pip` or `uv`:
91
+
92
+ ```bash
93
+ pip install datadoc-cli
94
+ ```
95
+
96
+ *(Requires Python 3.9+)*
97
+
98
+ ---
99
+
100
+ ## 🛠️ Quick Start (CLI)
101
+
102
+ You don't need to write a single line of Python to clean your data. Just use the CLI.
103
+
104
+ ```bash
105
+ # 1. Analyze your dataset's health (shows a beautiful terminal report)
106
+ datadoc analyze raw_data.csv
107
+
108
+ # 2. Get recommendations (DATADOC tells you exactly what is wrong)
109
+ datadoc recommend raw_data.csv
110
+
111
+ # 3. AUTO-ENGINEER! (Fixes everything and saves clean_raw_data.csv)
112
+ datadoc engineer raw_data.csv
113
+
114
+ # 4. Compare the before vs. after visually in your terminal
115
+ datadoc compare raw_data.csv clean_raw_data.csv
116
+
117
+ # 5. Export a standalone Python script to automate this in the future
118
+ datadoc pipeline raw_data.csv
119
+
120
+ # 6. Have an interactive AI session where the LLM engineers your data via chat!
121
+ datadoc chat raw_data.csv
122
+
123
+ *(Pro Tip: Add `--ai` to `analyze`, `recommend`, or `engineer` for AI-driven insights and orchestration!)*
124
+ ```
125
+
126
+ ---
127
+
128
+ ## 🐍 Python SDK (Library Usage)
129
+
130
+ DATADOC is also a powerful Python library. You can import the engine directly into your Jupyter Notebooks or backend servers:
131
+
132
+ ```python
133
+ from datadoc.core.engine import DATADOC
134
+
135
+ # Initialize the blazing-fast Polars engine
136
+ doc = DATADOC("raw_data.csv")
137
+
138
+ # Generate a diagnostic report
139
+ report = doc.analyze()
140
+ print(report)
141
+
142
+ # Automatically engineer the dataset
143
+ clean_df = doc.engineer()
144
+ clean_df.write_csv("clean_data.csv")
145
+
146
+ # Export the generated pipeline script
147
+ with open("my_pipeline.py", "w") as f:
148
+ f.write(doc.pipeline())
149
+ ```
150
+
151
+ ---
152
+
153
+ ## 💻 CLI Commands Reference
154
+
155
+ | Command | Description |
156
+ |---------|-------------|
157
+ | `datadoc analyze <file>` | Scans dataset and shows a health report with status indicators |
158
+ | `datadoc recommend <file>` | Lists suggested engineering steps without modifying data |
159
+ | `datadoc engineer <file>` | Automatically applies all recommended transformations |
160
+ | `datadoc chat <file>` | Starts an interactive AI session with autonomous tool-calling |
161
+ | `datadoc compare <file>` | Shows a before/after diff of the raw vs engineered dataset |
162
+ | `datadoc pipeline <file>` | Exports a standalone `.py` script with the exact Polars code |
163
+ | `datadoc visualize <file>` | Renders stunning terminal-based charts for numeric distributions |
164
+ | `datadoc plugin` | Lists all registered plugins with priority and descriptions |
165
+ | `datadoc version` | Displays the DATADOC version |
166
+
167
+ *(Note: Use the `--ai` flag on `analyze`, `recommend`, or `engineer` for LLM-powered execution!)*
168
+
169
+ ---
170
+
171
+ ## 🧩 Architecture & Plugins
172
+
173
+ DATADOC operates as an orchestrator. It passes your dataset through an isolated chain of plugins in a strict priority order.
174
+
175
+ | Priority | Plugin | Action Performed |
176
+ |----------|--------|-------------|
177
+ | 10 | **MissingValuePlugin** | Imputes missing numeric values with median, categorical with mode |
178
+ | 20 | **OutlierPlugin** | Detects outliers via IQR and clips them dynamically |
179
+ | 30 | **DatetimePlugin** | Detects date strings and extracts year, month, day, day_of_week |
180
+ | 40 | **CategoricalEncoderPlugin** | One-Hot Encodes categorical columns (< 10 unique values) |
181
+ | 45 | **ScalingPlugin** | Standard scales numeric columns when scale ratio exceeds 10x |
182
+
183
+ Every plugin implements a strict `BasePlugin` interface ensuring it can `analyze()`, `apply()`, `rollback()`, and `generate_code()`.
184
+
185
+ Want to build your own? See [CONTRIBUTING.md](CONTRIBUTING.md) to learn how to create and register custom plugins!
186
+
187
+ ---
188
+
189
+ ## 🗺️ Roadmap
190
+
191
+ - [x] Core Engine with plugin orchestration
192
+ - [x] 5 Built-in deterministic plugins
193
+ - [x] Stunning Rich Terminal UI
194
+ - [x] Pipeline export capability
195
+ - [x] **Polars Backend Migration (100x Performance Boost)**
196
+ - [x] PyPI Release (`pip install datadoc-cli`)
197
+ - [x] Phase 2: Agentic AI Planner (LLM Orchestration)
198
+ - [x] Interactive AI Chat with Tool-Calling capabilities
199
+ - [ ] Export targets for `dbt` and Apache Airflow
200
+ - [ ] REST API (FastAPI) wrapper
201
+
202
+ ---
203
+
204
+ ## ⚖️ License
205
+
206
+ This project is licensed under the MIT License. See [LICENSE](LICENSE) for details.
207
+
208
+ ## 🤝 Contributing
209
+
210
+ We welcome contributions from the community! If you'd like to add a new plugin or improve the core engine, please see [CONTRIBUTING.md](CONTRIBUTING.md).
@@ -0,0 +1,176 @@
1
+ <p align="center">
2
+ <pre align="center">
3
+ ____ _ _____ _ ____ ___ ____
4
+ | _ \ / \|_ _|/ \ | _ \ / _ \ / ___|
5
+ | | | |/ _ \ | | / _ \ | | | | | | | |
6
+ | |_| / ___ \| |/ ___ \| |_| | |_| | |___
7
+ |____/_/ \_\_/_/ \_\____/ \___/ \____|
8
+ </pre>
9
+ </p>
10
+
11
+ <h3 align="center">The Open Source Operating System for Dataset Engineering.</h3>
12
+
13
+ <p align="center">
14
+ <a href="https://narain-karti.github.io/DATADOC/"><b>&#x1F4D6; View Official Documentation Website</b></a>
15
+ </p>
16
+
17
+ <p align="center">
18
+ <a href="https://pypi.org/project/datadoc-cli/"><img alt="PyPI version" src="https://img.shields.io/pypi/v/datadoc-cli.svg"></a>
19
+ <a href="https://pypi.org/project/datadoc-cli/"><img alt="Python Versions" src="https://img.shields.io/pypi/pyversions/datadoc-cli.svg"></a>
20
+ <a href="https://github.com/narain-karti/DATADOC/blob/main/LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/License-MIT-yellow.svg"></a>
21
+ </p>
22
+
23
+ <p align="center">
24
+ <a href="#installation"><b>Install</b></a> &bull;
25
+ <a href="#why-datadoc"><b>Why DATADOC?</b></a> &bull;
26
+ <a href="#quick-start"><b>Quick Start</b></a> &bull;
27
+ <a href="#cli-commands"><b>CLI Commands</b></a> &bull;
28
+ <a href="#architecture--plugins"><b>Architecture</b></a>
29
+ </p>
30
+
31
+ <hr>
32
+
33
+ ## 🚀 What is DATADOC?
34
+
35
+ **DATADOC** is an intelligent, blazing-fast Command Line Interface (CLI) and Python Library designed to completely automate the most tedious part of Machine Learning: **Dataset Engineering and Data Cleaning**.
36
+
37
+ Powered by a high-performance **Polars** backend, DATADOC analyzes your raw CSV files, diagnoses missing values, outliers, and schema issues, and **automatically engineers a machine-learning-ready dataset in milliseconds**.
38
+
39
+ **DATADOC is NOT just another EDA (Exploratory Data Analysis) tool.** It doesn't just show you charts. It **fixes your data** and hands you a portable, deterministic Python script to replicate the pipeline anywhere.
40
+
41
+ ### ⚡ The Impact: Why Industry Professionals Use DATADOC
42
+
43
+ Data Scientists and ML Engineers spend **80% of their time cleaning data** and only 20% training models.
44
+ DATADOC eliminates the 80%.
45
+
46
+ - **Save Hundreds of Hours:** Stop writing boilerplate code to impute nulls, one-hot encode categorical variables, or clip outliers. DATADOC does it in one command.
47
+ - **Zero Black-Box AI:** Every transformation is strictly mathematical (IQR, medians, mode). It is 100% deterministic, explainable, and safe for enterprise production environments.
48
+ - **Lightning Fast:** By utilizing `polars` (written in Rust) instead of `pandas`, DATADOC processes millions of rows with minimal memory overhead.
49
+ - **Agentic AI Integration:** DATADOC features a built-in AI Planner and an interactive Chat Assistant that can autonomously analyze your dataset, generate engineering plans, and execute plugins using tool-calling!
50
+ - **Avoid Data Leakage:** Built-in safeguards ensure that data scaling and imputation are handled correctly.
51
+
52
+ ---
53
+
54
+ ## 📦 Installation
55
+
56
+ DATADOC is published on PyPI. You can install it globally via `pip` or `uv`:
57
+
58
+ ```bash
59
+ pip install datadoc-cli
60
+ ```
61
+
62
+ *(Requires Python 3.9+)*
63
+
64
+ ---
65
+
66
+ ## 🛠️ Quick Start (CLI)
67
+
68
+ You don't need to write a single line of Python to clean your data. Just use the CLI.
69
+
70
+ ```bash
71
+ # 1. Analyze your dataset's health (shows a beautiful terminal report)
72
+ datadoc analyze raw_data.csv
73
+
74
+ # 2. Get recommendations (DATADOC tells you exactly what is wrong)
75
+ datadoc recommend raw_data.csv
76
+
77
+ # 3. AUTO-ENGINEER! (Fixes everything and saves clean_raw_data.csv)
78
+ datadoc engineer raw_data.csv
79
+
80
+ # 4. Compare the before vs. after visually in your terminal
81
+ datadoc compare raw_data.csv clean_raw_data.csv
82
+
83
+ # 5. Export a standalone Python script to automate this in the future
84
+ datadoc pipeline raw_data.csv
85
+
86
+ # 6. Have an interactive AI session where the LLM engineers your data via chat!
87
+ datadoc chat raw_data.csv
88
+
89
+ *(Pro Tip: Add `--ai` to `analyze`, `recommend`, or `engineer` for AI-driven insights and orchestration!)*
90
+ ```
91
+
92
+ ---
93
+
94
+ ## 🐍 Python SDK (Library Usage)
95
+
96
+ DATADOC is also a powerful Python library. You can import the engine directly into your Jupyter Notebooks or backend servers:
97
+
98
+ ```python
99
+ from datadoc.core.engine import DATADOC
100
+
101
+ # Initialize the blazing-fast Polars engine
102
+ doc = DATADOC("raw_data.csv")
103
+
104
+ # Generate a diagnostic report
105
+ report = doc.analyze()
106
+ print(report)
107
+
108
+ # Automatically engineer the dataset
109
+ clean_df = doc.engineer()
110
+ clean_df.write_csv("clean_data.csv")
111
+
112
+ # Export the generated pipeline script
113
+ with open("my_pipeline.py", "w") as f:
114
+ f.write(doc.pipeline())
115
+ ```
116
+
117
+ ---
118
+
119
+ ## 💻 CLI Commands Reference
120
+
121
+ | Command | Description |
122
+ |---------|-------------|
123
+ | `datadoc analyze <file>` | Scans dataset and shows a health report with status indicators |
124
+ | `datadoc recommend <file>` | Lists suggested engineering steps without modifying data |
125
+ | `datadoc engineer <file>` | Automatically applies all recommended transformations |
126
+ | `datadoc chat <file>` | Starts an interactive AI session with autonomous tool-calling |
127
+ | `datadoc compare <file>` | Shows a before/after diff of the raw vs engineered dataset |
128
+ | `datadoc pipeline <file>` | Exports a standalone `.py` script with the exact Polars code |
129
+ | `datadoc visualize <file>` | Renders stunning terminal-based charts for numeric distributions |
130
+ | `datadoc plugin` | Lists all registered plugins with priority and descriptions |
131
+ | `datadoc version` | Displays the DATADOC version |
132
+
133
+ *(Note: Use the `--ai` flag on `analyze`, `recommend`, or `engineer` for LLM-powered execution!)*
134
+
135
+ ---
136
+
137
+ ## 🧩 Architecture & Plugins
138
+
139
+ DATADOC operates as an orchestrator. It passes your dataset through an isolated chain of plugins in a strict priority order.
140
+
141
+ | Priority | Plugin | Action Performed |
142
+ |----------|--------|-------------|
143
+ | 10 | **MissingValuePlugin** | Imputes missing numeric values with median, categorical with mode |
144
+ | 20 | **OutlierPlugin** | Detects outliers via IQR and clips them dynamically |
145
+ | 30 | **DatetimePlugin** | Detects date strings and extracts year, month, day, day_of_week |
146
+ | 40 | **CategoricalEncoderPlugin** | One-Hot Encodes categorical columns (< 10 unique values) |
147
+ | 45 | **ScalingPlugin** | Standard scales numeric columns when scale ratio exceeds 10x |
148
+
149
+ Every plugin implements a strict `BasePlugin` interface ensuring it can `analyze()`, `apply()`, `rollback()`, and `generate_code()`.
150
+
151
+ Want to build your own? See [CONTRIBUTING.md](CONTRIBUTING.md) to learn how to create and register custom plugins!
152
+
153
+ ---
154
+
155
+ ## 🗺️ Roadmap
156
+
157
+ - [x] Core Engine with plugin orchestration
158
+ - [x] 5 Built-in deterministic plugins
159
+ - [x] Stunning Rich Terminal UI
160
+ - [x] Pipeline export capability
161
+ - [x] **Polars Backend Migration (100x Performance Boost)**
162
+ - [x] PyPI Release (`pip install datadoc-cli`)
163
+ - [x] Phase 2: Agentic AI Planner (LLM Orchestration)
164
+ - [x] Interactive AI Chat with Tool-Calling capabilities
165
+ - [ ] Export targets for `dbt` and Apache Airflow
166
+ - [ ] REST API (FastAPI) wrapper
167
+
168
+ ---
169
+
170
+ ## ⚖️ License
171
+
172
+ This project is licensed under the MIT License. See [LICENSE](LICENSE) for details.
173
+
174
+ ## 🤝 Contributing
175
+
176
+ We welcome contributions from the community! If you'd like to add a new plugin or improve the core engine, please see [CONTRIBUTING.md](CONTRIBUTING.md).
@@ -0,0 +1,11 @@
1
+ ID,Name,Age,Income,Score,City,Membership_Level,Signup_Date
2
+ 1,Alice,25,50000,0.85,New York,Silver,2023-01-15 10:00:00
3
+ 2,Bob,30,60000,0.90,London,Gold,2023-02-20 14:30:00
4
+ 3,Charlie,,55000,0.75,,Bronze,2023-03-05 09:15:00
5
+ 4,David,45,10000000,0.92,Paris,Gold,2023-04-10 16:45:00
6
+ 5,Eve,28,45000,0.88,New York,Silver,2023-05-12 11:20:00
7
+ 6,Frank,35,70000,,London,Silver,2023-06-18 08:00:00
8
+ 7,Grace,40,-500000,0.95,New York,Bronze,2023-07-22 13:10:00
9
+ 8,Heidi,32,65000,0.82,Paris,Gold,2023-08-30 17:55:00
10
+ 9,Ivan,29,52000,0.78,London,,2023-09-14 10:40:00
11
+ 10,Judy,38,75000,0.89,New York,Bronze,2023-10-01 15:25:00