datadoc-cli 0.1.0__tar.gz

datadoc_cli-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,52 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  test:
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        os: [ubuntu-latest, windows-latest, macos-latest]
+        python-version: ["3.9", "3.10", "3.11", "3.12"]
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e ".[dev]"
+
+      - name: Run tests
+        run: pytest tests/ -v --tb=short
+
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install ruff
+
+      - name: Run Ruff linter
+        run: ruff check datadoc/
+
+      - name: Run Ruff formatter check
+        run: ruff format --check datadoc/

datadoc_cli-0.1.0/.gitignore

@@ -0,0 +1,33 @@
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.egg-info/
+dist/
+build/
+*.egg
+
+# Virtual environments
+.venv/
+venv/
+env/
+
+# IDE
+.vscode/
+.idea/
+*.swp
+*.swo
+
+# OS
+.DS_Store
+Thumbs.db
+
+# DATADOC generated outputs
+clean_*.csv
+pipeline_*.py
+report_*.md
+
+# Testing
+.pytest_cache/
+htmlcov/
+.coverage

datadoc_cli-0.1.0/CONTRIBUTING.md

@@ -0,0 +1,98 @@
+# Contributing to DATADOC
+
+Thank you for considering contributing to DATADOC! This guide will help you get started.
+
+## Development Setup
+
+```bash
+# Clone the repository
+git clone https://github.com/narain-karti/DATADOC.git
+cd DATADOC
+
+# Install in development mode with dev dependencies
+pip install -e ".[dev]"
+
+# Run the test suite
+pytest tests/ -v
+```
+
+## How to Build a Plugin
+
+DATADOC's entire architecture is built around plugins. Every feature engineering operation is an isolated plugin that inherits from `BasePlugin`.
+
+### Step 1: Create a new file
+
+Create `datadoc/plugins/your_plugin.py`:
+
+```python
+import pandas as pd
+from datadoc.plugins.base import BasePlugin
+
+class YourPlugin(BasePlugin):
+    @property
+    def name(self) -> str:
+        return "YourPlugin"
+
+    @property
+    def version(self) -> str:
+        return "0.1.0"
+
+    @property
+    def description(self) -> str:
+        return "What your plugin does in one sentence."
+
+    @property
+    def priority(self) -> int:
+        return 50  # Lower = runs first
+
+    def analyze(self, df: pd.DataFrame) -> dict:
+        # Detect if this plugin should run
+        return {"has_issue": True}
+
+    def recommend(self, analysis_result: dict) -> list[str]:
+        if analysis_result.get("has_issue"):
+            return ["Description of what should be done."]
+        return []
+
+    def generate_code(self, analysis_result: dict) -> str:
+        if not analysis_result.get("has_issue"):
+            return ""
+        return "# Your pandas code here\ndf['new_col'] = df['old_col'] * 2"
+
+    def apply(self, df: pd.DataFrame) -> pd.DataFrame:
+        df_clean = df.copy()
+        # Your transformation logic
+        return df_clean
+```
+
+### Step 2: Register the plugin
+
+Add your plugin to `datadoc/core/engine.py` in the `self.plugins` list inside `__init__`.
+
+### Step 3: Update the CLI
+
+Add handling for your plugin's analysis output in the `analyze` command in `datadoc/cli/app.py`.
+
+### Step 4: Write tests
+
+Add tests in `tests/test_core.py` to verify your plugin's `analyze()`, `apply()`, and `recommend()` methods.
+
+## Code Style
+
+- We use [Ruff](https://docs.astral.sh/ruff/) for linting and formatting.
+- Run `ruff check datadoc/` before submitting.
+- Target Python 3.9+.
+
+## Submitting Changes
+
+1. Fork the repository.
+2. Create a feature branch: `git checkout -b feature/your-plugin`
+3. Make your changes and add tests.
+4. Run the full test suite: `pytest tests/ -v`
+5. Submit a pull request.
+
+## Labels
+
+- `good first issue` - Great for newcomers
+- `plugin-idea` - Propose a new plugin
+- `core-engine` - Changes to the core orchestrator

datadoc_cli-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 narain-karti
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

datadoc_cli-0.1.0/PKG-INFO

@@ -0,0 +1,254 @@
+Metadata-Version: 2.4
+Name: datadoc-cli
+Version: 0.1.0
+Summary: The Open Source Operating System for Dataset Engineering.
+Project-URL: Homepage, https://github.com/narain-karti/DATADOC
+Project-URL: Repository, https://github.com/narain-karti/DATADOC
+Project-URL: Issues, https://github.com/narain-karti/DATADOC/issues
+Author: narain-karti
+License-Expression: MIT
+License-File: LICENSE
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering
+Requires-Python: >=3.9
+Requires-Dist: numpy>=1.24.0
+Requires-Dist: plotext>=5.2.8
+Requires-Dist: polars>=0.20.0
+Requires-Dist: rich>=13.7.0
+Requires-Dist: typer>=0.12.0
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0.0; extra == 'dev'
+Requires-Dist: ruff>=0.4.0; extra == 'dev'
+Description-Content-Type: text/markdown
+
+<p align="center">
+  <pre>
+ ____    _  _____  _    ____   ___   ____
+|  _ \  / \|_   _|/ \  |  _ \ / _ \ / ___|
+| | | |/ _ \ | | / _ \ | | | | | | | |
+| |_| / ___ \| |/ ___ \| |_| | |_| | |___
+|____/_/   \_\_/_/   \_\____/ \___/ \____|
+  </pre>
+</p>
+
+<h3 align="center">The Open Source Operating System for Dataset Engineering.</h3>
+
+<p align="center">
+  <a href="#installation">Install</a> |
+  <a href="#quick-start">Quick Start</a> |
+  <a href="#cli-commands">CLI Commands</a> |
+  <a href="#plugins">Plugins</a> |
+  <a href="#contributing">Contributing</a>
+</p>
+
+---
+
+## What is DATADOC?
+
+DATADOC is an intelligent, modular **Dataset Engineering Framework** that orchestrates existing powerful libraries (Pandas, NumPy, Scikit-learn) into a standardized, reusable workflow.
+
+**DATADOC is NOT another EDA tool.** It doesn't just show you charts. It **diagnoses**, **recommends**, and **automatically engineers** your dataset -- then hands you a portable Python script to replicate it anywhere.
+
+### Core Philosophy
+
+- **Orchestrate, don't replace** -- DATADOC wraps Pandas, NumPy, and Sklearn. It doesn't reinvent them.
+- **Modular Plugin Architecture** -- Every transformation is an isolated, testable plugin.
+- **Explainable by Default** -- Every action can be explained, rolled back, and exported as code.
+- **Deterministic First** -- v1.0 uses a rule-based engine. No black-box AI decisions.
+
+---
+
+## Installation
+
+```bash
+# Clone and install
+git clone https://github.com/narain-karti/DATADOC.git
+cd DATADOC
+pip install -e .
+
+# Or install with dev dependencies
+pip install -e ".[dev]"
+```
+
+---
+
+## Quick Start
+
+```bash
+# Analyze your dataset's health
+datadoc analyze your_data.csv
+
+# Get recommendations without changing anything
+datadoc recommend your_data.csv
+
+# Automatically engineer your dataset
+datadoc engineer your_data.csv
+
+# Compare before vs after
+datadoc compare your_data.csv
+
+# Export a standalone Python pipeline
+datadoc pipeline your_data.csv
+
+# Generate a Markdown report
+datadoc report your_data.csv
+
+# List all available plugins
+datadoc plugin
+```
+
+### Python SDK
+
+```python
+from datadoc.core.engine import DATADOC
+
+doc = DATADOC("your_data.csv")
+
+# Analyze
+report = doc.analyze()
+print(report)
+
+# Get recommendations
+for rec in doc.recommend():
+    print(rec)
+
+# Auto-engineer
+clean_df = doc.engineer()
+clean_df.to_csv("clean_data.csv", index=False)
+
+# Export pipeline
+with open("pipeline.py", "w") as f:
+    f.write(doc.pipeline())
+```
+
+---
+
+## CLI Commands
+
+| Command | Description |
+|---------|-------------|
+| `datadoc analyze <file>` | Scans dataset and shows a health report with status indicators |
+| `datadoc recommend <file>` | Lists suggested engineering steps without modifying data |
+| `datadoc engineer <file>` | Automatically applies all recommended transformations |
+| `datadoc compare <file>` | Shows a before/after diff of the raw vs engineered dataset |
+| `datadoc pipeline <file>` | Exports a standalone `.py` script with the exact Pandas code |
+| `datadoc report <file>` | Generates a full Markdown report |
+| `datadoc plugin` | Lists all registered plugins with priority and descriptions |
+| `datadoc version` | Displays the DATADOC version |
+
+---
+
+## Plugins
+
+DATADOC ships with 5 built-in plugins, executed in priority order:
+
+| Priority | Plugin | What it Does |
+|----------|--------|-------------|
+| 10 | **MissingValuePlugin** | Imputes missing numeric values with median, categorical with mode |
+| 20 | **OutlierPlugin** | Detects outliers via IQR and clips at 5th/95th percentiles |
+| 30 | **DatetimePlugin** | Detects date columns and extracts year, month, day, day_of_week |
+| 40 | **CategoricalEncoderPlugin** | One-Hot Encodes categorical columns (< 10 unique values) |
+| 45 | **ScalingPlugin** | Standard scales numeric columns when scale ratio exceeds 10x |
+
+### Plugin Interface
+
+Every plugin implements the full `BasePlugin` interface:
+
+```python
+class BasePlugin(ABC):
+    name: str              # Plugin identifier
+    version: str           # Semantic version
+    description: str       # One-line description
+    priority: int          # Execution order (lower = first)
+    supported_datatypes    # Which dtypes this plugin handles
+    dependencies           # Plugins that must run before this one
+
+    analyze(df) -> dict           # Detect issues
+    recommend(analysis) -> list   # Suggest fixes
+    generate_code(analysis) -> str # Export as Python code
+    apply(df) -> DataFrame        # Apply transformation
+    validate(df) -> bool          # Verify result
+    rollback(original) -> DataFrame # Undo transformation
+    explain() -> str              # Human-readable explanation
+    estimate_runtime(df) -> float # Performance estimate
+```
+
+### Building Your Own Plugin
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for a step-by-step guide on building and registering custom plugins.
+
+---
+
+## Architecture
+
+```
+datadoc/
+  core/
+    engine.py          # DATADOC orchestrator class
+  cli/
+    app.py             # Typer CLI with Rich terminal UI
+  plugins/
+    base.py            # Abstract BasePlugin interface
+    missing_values.py  # MissingValuePlugin
+    outliers.py        # OutlierPlugin
+    datetime_feat.py   # DatetimePlugin
+    encoders.py        # CategoricalEncoderPlugin
+    scaling.py         # ScalingPlugin
+tests/
+  test_core.py         # Comprehensive test suite
+```
+
+The CLI talks to the Core Engine, which orchestrates Plugins sorted by priority. Each plugin independently analyzes, recommends, and applies transformations.
+
+---
+
+## Development
+
+```bash
+# Install dev dependencies
+pip install -e ".[dev]"
+
+# Run tests
+pytest tests/ -v
+
+# Lint
+ruff check datadoc/
+```
+
+---
+
+## Roadmap
+
+- [x] Core Engine with plugin orchestration
+- [x] 5 built-in plugins (Missing Values, Outliers, Datetime, Encoding, Scaling)
+- [x] 7 CLI commands with Rich terminal UI
+- [x] Pipeline export (standalone `.py` scripts)
+- [x] Markdown report generation
+- [x] Full plugin interface (validate, rollback, explain, estimate_runtime)
+- [x] CI/CD with GitHub Actions
+- [ ] PyPI release (`pip install datadoc`)
+- [ ] Polars backend support
+- [ ] REST API (FastAPI)
+- [ ] Web Dashboard
+- [ ] AI Planner (LLM replaces Rule Engine)
+- [ ] Plugin Marketplace
+
+---
+
+## License
+
+MIT License. See [LICENSE](LICENSE) for details.
+
+---
+
+## Contributing
+
+We welcome contributions! See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.

datadoc_cli-0.1.0/README.md

@@ -0,0 +1,223 @@
+<p align="center">
+  <pre>
+ ____    _  _____  _    ____   ___   ____
+|  _ \  / \|_   _|/ \  |  _ \ / _ \ / ___|
+| | | |/ _ \ | | / _ \ | | | | | | | |
+| |_| / ___ \| |/ ___ \| |_| | |_| | |___
+|____/_/   \_\_/_/   \_\____/ \___/ \____|
+  </pre>
+</p>
+
+<h3 align="center">The Open Source Operating System for Dataset Engineering.</h3>
+
+<p align="center">
+  <a href="#installation">Install</a> |
+  <a href="#quick-start">Quick Start</a> |
+  <a href="#cli-commands">CLI Commands</a> |
+  <a href="#plugins">Plugins</a> |
+  <a href="#contributing">Contributing</a>
+</p>
+
+---
+
+## What is DATADOC?
+
+DATADOC is an intelligent, modular **Dataset Engineering Framework** that orchestrates existing powerful libraries (Pandas, NumPy, Scikit-learn) into a standardized, reusable workflow.
+
+**DATADOC is NOT another EDA tool.** It doesn't just show you charts. It **diagnoses**, **recommends**, and **automatically engineers** your dataset -- then hands you a portable Python script to replicate it anywhere.
+
+### Core Philosophy
+
+- **Orchestrate, don't replace** -- DATADOC wraps Pandas, NumPy, and Sklearn. It doesn't reinvent them.
+- **Modular Plugin Architecture** -- Every transformation is an isolated, testable plugin.
+- **Explainable by Default** -- Every action can be explained, rolled back, and exported as code.
+- **Deterministic First** -- v1.0 uses a rule-based engine. No black-box AI decisions.
+
+---
+
+## Installation
+
+```bash
+# Clone and install
+git clone https://github.com/narain-karti/DATADOC.git
+cd DATADOC
+pip install -e .
+
+# Or install with dev dependencies
+pip install -e ".[dev]"
+```
+
+---
+
+## Quick Start
+
+```bash
+# Analyze your dataset's health
+datadoc analyze your_data.csv
+
+# Get recommendations without changing anything
+datadoc recommend your_data.csv
+
+# Automatically engineer your dataset
+datadoc engineer your_data.csv
+
+# Compare before vs after
+datadoc compare your_data.csv
+
+# Export a standalone Python pipeline
+datadoc pipeline your_data.csv
+
+# Generate a Markdown report
+datadoc report your_data.csv
+
+# List all available plugins
+datadoc plugin
+```
+
+### Python SDK
+
+```python
+from datadoc.core.engine import DATADOC
+
+doc = DATADOC("your_data.csv")
+
+# Analyze
+report = doc.analyze()
+print(report)
+
+# Get recommendations
+for rec in doc.recommend():
+    print(rec)
+
+# Auto-engineer
+clean_df = doc.engineer()
+clean_df.to_csv("clean_data.csv", index=False)
+
+# Export pipeline
+with open("pipeline.py", "w") as f:
+    f.write(doc.pipeline())
+```
+
+---
+
+## CLI Commands
+
+| Command | Description |
+|---------|-------------|
+| `datadoc analyze <file>` | Scans dataset and shows a health report with status indicators |
+| `datadoc recommend <file>` | Lists suggested engineering steps without modifying data |
+| `datadoc engineer <file>` | Automatically applies all recommended transformations |
+| `datadoc compare <file>` | Shows a before/after diff of the raw vs engineered dataset |
+| `datadoc pipeline <file>` | Exports a standalone `.py` script with the exact Pandas code |
+| `datadoc report <file>` | Generates a full Markdown report |
+| `datadoc plugin` | Lists all registered plugins with priority and descriptions |
+| `datadoc version` | Displays the DATADOC version |
+
+---
+
+## Plugins
+
+DATADOC ships with 5 built-in plugins, executed in priority order:
+
+| Priority | Plugin | What it Does |
+|----------|--------|-------------|
+| 10 | **MissingValuePlugin** | Imputes missing numeric values with median, categorical with mode |
+| 20 | **OutlierPlugin** | Detects outliers via IQR and clips at 5th/95th percentiles |
+| 30 | **DatetimePlugin** | Detects date columns and extracts year, month, day, day_of_week |
+| 40 | **CategoricalEncoderPlugin** | One-Hot Encodes categorical columns (< 10 unique values) |
+| 45 | **ScalingPlugin** | Standard scales numeric columns when scale ratio exceeds 10x |
+
+### Plugin Interface
+
+Every plugin implements the full `BasePlugin` interface:
+
+```python
+class BasePlugin(ABC):
+    name: str              # Plugin identifier
+    version: str           # Semantic version
+    description: str       # One-line description
+    priority: int          # Execution order (lower = first)
+    supported_datatypes    # Which dtypes this plugin handles
+    dependencies           # Plugins that must run before this one
+
+    analyze(df) -> dict           # Detect issues
+    recommend(analysis) -> list   # Suggest fixes
+    generate_code(analysis) -> str # Export as Python code
+    apply(df) -> DataFrame        # Apply transformation
+    validate(df) -> bool          # Verify result
+    rollback(original) -> DataFrame # Undo transformation
+    explain() -> str              # Human-readable explanation
+    estimate_runtime(df) -> float # Performance estimate
+```
+
+### Building Your Own Plugin
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for a step-by-step guide on building and registering custom plugins.
+
+---
+
+## Architecture
+
+```
+datadoc/
+  core/
+    engine.py          # DATADOC orchestrator class
+  cli/
+    app.py             # Typer CLI with Rich terminal UI
+  plugins/
+    base.py            # Abstract BasePlugin interface
+    missing_values.py  # MissingValuePlugin
+    outliers.py        # OutlierPlugin
+    datetime_feat.py   # DatetimePlugin
+    encoders.py        # CategoricalEncoderPlugin
+    scaling.py         # ScalingPlugin
+tests/
+  test_core.py         # Comprehensive test suite
+```
+
+The CLI talks to the Core Engine, which orchestrates Plugins sorted by priority. Each plugin independently analyzes, recommends, and applies transformations.
+
+---
+
+## Development
+
+```bash
+# Install dev dependencies
+pip install -e ".[dev]"
+
+# Run tests
+pytest tests/ -v
+
+# Lint
+ruff check datadoc/
+```
+
+---
+
+## Roadmap
+
+- [x] Core Engine with plugin orchestration
+- [x] 5 built-in plugins (Missing Values, Outliers, Datetime, Encoding, Scaling)
+- [x] 7 CLI commands with Rich terminal UI
+- [x] Pipeline export (standalone `.py` scripts)
+- [x] Markdown report generation
+- [x] Full plugin interface (validate, rollback, explain, estimate_runtime)
+- [x] CI/CD with GitHub Actions
+- [ ] PyPI release (`pip install datadoc`)
+- [ ] Polars backend support
+- [ ] REST API (FastAPI)
+- [ ] Web Dashboard
+- [ ] AI Planner (LLM replaces Rule Engine)
+- [ ] Plugin Marketplace
+
+---
+
+## License
+
+MIT License. See [LICENSE](LICENSE) for details.
+
+---
+
+## Contributing
+
+We welcome contributions! See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.