datadoc-cli 0.2.0__tar.gz → 0.4.0__tar.gz

{datadoc_cli-0.2.0 → datadoc_cli-0.4.0}/.gitignore

@@ -26,9 +26,11 @@ Thumbs.db
 clean_*.csv
 pipeline_*.py
 report_*.md
+.release-verify/
 
 # Testing
 .pytest_cache/
+.playwright-cli/
 htmlcov/
 .coverage
 

datadoc_cli-0.4.0/FULL_DOCUMENTATION.md

@@ -0,0 +1,216 @@
+# DATADOC: Complete User and Developer Guide
+
+DATADOC is a local-first open-source toolkit for preparing tabular datasets for supervised machine learning. It profiles a dataset, explains possible data-quality problems, creates a transformation plan, fits that plan only on training data, and reuses the fitted rules for validation, test, and inference data.
+
+The central idea is simple:
+
+```text
+profile → plan → fit on train → transform validation/test/new data → evaluate
+```
+
+DATADOC does not promise that cleaning always improves model performance. When a target is supplied and the optional ML dependency is installed, it measures the result against a reproducible baseline and can honestly report that no improvement was found.
+
+For the implementation explanation, read [PROJECT_IMPLEMENTATION_GUIDE.md](PROJECT_IMPLEMENTATION_GUIDE.md). For migration from the original mutable API, read [MIGRATION.md](MIGRATION.md). The visual documentation site is in [`docs/`](docs/) and begins at [`docs/index.html`](docs/index.html).
+
+For maintainers, [RELEASE_CHECKLIST.md](RELEASE_CHECKLIST.md) describes the CI, artifact, Git tag, and PyPI upload steps.
+
+## 1. Installation
+
+The core package is intentionally small and works without an AI provider, web server, or scikit-learn:
+
+```bash
+pip install datadoc-cli
+```
+
+Optional capabilities are installed only when needed:
+
+```bash
+pip install "datadoc-cli[ml]"   # model benchmarking
+pip install "datadoc-cli[ui]"   # local FastAPI dashboard
+pip install "datadoc-cli[ai]"   # constrained advisory AI planner
+pip install "datadoc-cli[dev]"  # contributor tooling
+```
+
+DATADOC supports Python 3.10, 3.11, and 3.12. Input and output formats are CSV and Parquet.
+
+## 2. The mental model
+
+Imagine a dataset with a numeric `age`, a categorical `plan`, a timestamp, missing values, and a target called `churn`.
+
+1. `profile` looks at the columns and reports facts: null counts, cardinality, duplicates, constants, suspicious identifiers, parse confidence, and warnings.
+2. `plan` turns those facts into proposed operations. It does not modify the dataset.
+3. `fit` learns reusable values from training data: numeric medians, category vocabularies, missing-value indicators, outlier bounds when enabled, and scaling centers/spreads.
+4. `transform` applies those frozen values to another dataset. It never recalculates them from validation, test, or inference rows.
+5. `evaluate` can compare a minimal baseline with a candidate pipeline for a declared classification or regression task.
+
+This separation prevents data leakage. For example, a validation row cannot change the median used to fill a training feature, and a category appearing only in validation cannot enlarge the training feature schema.
+
+## 3. The recommended CLI workflow
+
+### Step 1: profile
+
+```bash
+datadoc profile data.csv --target churn --output profile.json
+```
+
+This is read-only. A declared target is protected and is never treated as a feature. The output contains schema, null counts, cardinality, confidence-scored roles, findings, duplicate-row count, and a schema fingerprint.
+
+### Step 2: review a plan
+
+```bash
+datadoc plan data.csv --target churn --task classification --output plan.json
+```
+
+The plan lists operations such as numeric median imputation, categorical missing-value handling, one-hot encoding, optional missingness indicators, and optional scaling. Identifier suggestions are findings; they are not silently dropped.
+
+### Step 3: fit on training data
+
+```bash
+datadoc fit train.csv \
+  --target churn \
+  --task classification \
+  --output artifacts/churn-pipeline.json
+```
+
+Only `train.csv` is used to learn pipeline state. The JSON artifact includes the input schema, output schema, configuration, profile, plan, artifact version, and fitted state.
+
+### Step 4: transform later data
+
+```bash
+datadoc transform validation.csv \
+  --pipeline artifacts/churn-pipeline.json \
+  --output validation-features.parquet
+
+datadoc transform test.csv \
+  --pipeline artifacts/churn-pipeline.json \
+  --output test-features.parquet
+```
+
+The pipeline checks compatibility before transforming. Missing required columns or incompatible artifact versions produce actionable errors. Unseen categories are handled without changing the output schema.
+
+### Step 5: evaluate when a target exists
+
+```bash
+pip install "datadoc-cli[ml]"
+datadoc evaluate train.csv \
+  --target churn \
+  --task classification \
+  --estimator linear \
+  --output evaluation.json
+```
+
+Classification uses balanced accuracy as the primary comparison. Regression uses RMSE. The report names the estimator family, metric, split strategy, baseline score, selected score, improvement, selected pipeline, feature count, and warnings. Use `--time-column` for ordered evaluation and `--group-column` when rows must stay grouped.
+
+For benchmarking, DATADOC passes numeric transformed features to the estimator. Retained text or identifier columns remain visible in the prepared output but are excluded from the model matrix with a warning. Any remaining numeric missing values are filled with medians learned from the current training fold, never from the validation fold.
+
+### One-command run
+
+```bash
+datadoc run data.csv \
+  --target churn \
+  --task classification \
+  --output-dir runs/churn-v1 \
+  --evaluate
+```
+
+This creates `profile.json`, `plan.json`, `pipeline.json`, `transformed.parquet`, and `manifest.json`; `evaluation.json` is added when evaluation is requested and the ML extra is available. For a production model decision, keep an untouched external test set and do not treat the convenience run as a replacement for a deliberate train/validation/test split.
+
+### Export code
+
+```bash
+datadoc export \
+  --pipeline artifacts/churn-pipeline.json \
+  --output pipeline.py
+```
+
+The export is a small executable wrapper that loads the same artifact and calls the same runtime implementation. It is intended for transparent deployment and review; it does not duplicate or reinterpret fitted statistics.
+
+## 4. Python SDK
+
+```python
+import polars as pl
+from datadoc import DataDocPipeline, PipelineConfig
+
+train = pl.read_csv("train.csv")
+validation = pl.read_csv("validation.csv")
+
+config = PipelineConfig(
+    target="churn",
+    task="classification",
+    scaling="standard",
+    clip_outliers=False,
+    random_seed=42,
+)
+
+pipeline = DataDocPipeline(config)
+profile = pipeline.profile(train)
+plan = pipeline.plan(train)
+pipeline.fit(train)
+
+validation_features = pipeline.transform(validation)
+pipeline.save("artifacts/churn-pipeline.json")
+
+loaded = DataDocPipeline.load("artifacts/churn-pipeline.json")
+same_features = loaded.transform(validation)
+assert same_features.columns == validation_features.columns
+
+# Optional model-oriented evidence:
+report = DataDocPipeline(config).evaluate(train, target="churn")
+print(report.to_dict())
+```
+
+Use `profile` and `plan` before `fit` when you want to inspect decisions. Use `load` in a separate inference process. The target remains in the transformed frame so callers can separate labels from features explicitly.
+
+## 5. What is protected and what is configurable?
+
+- A declared target is not imputed, encoded, scaled, clipped, or dropped as a feature.
+- Numeric missing values use a training median; categorical missing values use a configured marker.
+- Invalid numeric infinities become missing before imputation.
+- Missingness indicators can preserve signal carried by the fact that a value was absent.
+- One-hot vocabularies are learned on training data; unseen inference categories are safe.
+- High-cardinality strings are reported and can use frequency encoding; they are not silently deleted.
+- Outlier clipping is opt-in and is not applied to binary, target, identifier, or datetime-derived fields.
+- Scaling is configurable and defaults according to estimator family; tree-style models do not need it.
+- Date-like fields are parsed only when confidence is sufficient; parse warnings remain visible.
+- Numeric uniqueness alone never proves that a column is an identifier. Name and value evidence are combined.
+- Unsupported text, joins, images, geospatial features, and distributed processing are outside the stable tabular scope and are reported rather than guessed.
+
+## 6. Legacy compatibility
+
+The original `DATADOC(file).analyze()`, `.recommend()`, `.engineer()`, `.compare()`, `.visualize()`, `.report()`, and `.pipeline()` commands remain available as compatibility paths. New work should use `DataDocPipeline`.
+
+`engineer()` is deprecated for production use because fitting and transforming in one mutable operation makes train/test boundaries easy to misuse. It now warns that it cannot make a model-performance claim without an explicit target and validation setup. The old arbitrary-code execution path is not part of the production workflow.
+
+## 7. Local web UI
+
+Install the UI extra and start the dashboard with a local file:
+
+```bash
+pip install "datadoc-cli[ui]"
+datadoc ui data.csv
+```
+
+The UI consumes the same profile, plan, fit, transform-preview, and code-export concepts as the CLI. It uses session-scoped state, binds locally by default, restricts CORS to configured local origins, and does not execute AI-generated code. The frontend is in `web/src/App.jsx`; build it with `npm ci` and `npm run build` from `web/`.
+
+## 8. Open-source development
+
+Install contributor dependencies and run the checks:
+
+```bash
+pip install -e ".[dev,ml,ui]"
+pytest -q
+ruff check datadoc tests
+ruff format --check datadoc tests
+python -m compileall -q datadoc
+python -m build
+```
+
+Transformation contributions must prove four things: learned values come from training data only, the transform is stable for unseen or missing values, the artifact round-trips, and the generated wrapper agrees with runtime output. Prefer small focused pull requests, document behavior changes, and keep optional dependencies optional.
+
+## 9. Current boundaries and future work
+
+The stable product is tabular classification and regression preparation. Future work can add out-of-fold target encoding, richer repeated cross-validation, calibrated uncertainty, stronger artifact schemas, text and geospatial adapters, and distributed execution. Those additions should preserve the same safety contract rather than make automation more aggressive.
+
+## 10. License
+
+DATADOC is licensed under the MIT License. See [LICENSE](LICENSE).

{datadoc_cli-0.2.0 → datadoc_cli-0.4.0}/LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2024 narain-karti
+Copyright (c) 2026 narain-karti
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

datadoc_cli-0.4.0/MIGRATION.md

@@ -0,0 +1,16 @@
+# Migrating to DATADOC's fitted pipeline API
+
+Version 0.3 keeps the original `DATADOC` facade for compatibility, but new projects should use `DataDocPipeline`.
+
+```python
+from datadoc import DataDocPipeline, PipelineConfig
+
+pipeline = DataDocPipeline(PipelineConfig(target="label"))
+pipeline.fit(train_df)
+validation_features = pipeline.transform(validation_df)
+pipeline.save("pipeline.json")
+```
+
+`DATADOC.engineer()` fits and transforms the same file and is therefore only suitable for exploratory cleaning. It emits a deprecation warning for supervised work. Use `fit()` on a training split and `transform()` on validation, test, and inference data instead.
+
+The legacy AI code-execution path has been removed. AI planning, when installed with `datadoc-cli[ai]`, is limited to registered transformations and never executes provider-generated Python.

datadoc_cli-0.4.0/PKG-INFO

@@ -0,0 +1,217 @@
+Metadata-Version: 2.4
+Name: datadoc-cli
+Version: 0.4.0
+Summary: Leakage-safe, local-first preparation pipelines for tabular machine learning.
+Project-URL: Homepage, https://github.com/narain-karti/DATADOC
+Project-URL: Repository, https://github.com/narain-karti/DATADOC
+Project-URL: Issues, https://github.com/narain-karti/DATADOC/issues
+Author: narain-karti
+License-Expression: MIT
+License-File: LICENSE
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering
+Requires-Python: >=3.10
+Requires-Dist: numpy>=1.24.0
+Requires-Dist: plotext>=5.2.8
+Requires-Dist: polars>=0.20.0
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: python-dotenv>=1.0.0
+Requires-Dist: rich>=13.7.0
+Requires-Dist: typer>=0.12.0
+Provides-Extra: ai
+Requires-Dist: litellm>=1.40.0; extra == 'ai'
+Provides-Extra: dev
+Requires-Dist: build>=1.2.0; extra == 'dev'
+Requires-Dist: hatchling>=1.25.0; extra == 'dev'
+Requires-Dist: pytest>=8.0.0; extra == 'dev'
+Requires-Dist: ruff>=0.4.0; extra == 'dev'
+Requires-Dist: scikit-learn>=1.4.0; extra == 'dev'
+Provides-Extra: ml
+Requires-Dist: scikit-learn>=1.4.0; extra == 'ml'
+Provides-Extra: ui
+Requires-Dist: fastapi>=0.115.0; extra == 'ui'
+Requires-Dist: python-multipart>=0.0.20; extra == 'ui'
+Requires-Dist: uvicorn>=0.30.0; extra == 'ui'
+Description-Content-Type: text/markdown
+
+<p align="center">
+  <pre align="center">
+ ____    _  _____  _    ____   ___   ____
+|  _ \  / \|_   _|/ \  |  _ \ / _ \ / ___|
+| | | |/ _ \ | | / _ \ | | | | | | | |
+| |_| / ___ \| |/ ___ \| |_| | |_| | |___
+|____/_/   \_\_/_/   \_\____/ \___/ \____|
+  </pre>
+</p>
+
+<h3 align="center">The Open Source Operating System for Dataset Engineering.</h3>
+
+<p align="center">
+  <a href="https://narain-karti.github.io/DATADOC/"><b>&#x1F4D6; View Official Documentation Website</b></a>
+</p>
+
+<p align="center">
+  <a href="https://pypi.org/project/datadoc-cli/"><img alt="PyPI version" src="https://img.shields.io/pypi/v/datadoc-cli.svg"></a>
+  <a href="https://pypi.org/project/datadoc-cli/"><img alt="Python Versions" src="https://img.shields.io/pypi/pyversions/datadoc-cli.svg"></a>
+  <a href="https://github.com/narain-karti/DATADOC/blob/main/LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/License-MIT-yellow.svg"></a>
+</p>
+
+<p align="center">
+  <a href="#installation"><b>Install</b></a> &bull;
+  <a href="#why-datadoc"><b>Why DATADOC?</b></a> &bull;
+  <a href="#quick-start"><b>Quick Start</b></a> &bull;
+  <a href="#cli-commands"><b>CLI Commands</b></a> &bull;
+  <a href="#architecture--plugins"><b>Architecture</b></a>
+</p>
+
+<hr>
+
+## 🚀 What is DATADOC?
+
+**DATADOC** is a local-first CLI and Python library for preparing tabular data for machine learning. It profiles dataset risks, creates explainable transformation plans, and saves fitted pipelines that apply the same training-derived rules to validation, test, and inference data.
+
+Powered by **Polars**, DATADOC reads CSV and Parquet files, diagnoses missing values, identifiers, schema issues, duplicates, constants, and unsafe feature types. It does not promise model improvement: optional evaluation reports the observed result against a baseline under a reproducible split.
+
+**DATADOC is NOT just another EDA (Exploratory Data Analysis) tool.** It profiles data quality, lets you review a plan, fits transformations from training data, and hands you a portable artifact and Python wrapper for reuse.
+
+### ⚡ The Impact: Why Industry Professionals Use DATADOC
+
+Data Scientists and ML Engineers repeatedly rebuild the same preparation steps across projects.
+DATADOC turns those steps into a reviewable, reusable pipeline.
+
+- **Save boilerplate:** Review recommendations for imputing nulls, encoding categories, and optional scaling or clipping before applying them.
+- **Explainable by default:** The deterministic core records roles, findings, operations, protected columns, and fitted statistics in an inspectable artifact.
+- **Local-first:** The core package works offline. Optional ML, UI, and AI features are separate extras.
+- **Optional AI planning:** AI can help explain or rank a constrained plan; it is never allowed to execute arbitrary generated code.
+- **Leakage-safe workflows:** Fitted statistics for imputation, categorical vocabularies, clipping, and scaling are learned from training data and saved as an artifact.
+
+---
+
+## 📦 Installation
+
+DATADOC is published on PyPI. You can install it globally via `pip` or `uv`:
+
+```bash
+pip install datadoc-cli
+```
+
+*(Requires Python 3.10+)*
+
+---
+
+## 🛠️ Quick Start (CLI)
+
+You don't need to write a single line of Python to clean your data. Just use the CLI.
+
+```bash
+# 1. Inspect data-quality findings and column roles
+datadoc profile raw_data.csv --target churn --output profile.json
+
+# 2. Review the proposed transformations before applying them
+datadoc plan raw_data.csv --target churn --output plan.json
+
+# 3. Fit only on a training dataset, then save a reusable artifact
+datadoc fit train.csv --target churn --output artifacts/churn-pipeline.json
+
+# 4. Apply the fitted artifact to validation, test, or new data
+datadoc transform validation.csv --pipeline artifacts/churn-pipeline.json --output validation-features.parquet
+
+# 5. Optionally benchmark a safe candidate pipeline against a baseline
+pip install "datadoc-cli[ml]"
+datadoc evaluate train.csv --target churn --task classification
+
+# 6. Export a small executable wrapper around the fitted artifact
+datadoc export --pipeline artifacts/churn-pipeline.json --output pipeline.py
+
+
+```
+
+---
+
+## 🐍 Python SDK (Library Usage)
+
+DATADOC is also a Python library. The stable workflow is `profile → plan → fit → transform`; the same fitted artifact can be used in notebooks, services, and batch jobs:
+
+```python
+from datadoc import DataDocPipeline, PipelineConfig
+import polars as pl
+
+# Fit only on the training split. The target is protected from feature transforms.
+train_df = pl.read_csv("train.csv")
+pipeline = DataDocPipeline(PipelineConfig(target="churn")).fit(train_df)
+pipeline.save("artifacts/churn-pipeline.json")
+
+# Transform data that was never used to fit statistics.
+validation_df = pl.read_csv("validation.csv")
+validation_features = pipeline.transform(validation_df)
+```
+
+For an observed model comparison, install the optional ML extra and call `pipeline.evaluate(train_df)` or `datadoc evaluate`. Evaluation is evidence for the declared task and split strategy; it is not a promise that cleaning always improves a model.
+
+---
+
+## 💻 CLI Commands Reference
+
+| Command | Description |
+|---------|-------------|
+| `datadoc profile <file>` | Produces a data-quality report and confidence-scored column roles |
+| `datadoc plan <file>` | Outputs an explainable transformation plan without modifying data |
+| `datadoc fit <train>` | Learns a pipeline only from training data and saves JSON state |
+| `datadoc transform <file>` | Applies a saved pipeline to validation, test, or inference data |
+| `datadoc evaluate <file>` | Optionally compares a candidate pipeline with a baseline using leakage-aware splits |
+| `datadoc export` | Creates an executable wrapper for a saved pipeline artifact |
+| `datadoc run <file>` | Writes a profile, plan, artifact, transformed data, and lineage manifest |
+
+
+
+---
+
+## 🧩 Architecture & Plugins
+
+DATADOC operates as a fitted pipeline. Every transformation learns state only from training data, saves that state to JSON, and reuses it unchanged for later datasets.
+
+| Priority | Plugin | Action Performed |
+|----------|--------|-------------|
+| 10 | **MissingValuePlugin** | Imputes missing numeric values with median, categorical with mode |
+| 20 | **OutlierPlugin** | Offers optional IQR clipping; clipping is not forced by default |
+| 30 | **DatetimePlugin** | Detects date strings and extracts year, month, day, day_of_week |
+| 40 | **CategoricalEncoderPlugin** | Encodes categories using training vocabularies and handles unseen values |
+| 45 | **ScalingPlugin** | Applies configured standard or robust scaling, fit on training data only |
+
+The fitted pipeline is the production source of truth. Plugin work should follow the lifecycle `analyze → fit → transform → validate → export_code`, with all learned state serializable and testable.
+
+Want to build your own? See [CONTRIBUTING.md](CONTRIBUTING.md) to learn how to create and register custom plugins!
+
+---
+
+## 🗺️ Roadmap
+
+- [x] Core Engine with plugin orchestration
+- [x] 5 Built-in deterministic plugins
+- [x] Stunning Rich Terminal UI
+- [x] Pipeline export capability
+- [x] Polars backend and local-first pipeline artifacts
+- [x] PyPI Release (`pip install datadoc-cli`)
+- [x] Constrained optional AI planning path
+- [x] Session-scoped local FastAPI dashboard
+- [ ] Export targets for `dbt` and Apache Airflow
+- [x] Local FastAPI dashboard/API companion
+
+---
+
+## ⚖️ License
+
+This project is licensed under the MIT License. See [LICENSE](LICENSE) for details.
+
+## 🤝 Contributing
+
+We welcome contributions from the community! If you'd like to add a new plugin or improve the core engine, please see [CONTRIBUTING.md](CONTRIBUTING.md).
+
+Maintainers can use the [0.4.0 release checklist](RELEASE_CHECKLIST.md) when preparing a tag and PyPI upload.