datadoc-cli 0.3.0__tar.gz → 0.6.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- datadoc_cli-0.6.0/.gitignore +65 -0
- datadoc_cli-0.6.0/CHANGELOG.md +80 -0
- datadoc_cli-0.6.0/MIGRATION.md +16 -0
- datadoc_cli-0.6.0/PKG-INFO +286 -0
- datadoc_cli-0.6.0/README.md +238 -0
- datadoc_cli-0.6.0/RELEASE_CHECKLIST.md +74 -0
- datadoc_cli-0.6.0/datadoc/__init__.py +5 -0
- datadoc_cli-0.6.0/datadoc/cli/app.py +1367 -0
- datadoc_cli-0.6.0/datadoc/cli/ui_server.py +300 -0
- datadoc_cli-0.6.0/datadoc/core/__init__.py +25 -0
- datadoc_cli-0.6.0/datadoc/core/compare.py +428 -0
- datadoc_cli-0.6.0/datadoc/core/pipeline.py +1244 -0
- datadoc_cli-0.6.0/datadoc/core/report.py +668 -0
- {datadoc_cli-0.3.0 → datadoc_cli-0.6.0}/datadoc/plugins/base.py +1 -23
- {datadoc_cli-0.3.0 → datadoc_cli-0.6.0}/datadoc/plugins/datetime_feat.py +16 -37
- datadoc_cli-0.6.0/datadoc/plugins/duplicates.py +51 -0
- {datadoc_cli-0.3.0 → datadoc_cli-0.6.0}/datadoc/plugins/encoders.py +13 -25
- {datadoc_cli-0.3.0 → datadoc_cli-0.6.0}/datadoc/plugins/missing_values.py +4 -16
- {datadoc_cli-0.3.0 → datadoc_cli-0.6.0}/datadoc/plugins/outliers.py +9 -20
- datadoc_cli-0.6.0/datadoc/plugins/polynomial.py +111 -0
- datadoc_cli-0.6.0/datadoc/plugins/rare.py +95 -0
- datadoc_cli-0.6.0/datadoc/plugins/registry.py +97 -0
- {datadoc_cli-0.3.0 → datadoc_cli-0.6.0}/datadoc/plugins/scaling.py +16 -24
- datadoc_cli-0.6.0/datadoc/plugins/target_encoder.py +156 -0
- datadoc_cli-0.6.0/pyproject.toml +125 -0
- datadoc_cli-0.6.0/web/README.md +23 -0
- datadoc_cli-0.6.0/web/dist/assets/index-BPDnt264.js +17 -0
- datadoc_cli-0.6.0/web/dist/assets/index-C_j7uPW0.css +2 -0
- datadoc_cli-0.6.0/web/dist/favicon.svg +1 -0
- datadoc_cli-0.6.0/web/dist/icons.svg +24 -0
- datadoc_cli-0.6.0/web/dist/index.html +17 -0
- datadoc_cli-0.6.0/web/node_modules/@babel/runtime/LICENSE +22 -0
- datadoc_cli-0.6.0/web/node_modules/@babel/runtime/README.md +19 -0
- datadoc_cli-0.6.0/web/node_modules/@emnapi/core/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/@emnapi/core/README.md +1 -0
- datadoc_cli-0.6.0/web/node_modules/@emnapi/runtime/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/@emnapi/runtime/README.md +1 -0
- datadoc_cli-0.6.0/web/node_modules/@emnapi/wasi-threads/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/@emnapi/wasi-threads/README.md +203 -0
- datadoc_cli-0.6.0/web/node_modules/@hello-pangea/dnd/LICENSE +15 -0
- datadoc_cli-0.6.0/web/node_modules/@hello-pangea/dnd/README.md +172 -0
- datadoc_cli-0.6.0/web/node_modules/@jridgewell/gen-mapping/LICENSE +19 -0
- datadoc_cli-0.6.0/web/node_modules/@jridgewell/gen-mapping/README.md +227 -0
- datadoc_cli-0.6.0/web/node_modules/@jridgewell/remapping/LICENSE +19 -0
- datadoc_cli-0.6.0/web/node_modules/@jridgewell/remapping/README.md +218 -0
- datadoc_cli-0.6.0/web/node_modules/@jridgewell/resolve-uri/LICENSE +19 -0
- datadoc_cli-0.6.0/web/node_modules/@jridgewell/resolve-uri/README.md +40 -0
- datadoc_cli-0.6.0/web/node_modules/@jridgewell/sourcemap-codec/LICENSE +19 -0
- datadoc_cli-0.6.0/web/node_modules/@jridgewell/sourcemap-codec/README.md +264 -0
- datadoc_cli-0.6.0/web/node_modules/@jridgewell/trace-mapping/LICENSE +19 -0
- datadoc_cli-0.6.0/web/node_modules/@jridgewell/trace-mapping/README.md +348 -0
- datadoc_cli-0.6.0/web/node_modules/@oxc-project/types/LICENSE +22 -0
- datadoc_cli-0.6.0/web/node_modules/@oxc-project/types/README.md +3 -0
- datadoc_cli-0.6.0/web/node_modules/@oxlint/binding-win32-x64-msvc/README.md +3 -0
- datadoc_cli-0.6.0/web/node_modules/@reduxjs/toolkit/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/@reduxjs/toolkit/README.md +112 -0
- datadoc_cli-0.6.0/web/node_modules/@rolldown/binding-win32-x64-msvc/README.md +3 -0
- datadoc_cli-0.6.0/web/node_modules/@rolldown/pluginutils/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/@rolldown/pluginutils/README.md +145 -0
- datadoc_cli-0.6.0/web/node_modules/@standard-schema/spec/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/@standard-schema/spec/README.md +198 -0
- datadoc_cli-0.6.0/web/node_modules/@standard-schema/utils/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/@standard-schema/utils/README.md +61 -0
- datadoc_cli-0.6.0/web/node_modules/@tailwindcss/node/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/@tailwindcss/node/README.md +36 -0
- datadoc_cli-0.6.0/web/node_modules/@tailwindcss/node/node_modules/lightningcss/LICENSE +373 -0
- datadoc_cli-0.6.0/web/node_modules/@tailwindcss/node/node_modules/lightningcss/README.md +105 -0
- datadoc_cli-0.6.0/web/node_modules/@tailwindcss/node/node_modules/lightningcss-win32-x64-msvc/LICENSE +373 -0
- datadoc_cli-0.6.0/web/node_modules/@tailwindcss/node/node_modules/lightningcss-win32-x64-msvc/README.md +1 -0
- datadoc_cli-0.6.0/web/node_modules/@tailwindcss/oxide/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/@tailwindcss/oxide-win32-x64-msvc/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/@tailwindcss/oxide-win32-x64-msvc/README.md +3 -0
- datadoc_cli-0.6.0/web/node_modules/@tailwindcss/vite/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/@tailwindcss/vite/README.md +76 -0
- datadoc_cli-0.6.0/web/node_modules/@tybys/wasm-util/README.md +193 -0
- datadoc_cli-0.6.0/web/node_modules/@types/d3-array/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/@types/d3-array/README.md +15 -0
- datadoc_cli-0.6.0/web/node_modules/@types/d3-color/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/@types/d3-color/README.md +15 -0
- datadoc_cli-0.6.0/web/node_modules/@types/d3-ease/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/@types/d3-ease/README.md +15 -0
- datadoc_cli-0.6.0/web/node_modules/@types/d3-interpolate/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/@types/d3-interpolate/README.md +15 -0
- datadoc_cli-0.6.0/web/node_modules/@types/d3-path/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/@types/d3-path/README.md +15 -0
- datadoc_cli-0.6.0/web/node_modules/@types/d3-scale/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/@types/d3-scale/README.md +15 -0
- datadoc_cli-0.6.0/web/node_modules/@types/d3-shape/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/@types/d3-shape/README.md +15 -0
- datadoc_cli-0.6.0/web/node_modules/@types/d3-time/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/@types/d3-time/README.md +15 -0
- datadoc_cli-0.6.0/web/node_modules/@types/d3-timer/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/@types/d3-timer/README.md +15 -0
- datadoc_cli-0.6.0/web/node_modules/@types/react/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/@types/react/README.md +15 -0
- datadoc_cli-0.6.0/web/node_modules/@types/react-dom/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/@types/react-dom/README.md +16 -0
- datadoc_cli-0.6.0/web/node_modules/@types/use-sync-external-store/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/@types/use-sync-external-store/README.md +15 -0
- datadoc_cli-0.6.0/web/node_modules/@vitejs/plugin-react/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/@vitejs/plugin-react/README.md +169 -0
- datadoc_cli-0.6.0/web/node_modules/agent-base/README.md +145 -0
- datadoc_cli-0.6.0/web/node_modules/asynckit/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/asynckit/README.md +233 -0
- datadoc_cli-0.6.0/web/node_modules/axios/CHANGELOG.md +1872 -0
- datadoc_cli-0.6.0/web/node_modules/axios/LICENSE +7 -0
- datadoc_cli-0.6.0/web/node_modules/axios/README.md +2559 -0
- datadoc_cli-0.6.0/web/node_modules/axios/lib/adapters/README.md +36 -0
- datadoc_cli-0.6.0/web/node_modules/axios/lib/core/README.md +8 -0
- datadoc_cli-0.6.0/web/node_modules/axios/lib/helpers/README.md +7 -0
- datadoc_cli-0.6.0/web/node_modules/call-bind-apply-helpers/CHANGELOG.md +30 -0
- datadoc_cli-0.6.0/web/node_modules/call-bind-apply-helpers/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/call-bind-apply-helpers/README.md +62 -0
- datadoc_cli-0.6.0/web/node_modules/css-box-model/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/css-box-model/README.md +290 -0
- datadoc_cli-0.6.0/web/node_modules/csstype/LICENSE +19 -0
- datadoc_cli-0.6.0/web/node_modules/csstype/README.md +291 -0
- datadoc_cli-0.6.0/web/node_modules/d3-array/LICENSE +13 -0
- datadoc_cli-0.6.0/web/node_modules/d3-array/README.md +954 -0
- datadoc_cli-0.6.0/web/node_modules/d3-color/LICENSE +13 -0
- datadoc_cli-0.6.0/web/node_modules/d3-color/README.md +203 -0
- datadoc_cli-0.6.0/web/node_modules/d3-ease/LICENSE +28 -0
- datadoc_cli-0.6.0/web/node_modules/d3-ease/README.md +253 -0
- datadoc_cli-0.6.0/web/node_modules/d3-format/LICENSE +13 -0
- datadoc_cli-0.6.0/web/node_modules/d3-format/README.md +12 -0
- datadoc_cli-0.6.0/web/node_modules/d3-interpolate/LICENSE +13 -0
- datadoc_cli-0.6.0/web/node_modules/d3-interpolate/README.md +268 -0
- datadoc_cli-0.6.0/web/node_modules/d3-path/LICENSE +13 -0
- datadoc_cli-0.6.0/web/node_modules/d3-path/README.md +94 -0
- datadoc_cli-0.6.0/web/node_modules/d3-scale/LICENSE +13 -0
- datadoc_cli-0.6.0/web/node_modules/d3-scale/README.md +1003 -0
- datadoc_cli-0.6.0/web/node_modules/d3-shape/LICENSE +13 -0
- datadoc_cli-0.6.0/web/node_modules/d3-shape/README.md +1227 -0
- datadoc_cli-0.6.0/web/node_modules/d3-time/LICENSE +13 -0
- datadoc_cli-0.6.0/web/node_modules/d3-time/README.md +380 -0
- datadoc_cli-0.6.0/web/node_modules/d3-time-format/LICENSE +13 -0
- datadoc_cli-0.6.0/web/node_modules/d3-time-format/README.md +209 -0
- datadoc_cli-0.6.0/web/node_modules/d3-timer/LICENSE +13 -0
- datadoc_cli-0.6.0/web/node_modules/d3-timer/README.md +87 -0
- datadoc_cli-0.6.0/web/node_modules/debug/LICENSE +20 -0
- datadoc_cli-0.6.0/web/node_modules/debug/README.md +481 -0
- datadoc_cli-0.6.0/web/node_modules/decimal.js-light/CHANGELOG.md +87 -0
- datadoc_cli-0.6.0/web/node_modules/decimal.js-light/README.md +92 -0
- datadoc_cli-0.6.0/web/node_modules/detect-libc/LICENSE +201 -0
- datadoc_cli-0.6.0/web/node_modules/detect-libc/README.md +163 -0
- datadoc_cli-0.6.0/web/node_modules/dunder-proto/CHANGELOG.md +24 -0
- datadoc_cli-0.6.0/web/node_modules/dunder-proto/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/dunder-proto/README.md +54 -0
- datadoc_cli-0.6.0/web/node_modules/enhanced-resolve/LICENSE +20 -0
- datadoc_cli-0.6.0/web/node_modules/enhanced-resolve/README.md +862 -0
- datadoc_cli-0.6.0/web/node_modules/es-define-property/CHANGELOG.md +29 -0
- datadoc_cli-0.6.0/web/node_modules/es-define-property/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/es-define-property/README.md +49 -0
- datadoc_cli-0.6.0/web/node_modules/es-errors/CHANGELOG.md +40 -0
- datadoc_cli-0.6.0/web/node_modules/es-errors/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/es-errors/README.md +55 -0
- datadoc_cli-0.6.0/web/node_modules/es-object-atoms/CHANGELOG.md +44 -0
- datadoc_cli-0.6.0/web/node_modules/es-object-atoms/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/es-object-atoms/README.md +63 -0
- datadoc_cli-0.6.0/web/node_modules/es-set-tostringtag/CHANGELOG.md +67 -0
- datadoc_cli-0.6.0/web/node_modules/es-set-tostringtag/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/es-set-tostringtag/README.md +53 -0
- datadoc_cli-0.6.0/web/node_modules/es-toolkit/CHANGELOG.md +936 -0
- datadoc_cli-0.6.0/web/node_modules/es-toolkit/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/es-toolkit/README.md +69 -0
- datadoc_cli-0.6.0/web/node_modules/eventemitter3/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/eventemitter3/README.md +85 -0
- datadoc_cli-0.6.0/web/node_modules/fdir/LICENSE +7 -0
- datadoc_cli-0.6.0/web/node_modules/fdir/README.md +91 -0
- datadoc_cli-0.6.0/web/node_modules/follow-redirects/LICENSE +18 -0
- datadoc_cli-0.6.0/web/node_modules/follow-redirects/README.md +157 -0
- datadoc_cli-0.6.0/web/node_modules/form-data/CHANGELOG.md +686 -0
- datadoc_cli-0.6.0/web/node_modules/form-data/README.md +355 -0
- datadoc_cli-0.6.0/web/node_modules/function-bind/CHANGELOG.md +136 -0
- datadoc_cli-0.6.0/web/node_modules/function-bind/LICENSE +20 -0
- datadoc_cli-0.6.0/web/node_modules/function-bind/README.md +46 -0
- datadoc_cli-0.6.0/web/node_modules/get-intrinsic/CHANGELOG.md +186 -0
- datadoc_cli-0.6.0/web/node_modules/get-intrinsic/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/get-intrinsic/README.md +71 -0
- datadoc_cli-0.6.0/web/node_modules/get-proto/CHANGELOG.md +21 -0
- datadoc_cli-0.6.0/web/node_modules/get-proto/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/get-proto/README.md +50 -0
- datadoc_cli-0.6.0/web/node_modules/gopd/CHANGELOG.md +45 -0
- datadoc_cli-0.6.0/web/node_modules/gopd/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/gopd/README.md +40 -0
- datadoc_cli-0.6.0/web/node_modules/graceful-fs/LICENSE +15 -0
- datadoc_cli-0.6.0/web/node_modules/graceful-fs/README.md +143 -0
- datadoc_cli-0.6.0/web/node_modules/has-symbols/CHANGELOG.md +91 -0
- datadoc_cli-0.6.0/web/node_modules/has-symbols/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/has-symbols/README.md +46 -0
- datadoc_cli-0.6.0/web/node_modules/has-tostringtag/CHANGELOG.md +42 -0
- datadoc_cli-0.6.0/web/node_modules/has-tostringtag/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/has-tostringtag/README.md +46 -0
- datadoc_cli-0.6.0/web/node_modules/hasown/CHANGELOG.md +58 -0
- datadoc_cli-0.6.0/web/node_modules/hasown/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/hasown/README.md +40 -0
- datadoc_cli-0.6.0/web/node_modules/https-proxy-agent/README.md +137 -0
- datadoc_cli-0.6.0/web/node_modules/immer/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/internmap/LICENSE +13 -0
- datadoc_cli-0.6.0/web/node_modules/internmap/README.md +94 -0
- datadoc_cli-0.6.0/web/node_modules/jiti/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/jiti/README.md +258 -0
- datadoc_cli-0.6.0/web/node_modules/lightningcss/LICENSE +373 -0
- datadoc_cli-0.6.0/web/node_modules/lightningcss/README.md +105 -0
- datadoc_cli-0.6.0/web/node_modules/lightningcss-win32-x64-msvc/LICENSE +373 -0
- datadoc_cli-0.6.0/web/node_modules/lightningcss-win32-x64-msvc/README.md +1 -0
- datadoc_cli-0.6.0/web/node_modules/lucide-react/LICENSE +43 -0
- datadoc_cli-0.6.0/web/node_modules/lucide-react/README.md +87 -0
- datadoc_cli-0.6.0/web/node_modules/magic-string/LICENSE +7 -0
- datadoc_cli-0.6.0/web/node_modules/magic-string/README.md +325 -0
- datadoc_cli-0.6.0/web/node_modules/math-intrinsics/CHANGELOG.md +24 -0
- datadoc_cli-0.6.0/web/node_modules/math-intrinsics/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/math-intrinsics/README.md +50 -0
- datadoc_cli-0.6.0/web/node_modules/mime-db/LICENSE +23 -0
- datadoc_cli-0.6.0/web/node_modules/mime-db/README.md +100 -0
- datadoc_cli-0.6.0/web/node_modules/mime-types/LICENSE +23 -0
- datadoc_cli-0.6.0/web/node_modules/mime-types/README.md +113 -0
- datadoc_cli-0.6.0/web/node_modules/nanoid/LICENSE +20 -0
- datadoc_cli-0.6.0/web/node_modules/nanoid/README.md +36 -0
- datadoc_cli-0.6.0/web/node_modules/oxlint/LICENSE +22 -0
- datadoc_cli-0.6.0/web/node_modules/oxlint/README.md +58 -0
- datadoc_cli-0.6.0/web/node_modules/picocolors/LICENSE +15 -0
- datadoc_cli-0.6.0/web/node_modules/picocolors/README.md +21 -0
- datadoc_cli-0.6.0/web/node_modules/picomatch/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/picomatch/README.md +743 -0
- datadoc_cli-0.6.0/web/node_modules/postcss/LICENSE +20 -0
- datadoc_cli-0.6.0/web/node_modules/postcss/README.md +24 -0
- datadoc_cli-0.6.0/web/node_modules/proxy-from-env/LICENSE +20 -0
- datadoc_cli-0.6.0/web/node_modules/proxy-from-env/README.md +163 -0
- datadoc_cli-0.6.0/web/node_modules/raf-schd/CHANGELOG.md +4 -0
- datadoc_cli-0.6.0/web/node_modules/raf-schd/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/raf-schd/README.md +154 -0
- datadoc_cli-0.6.0/web/node_modules/react/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/react/README.md +37 -0
- datadoc_cli-0.6.0/web/node_modules/react-dom/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/react-dom/README.md +60 -0
- datadoc_cli-0.6.0/web/node_modules/react-is/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/react-is/README.md +103 -0
- datadoc_cli-0.6.0/web/node_modules/react-redux/README.md +63 -0
- datadoc_cli-0.6.0/web/node_modules/recharts/CHANGELOG.md +1965 -0
- datadoc_cli-0.6.0/web/node_modules/recharts/LICENSE +22 -0
- datadoc_cli-0.6.0/web/node_modules/recharts/README.md +98 -0
- datadoc_cli-0.6.0/web/node_modules/redux/README.md +157 -0
- datadoc_cli-0.6.0/web/node_modules/redux-thunk/README.md +403 -0
- datadoc_cli-0.6.0/web/node_modules/reselect/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/reselect/README.md +206 -0
- datadoc_cli-0.6.0/web/node_modules/rolldown/LICENSE +25 -0
- datadoc_cli-0.6.0/web/node_modules/rolldown/README.md +11 -0
- datadoc_cli-0.6.0/web/node_modules/scheduler/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/scheduler/README.md +9 -0
- datadoc_cli-0.6.0/web/node_modules/source-map-js/LICENSE +28 -0
- datadoc_cli-0.6.0/web/node_modules/source-map-js/README.md +765 -0
- datadoc_cli-0.6.0/web/node_modules/tailwindcss/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/tailwindcss/README.md +36 -0
- datadoc_cli-0.6.0/web/node_modules/tapable/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/tapable/README.md +715 -0
- datadoc_cli-0.6.0/web/node_modules/tiny-invariant/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/tiny-invariant/README.md +109 -0
- datadoc_cli-0.6.0/web/node_modules/tinyglobby/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/tinyglobby/README.md +25 -0
- datadoc_cli-0.6.0/web/node_modules/tslib/README.md +164 -0
- datadoc_cli-0.6.0/web/node_modules/use-sync-external-store/LICENSE +21 -0
- datadoc_cli-0.6.0/web/node_modules/use-sync-external-store/README.md +5 -0
- datadoc_cli-0.6.0/web/node_modules/victory-vendor/CHANGELOG.md +117 -0
- datadoc_cli-0.6.0/web/node_modules/victory-vendor/README.md +70 -0
- datadoc_cli-0.6.0/web/node_modules/victory-vendor/lib-vendor/d3-array/LICENSE +13 -0
- datadoc_cli-0.6.0/web/node_modules/victory-vendor/lib-vendor/d3-color/LICENSE +13 -0
- datadoc_cli-0.6.0/web/node_modules/victory-vendor/lib-vendor/d3-ease/LICENSE +28 -0
- datadoc_cli-0.6.0/web/node_modules/victory-vendor/lib-vendor/d3-format/LICENSE +13 -0
- datadoc_cli-0.6.0/web/node_modules/victory-vendor/lib-vendor/d3-interpolate/LICENSE +13 -0
- datadoc_cli-0.6.0/web/node_modules/victory-vendor/lib-vendor/d3-path/LICENSE +13 -0
- datadoc_cli-0.6.0/web/node_modules/victory-vendor/lib-vendor/d3-scale/LICENSE +13 -0
- datadoc_cli-0.6.0/web/node_modules/victory-vendor/lib-vendor/d3-shape/LICENSE +13 -0
- datadoc_cli-0.6.0/web/node_modules/victory-vendor/lib-vendor/d3-time/LICENSE +13 -0
- datadoc_cli-0.6.0/web/node_modules/victory-vendor/lib-vendor/d3-time-format/LICENSE +13 -0
- datadoc_cli-0.6.0/web/node_modules/victory-vendor/lib-vendor/d3-timer/LICENSE +13 -0
- datadoc_cli-0.6.0/web/node_modules/victory-vendor/lib-vendor/d3-voronoi/LICENSE +50 -0
- datadoc_cli-0.6.0/web/node_modules/victory-vendor/lib-vendor/internmap/LICENSE +13 -0
- datadoc_cli-0.6.0/web/node_modules/vite/README.md +20 -0
- datadoc_cli-0.3.0/.github/workflows/ci.yml +0 -52
- datadoc_cli-0.3.0/.gitignore +0 -37
- datadoc_cli-0.3.0/CONTRIBUTING.md +0 -98
- datadoc_cli-0.3.0/FULL_DOCUMENTATION.md +0 -144
- datadoc_cli-0.3.0/PKG-INFO +0 -210
- datadoc_cli-0.3.0/README.md +0 -176
- datadoc_cli-0.3.0/comprehensive_test.csv +0 -11
- datadoc_cli-0.3.0/datadoc/cli/app.py +0 -805
- datadoc_cli-0.3.0/datadoc/core/__init__.py +0 -0
- datadoc_cli-0.3.0/datadoc/core/engine.py +0 -353
- datadoc_cli-0.3.0/datadoc/plugins/__init__.py +0 -0
- datadoc_cli-0.3.0/diagnose.py +0 -97
- datadoc_cli-0.3.0/diagnose_output.txt +0 -130
- datadoc_cli-0.3.0/diagnose_output2.txt +0 -76
- datadoc_cli-0.3.0/docs/api-docs.html +0 -140
- datadoc_cli-0.3.0/docs/contribute.html +0 -128
- datadoc_cli-0.3.0/docs/css/style.css +0 -522
- datadoc_cli-0.3.0/docs/how-it-works.html +0 -127
- datadoc_cli-0.3.0/docs/index.html +0 -259
- datadoc_cli-0.3.0/docs/license.html +0 -106
- datadoc_cli-0.3.0/docs/sdk.html +0 -137
- datadoc_cli-0.3.0/docs/setup.html +0 -160
- datadoc_cli-0.3.0/generate_heavy_test.py +0 -113
- datadoc_cli-0.3.0/heavy_test.csv +0 -10001
- datadoc_cli-0.3.0/heavy_test_output.txt +0 -52
- datadoc_cli-0.3.0/pyproject.toml +0 -61
- datadoc_cli-0.3.0/test.csv +0 -6
- datadoc_cli-0.3.0/test_output.txt +0 -7
- datadoc_cli-0.3.0/tests/__init__.py +0 -1
- datadoc_cli-0.3.0/tests/test_core.py +0 -390
- datadoc_cli-0.3.0/uv.lock +0 -3940
- {datadoc_cli-0.3.0 → datadoc_cli-0.6.0}/LICENSE +0 -0
- {datadoc_cli-0.3.0/datadoc → datadoc_cli-0.6.0/datadoc/cli}/__init__.py +0 -0
- {datadoc_cli-0.3.0/datadoc/cli → datadoc_cli-0.6.0/datadoc/plugins}/__init__.py +0 -0
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
# Python
|
|
2
|
+
__pycache__/
|
|
3
|
+
*.py[cod]
|
|
4
|
+
*$py.class
|
|
5
|
+
*.egg-info/
|
|
6
|
+
dist/
|
|
7
|
+
build/
|
|
8
|
+
*.egg
|
|
9
|
+
|
|
10
|
+
# Virtual environments
|
|
11
|
+
.venv/
|
|
12
|
+
venv/
|
|
13
|
+
env/
|
|
14
|
+
|
|
15
|
+
# IDE
|
|
16
|
+
.vscode/
|
|
17
|
+
.idea/
|
|
18
|
+
*.swp
|
|
19
|
+
*.swo
|
|
20
|
+
|
|
21
|
+
# OS
|
|
22
|
+
.DS_Store
|
|
23
|
+
Thumbs.db
|
|
24
|
+
|
|
25
|
+
# DATADOC generated outputs
|
|
26
|
+
clean_*.csv
|
|
27
|
+
pipeline_*.py
|
|
28
|
+
report_*.md
|
|
29
|
+
.release-verify/
|
|
30
|
+
datadoc-run/
|
|
31
|
+
full-run/
|
|
32
|
+
titanic-run/
|
|
33
|
+
telco-run/
|
|
34
|
+
datadoc.toml
|
|
35
|
+
|
|
36
|
+
# Local audit datasets (re-download from Kaggle; never ship)
|
|
37
|
+
# NOTE: demo.csv and docs/demo.csv are TRACKED fixtures and stay tracked.
|
|
38
|
+
/california_housing.csv
|
|
39
|
+
/telco_churn.csv
|
|
40
|
+
/titanic.csv
|
|
41
|
+
/titanic-audit/*.csv
|
|
42
|
+
/titanic-audit/*.joblib
|
|
43
|
+
/titanic-audit/*.parquet
|
|
44
|
+
/titanic-audit/*.txt
|
|
45
|
+
/titanic-audit/full-run/
|
|
46
|
+
/titanic-audit/wizard-run/
|
|
47
|
+
/titanic-test/
|
|
48
|
+
|
|
49
|
+
# Scratch / debug (dev-only, never ship)
|
|
50
|
+
scratch_*
|
|
51
|
+
diagnose*
|
|
52
|
+
changes.diff
|
|
53
|
+
*_test.csv
|
|
54
|
+
*_output*.txt
|
|
55
|
+
heavy_test.csv
|
|
56
|
+
|
|
57
|
+
# Testing
|
|
58
|
+
.pytest_cache/
|
|
59
|
+
.playwright-cli/
|
|
60
|
+
htmlcov/
|
|
61
|
+
.coverage
|
|
62
|
+
|
|
63
|
+
# Environment & Secrets
|
|
64
|
+
.env
|
|
65
|
+
.env.*
|
|
@@ -0,0 +1,80 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to DATADOC are documented here. Format follows Keep a Changelog and SemVer.
|
|
4
|
+
|
|
5
|
+
## [0.6.0] - 2026-09-13
|
|
6
|
+
### Added
|
|
7
|
+
- `datadoc report`: Automated, standalone, high-fidelity HTML report generation (`datadoc report <dataset> [--target] [--output] [--title] [--preset]`). 100% self-contained single-file HTML with embedded responsive CSS, health grade scoring (A+ to F), column role classification, distribution statistics, and transformation plan.
|
|
8
|
+
- `datadoc compare`: Visual side-by-side dataset comparison command (`datadoc compare <raw> <transformed> [--target] [--html] [--json]`). Computes dimension deltas, missing cell reduction (100% resolution tracking), column lineage/lifecycle (retained, dropped, engineered), and numeric distribution shifts (imputation and scaling effects) with color-coded terminal tables and interactive HTML export.
|
|
9
|
+
- `TargetEncoderPlugin` (Priority 41): Empirical Bayes smoothed target encoding for high-cardinality categoricals: `y_hat = (n * cat_mean + m * global_mean) / (n + m)` to prevent target leakage and dimension explosion.
|
|
10
|
+
- `PolynomialFeaturesPlugin` (Priority 44): Degree-2 interactions (`x1 * x2`) and squared features (`x^2`) for numeric columns to capture non-linear signal for linear models.
|
|
11
|
+
- Dashboard enhancements: Added "View HTML Report" direct action in the header, navigation sidebar, and artifact actions panel.
|
|
12
|
+
- UI Server API: Added `GET /api/dataset/report` (standalone HTML audit report) and `GET /api/dataset/compare` (side-by-side comparison metrics).
|
|
13
|
+
- Standalone execution: Added `if __name__ == "__main__": app()` to `datadoc.cli.app`.
|
|
14
|
+
|
|
15
|
+
## [0.5.0] - 2026-09-14
|
|
16
|
+
### Added
|
|
17
|
+
- `datadoc.toml` / `pyproject.toml [tool.datadoc]` config file support with flag overrides.
|
|
18
|
+
- `datadoc wizard` guided TUI for `target`, `preset`, `drop_identifiers`, `deduplicate`, `scaling`, `clip_outliers`, `rare_frequency`; writes `datadoc.toml` + run dir.
|
|
19
|
+
- Presets `--preset quick|balanced|linear|tree|time|robust` mapping to `scaling`, `clip_outliers`, `estimator_family`.
|
|
20
|
+
- Short aliases `-t` / `-o` / `-p` / `-f` for `--target` / `--output` / `--pipeline` / `--format`.
|
|
21
|
+
- Shell completion via Typer (`add_completion=True`) — `datadoc --install-completion`.
|
|
22
|
+
- `datadoc diff` for `profile`/`plan`/`pipeline` JSON comparison (incl. ops added/removed).
|
|
23
|
+
- `datadoc lint` leakage lint (target duplication, null target, infinities, duplicates, profile findings).
|
|
24
|
+
- `datadoc plugins list|show` registry introspection, `datadoc.plugins` entry-points auto-discovery.
|
|
25
|
+
- New deterministic plugins: `DuplicateRemoverPlugin` (P05), `RareCategoryPlugin` (P42); `CategoricalEncoderPlugin` threshold reconciled 10 → 20.
|
|
26
|
+
- Pipeline artifact v2 with `provenance` (schema/columns hash, rows, dedup count, version); v1 artifacts still load.
|
|
27
|
+
- `plan` now surfaces `deduplicate`, `outlier_clipping`, `scaling`, rare-grouping details; new `explain_plan()`.
|
|
28
|
+
- `fit`/`run` flags: `--deduplicate`, `--rare-frequency`, `--cyclical`, `--no-hour`.
|
|
29
|
+
- Repeatable `--identifier-column` / `--ignore-column` on `plan`/`fit`/`run` (also via `datadoc.toml` lists), reaching `PipelineConfig.identifier_columns` / `ignored_columns` which previously had no CLI path.
|
|
30
|
+
- `transform --validate` via `drift_report()` (schema + median-shift drift).
|
|
31
|
+
- `evaluate --ablation` via `evaluate_ablation()` (full / no_clip / no_scaling / minimal).
|
|
32
|
+
- Notebook widgets: `profile_to_html()` + `_repr_html_()` for Jupyter/marimo.
|
|
33
|
+
- `export --format python|joblib` via `export_sklearn_artifact()` (joblib needs `ml` extra).
|
|
34
|
+
- Dashboard: Ctrl+K/Cmd+K palette, lineage panel (`/api/pipeline/lineage`), drift endpoint, new config knobs.
|
|
35
|
+
- PyPI-ready wheel: `web/dist` force-included as `datadoc/_webui`, so `datadoc ui` serves the React app from a plain pip install (verified in a clean venv); `ui_server` falls back to the checkout path in dev.
|
|
36
|
+
- `tomli>=2.0.0` dependency on Python < 3.11 for real TOML parsing.
|
|
37
|
+
- New `RELEASE_CHECKLIST.md` (referenced by `docs/production.html`); `uv.lock` re-synced; `datadoc.__version__` added as the code-level version source.
|
|
38
|
+
### Changed
|
|
39
|
+
- `scikit-learn` moved back to optional `ml` extra for lighter core install.
|
|
40
|
+
- `PipelineConfig` docs clarified `resolved_scaling` and `plan` exposure.
|
|
41
|
+
- `run` now parities all `fit` flags.
|
|
42
|
+
### Fixed
|
|
43
|
+
- Removed `product_specification.md` (Retrod PMS orphan).
|
|
44
|
+
- Fixed `CONTRIBUTING.md` pandas/engine refs.
|
|
45
|
+
- `plan` no longer lets `datadoc.toml` silently override explicit `--drop-identifiers`/`--task` flags.
|
|
46
|
+
- `wizard` crashed when delegating to `run` (unpassed params hit raw `typer.Option` defaults); all params are now explicit.
|
|
47
|
+
- `diff`/`plan --diff` labels corrected to "Only in A/B" (were backwards "added/removed").
|
|
48
|
+
- `estimator_family="both"` now truly fits linear + tree and keeps the stronger validation score (was silently tree-only); `resolved_scaling` maps `both` → `standard`.
|
|
49
|
+
- Dashboard export template now emits an executable script (`__main__` + argparse) and restores `train_provenance_`.
|
|
50
|
+
- Commented `datadoc.toml` (including the file `datadoc init` itself writes) crashed config loading on Python 3.10 without `tomli` — the fallback parser now strips inline comments and parses lists/quoted strings.
|
|
51
|
+
- Fixed installed-dashboard path resolution (`datadoc/cli/_webui` → `datadoc/_webui`).
|
|
52
|
+
- `strict_schema` now exposed via CLI/config.
|
|
53
|
+
- CI re-added.
|
|
54
|
+
|
|
55
|
+
## [0.4.0] - 2026-08-09
|
|
56
|
+
### Added
|
|
57
|
+
- Leakage-safe `DataDocPipeline` (`fit`/`transform`/`save`/`load`/`evaluate`/`export_python`) in `datadoc/core/pipeline.py`.
|
|
58
|
+
- `PipelineConfig` with 17 knobs (`target`, `task`, `drop_identifiers`, `scaling`, `clip_outliers`, ...).
|
|
59
|
+
- FastAPI dashboard `datadoc ui` + `web/` Vite+React.
|
|
60
|
+
- Optional extras `ai` (litellm), `ml`, `ui`.
|
|
61
|
+
### Changed
|
|
62
|
+
- CLI narrowed to 7 commands `profile,plan,fit,transform,evaluate,export,run`; legacy `engine.py` removed.
|
|
63
|
+
- Polars as primary backend.
|
|
64
|
+
|
|
65
|
+
## [0.3.0] - 2026-07-28
|
|
66
|
+
### Added
|
|
67
|
+
- Agentic `agent.py` + `chat` command.
|
|
68
|
+
- FastAPI `ui` via `fastapi/uvicorn/multipart`.
|
|
69
|
+
|
|
70
|
+
## [0.2.0] - 2026-07-26
|
|
71
|
+
### Added
|
|
72
|
+
- Phase 2 AI planner `litellm`, `pydantic`, `dotenv`.
|
|
73
|
+
- `chat` + `engineer --ai`.
|
|
74
|
+
|
|
75
|
+
## [0.1.0] - 2026-07-26
|
|
76
|
+
- Initial scaffold `typer/rich/pandas` + `MissingValuePlugin`.
|
|
77
|
+
|
|
78
|
+
## Roadmap
|
|
79
|
+
- Export targets `dbt` / Airflow (see README).
|
|
80
|
+
- Text embeddings optional plugin.
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
# Migrating to DATADOC's fitted pipeline API
|
|
2
|
+
|
|
3
|
+
Version 0.3 keeps the original `DATADOC` facade for compatibility, but new projects should use `DataDocPipeline`.
|
|
4
|
+
|
|
5
|
+
```python
|
|
6
|
+
from datadoc import DataDocPipeline, PipelineConfig
|
|
7
|
+
|
|
8
|
+
pipeline = DataDocPipeline(PipelineConfig(target="label"))
|
|
9
|
+
pipeline.fit(train_df)
|
|
10
|
+
validation_features = pipeline.transform(validation_df)
|
|
11
|
+
pipeline.save("pipeline.json")
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
`DATADOC.engineer()` fits and transforms the same file and is therefore only suitable for exploratory cleaning. It emits a deprecation warning for supervised work. Use `fit()` on a training split and `transform()` on validation, test, and inference data instead.
|
|
15
|
+
|
|
16
|
+
The legacy AI code-execution path has been removed. AI planning, when installed with `datadoc-cli[ai]`, is limited to registered transformations and never executes provider-generated Python.
|
|
@@ -0,0 +1,286 @@
|
|
|
1
|
+
Metadata-Version: 2.3
|
|
2
|
+
Name: datadoc-cli
|
|
3
|
+
Version: 0.6.0
|
|
4
|
+
Summary: Leakage-safe, local-first preparation pipelines for tabular machine learning.
|
|
5
|
+
Project-URL: Homepage, https://github.com/narain-karti/DATADOC
|
|
6
|
+
Project-URL: Repository, https://github.com/narain-karti/DATADOC
|
|
7
|
+
Project-URL: Issues, https://github.com/narain-karti/DATADOC/issues
|
|
8
|
+
Author: narain-karti
|
|
9
|
+
License: MIT
|
|
10
|
+
Classifier: Development Status :: 3 - Alpha
|
|
11
|
+
Classifier: Intended Audience :: Developers
|
|
12
|
+
Classifier: Intended Audience :: Science/Research
|
|
13
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
14
|
+
Classifier: Programming Language :: Python :: 3
|
|
15
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
18
|
+
Classifier: Topic :: Scientific/Engineering
|
|
19
|
+
Requires-Python: >=3.10
|
|
20
|
+
Requires-Dist: plotext>=5.2.8
|
|
21
|
+
Requires-Dist: polars>=0.20.0
|
|
22
|
+
Requires-Dist: pydantic>=2.0.0
|
|
23
|
+
Requires-Dist: python-dotenv>=1.0.0
|
|
24
|
+
Requires-Dist: rich>=13.7.0
|
|
25
|
+
Requires-Dist: tomli>=2.0.0; python_version < '3.11'
|
|
26
|
+
Requires-Dist: typer>=0.12.0
|
|
27
|
+
Provides-Extra: ai
|
|
28
|
+
Requires-Dist: litellm>=1.40.0; extra == 'ai'
|
|
29
|
+
Provides-Extra: all
|
|
30
|
+
Requires-Dist: fastapi>=0.115.0; extra == 'all'
|
|
31
|
+
Requires-Dist: litellm>=1.40.0; extra == 'all'
|
|
32
|
+
Requires-Dist: python-multipart>=0.0.20; extra == 'all'
|
|
33
|
+
Requires-Dist: scikit-learn>=1.7.2; extra == 'all'
|
|
34
|
+
Requires-Dist: uvicorn>=0.30.0; extra == 'all'
|
|
35
|
+
Provides-Extra: dev
|
|
36
|
+
Requires-Dist: build>=1.2.0; extra == 'dev'
|
|
37
|
+
Requires-Dist: hatchling>=1.25.0; extra == 'dev'
|
|
38
|
+
Requires-Dist: pytest>=8.0.0; extra == 'dev'
|
|
39
|
+
Requires-Dist: ruff>=0.4.0; extra == 'dev'
|
|
40
|
+
Requires-Dist: scikit-learn>=1.7.2; extra == 'dev'
|
|
41
|
+
Provides-Extra: ml
|
|
42
|
+
Requires-Dist: scikit-learn>=1.7.2; extra == 'ml'
|
|
43
|
+
Provides-Extra: ui
|
|
44
|
+
Requires-Dist: fastapi>=0.115.0; extra == 'ui'
|
|
45
|
+
Requires-Dist: python-multipart>=0.0.20; extra == 'ui'
|
|
46
|
+
Requires-Dist: uvicorn>=0.30.0; extra == 'ui'
|
|
47
|
+
Description-Content-Type: text/markdown
|
|
48
|
+
|
|
49
|
+
<p align="center">
|
|
50
|
+
<pre align="center">
|
|
51
|
+
____ _ _____ _ ____ ___ ____
|
|
52
|
+
| _ \ / \|_ _|/ \ | _ \ / _ \ / ___|
|
|
53
|
+
| | | |/ _ \ | | / _ \ | | | | | | | |
|
|
54
|
+
| |_| / ___ \| |/ ___ \| |_| | |_| | |___
|
|
55
|
+
|____/_/ \_\_/_/ \_\____/ \___/ \____|
|
|
56
|
+
</pre>
|
|
57
|
+
</p>
|
|
58
|
+
|
|
59
|
+
<h3 align="center">The Open Source Operating System for Dataset Engineering.</h3>
|
|
60
|
+
|
|
61
|
+
<p align="center">
|
|
62
|
+
<a href="https://narain-karti.github.io/DATADOC/"><b>📖 View Official Documentation Website</b></a>
|
|
63
|
+
</p>
|
|
64
|
+
|
|
65
|
+
<p align="center">
|
|
66
|
+
<a href="https://pypi.org/project/datadoc-cli/"><img alt="PyPI version" src="https://img.shields.io/pypi/v/datadoc-cli.svg"></a>
|
|
67
|
+
<a href="https://pypi.org/project/datadoc-cli/"><img alt="Python Versions" src="https://img.shields.io/pypi/pyversions/datadoc-cli.svg"></a>
|
|
68
|
+
<a href="https://github.com/narain-karti/DATADOC/blob/main/LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/License-MIT-yellow.svg"></a>
|
|
69
|
+
</p>
|
|
70
|
+
|
|
71
|
+
<p align="center">
|
|
72
|
+
<a href="#installation"><b>Install</b></a> •
|
|
73
|
+
<a href="#why-datadoc"><b>Why DATADOC?</b></a> •
|
|
74
|
+
<a href="#quick-start"><b>Quick Start</b></a> •
|
|
75
|
+
<a href="#cli-commands"><b>CLI Commands</b></a> •
|
|
76
|
+
<a href="#architecture--plugins"><b>Architecture</b></a>
|
|
77
|
+
</p>
|
|
78
|
+
|
|
79
|
+
<hr>
|
|
80
|
+
|
|
81
|
+
## 🚀 What is DATADOC?
|
|
82
|
+
|
|
83
|
+
**DATADOC** is a local-first CLI and Python library for preparing tabular data for machine learning. It profiles dataset risks, creates explainable transformation plans, and saves fitted pipelines that apply the same training-derived rules to validation, test, and inference data.
|
|
84
|
+
|
|
85
|
+
Powered by **Polars**, DATADOC reads CSV and Parquet files, diagnoses missing values, identifiers, schema issues, duplicates, constants, and unsafe feature types. It does not promise model improvement: optional evaluation reports the observed result against a baseline under a reproducible split.
|
|
86
|
+
|
|
87
|
+
**DATADOC is NOT just another EDA (Exploratory Data Analysis) tool.** It profiles data quality, lets you review a plan, fits transformations from training data, and hands you a portable artifact and Python wrapper for reuse.
|
|
88
|
+
|
|
89
|
+
### ⚡ The Impact: Why Industry Professionals Use DATADOC
|
|
90
|
+
|
|
91
|
+
Data Scientists and ML Engineers repeatedly rebuild the same preparation steps across projects.
|
|
92
|
+
DATADOC turns those steps into a reviewable, reusable pipeline.
|
|
93
|
+
|
|
94
|
+
- **Save boilerplate:** Review recommendations for imputing nulls, encoding categories, and optional scaling or clipping before applying them.
|
|
95
|
+
- **Explainable by default:** The deterministic core records roles, findings, operations, protected columns, and fitted statistics in an inspectable artifact.
|
|
96
|
+
- **Local-first:** The core package works offline. Optional ML, UI, and AI features are separate extras.
|
|
97
|
+
- **Optional AI planning:** AI can help explain or rank a constrained plan; it is never allowed to execute arbitrary generated code.
|
|
98
|
+
- **Leakage-safe workflows:** Fitted statistics for imputation, categorical vocabularies, clipping, and scaling are learned from training data and saved as an artifact.
|
|
99
|
+
|
|
100
|
+
### 📊 Empirical Proof: Does DATADOC Actually Improve Models?
|
|
101
|
+
|
|
102
|
+
Yes. We benchmarked naive manual preparation vs. DATADOC automated preprocessing on Kaggle's Titanic dataset under **5-fold Stratified Cross-Validation**:
|
|
103
|
+
|
|
104
|
+
| Model | Baseline (Naive Prep) | DATADOC Cleaned | Accuracy Δ | Relative Lift |
|
|
105
|
+
|:---|:---:|:---:|:---:|:---:|
|
|
106
|
+
| **Logistic Regression** | 78.90% ± 0.99% | **79.91% ± 1.90%** | **+1.01%** | **+1.28%** |
|
|
107
|
+
| **Random Forest** | 82.15% ± 2.45% | **82.82% ± 2.40%** | **+0.67%** | **+0.82%** |
|
|
108
|
+
|
|
109
|
+
*Why?* DATADOC extracts informative missingness indicators (`Age__missing`, `Cabin__missing`), frequency-encodes high-cardinality features (`Ticket`, `Cabin`), applies standard scaling, and isolates all statistics strictly to training splits to eliminate target leakage.
|
|
110
|
+
|
|
111
|
+
---
|
|
112
|
+
|
|
113
|
+
## 📦 Installation
|
|
114
|
+
|
|
115
|
+
DATADOC is published on PyPI. You can install it globally via `pip` or `uv`:
|
|
116
|
+
|
|
117
|
+
```bash
|
|
118
|
+
pip install datadoc-cli
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
*(Requires Python 3.10+)*
|
|
122
|
+
|
|
123
|
+
---
|
|
124
|
+
|
|
125
|
+
## 🛠️ Quick Start (CLI)
|
|
126
|
+
|
|
127
|
+
You don't need to write a single line of Python to clean your data. Just use the CLI.
|
|
128
|
+
New here? Run the guided wizard — it asks for target + preset and runs everything:
|
|
129
|
+
|
|
130
|
+
```bash
|
|
131
|
+
datadoc wizard train.csv
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
Or run the one-shot happy path (profile → plan → fit → transform + manifest):
|
|
135
|
+
|
|
136
|
+
```bash
|
|
137
|
+
datadoc run train.csv --target churn --preset balanced --evaluate
|
|
138
|
+
```
|
|
139
|
+
|
|
140
|
+
Full step-by-step (auditable) workflow:
|
|
141
|
+
|
|
142
|
+
```bash
|
|
143
|
+
# 0. Optional: save repeatable settings (target, preset, scaling, ...)
|
|
144
|
+
datadoc init --preset balanced # writes datadoc.toml
|
|
145
|
+
|
|
146
|
+
# 1. Inspect data-quality findings and column roles
|
|
147
|
+
datadoc profile raw_data.csv --target churn --explain --output profile.json
|
|
148
|
+
|
|
149
|
+
# 2. Review the proposed transformations before applying them
|
|
150
|
+
datadoc plan raw_data.csv --target churn --explain --output plan.json
|
|
151
|
+
|
|
152
|
+
# 3. Fit only on a training dataset, then save a reusable artifact
|
|
153
|
+
datadoc fit train.csv --target churn --preset balanced --rare-frequency 0.02 --output artifacts/churn-pipeline.json
|
|
154
|
+
|
|
155
|
+
# 4. Apply the fitted artifact to validation, test, or new data
|
|
156
|
+
datadoc transform validation.csv --pipeline artifacts/churn-pipeline.json --output validation-features.parquet --validate
|
|
157
|
+
|
|
158
|
+
# 5. Optionally benchmark a safe candidate pipeline against a baseline
|
|
159
|
+
pip install "datadoc-cli[ml]"
|
|
160
|
+
datadoc evaluate train.csv --target churn --task classification --ablation
|
|
161
|
+
|
|
162
|
+
# 6. Export a small executable wrapper around the fitted artifact
|
|
163
|
+
datadoc export --pipeline artifacts/churn-pipeline.json --output pipeline.py
|
|
164
|
+
# or: datadoc export --pipeline artifacts/churn-pipeline.json --format joblib --output pipeline.joblib
|
|
165
|
+
|
|
166
|
+
# 7. Generate a standalone, shareable HTML audit report
|
|
167
|
+
datadoc report train.csv --target churn --output report.html
|
|
168
|
+
|
|
169
|
+
# 8. Visually compare raw vs transformed datasets side-by-side
|
|
170
|
+
datadoc compare train.csv validation-features.parquet --target churn --html compare.html
|
|
171
|
+
|
|
172
|
+
# 9. Lint for leakage risks / diff two plans
|
|
173
|
+
datadoc lint train.csv --target churn
|
|
174
|
+
datadoc diff plan-v1.json plan-v2.json
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
### 🖥️ Web dashboard (same pipeline, visual)
|
|
178
|
+
|
|
179
|
+
```bash
|
|
180
|
+
pip install "datadoc-cli[ui]"
|
|
181
|
+
datadoc ui train.csv --port 8000
|
|
182
|
+
```
|
|
183
|
+
|
|
184
|
+
The local dashboard calls the same `DataDocPipeline` behind the CLI: profile findings and roles, preparation settings (target, scaling, identifiers, dedup, clipping, cyclical datetime, rare frequency), reviewable plan, fit with output-schema preview, lineage/provenance panel, transformed-CSV download, and an executable Python export. Press `Ctrl+K`/`Cmd+K` for the command palette. Full guide: [docs/ui.html](https://narain-karti.github.io/DATADOC/ui.html).
|
|
185
|
+
|
|
186
|
+
---
|
|
187
|
+
|
|
188
|
+
## 🐍 Python SDK (Library Usage)
|
|
189
|
+
|
|
190
|
+
DATADOC is also a Python library. The stable workflow is `profile → plan → fit → transform`; the same fitted artifact can be used in notebooks, services, and batch jobs:
|
|
191
|
+
|
|
192
|
+
```python
|
|
193
|
+
from datadoc import DataDocPipeline, PipelineConfig
|
|
194
|
+
import polars as pl
|
|
195
|
+
|
|
196
|
+
# Fit only on the training split. The target is protected from feature transforms.
|
|
197
|
+
train_df = pl.read_csv("train.csv")
|
|
198
|
+
pipeline = DataDocPipeline(PipelineConfig(target="churn")).fit(train_df)
|
|
199
|
+
pipeline.save("artifacts/churn-pipeline.json")
|
|
200
|
+
|
|
201
|
+
# Transform data that was never used to fit statistics.
|
|
202
|
+
validation_df = pl.read_csv("validation.csv")
|
|
203
|
+
validation_features = pipeline.transform(validation_df)
|
|
204
|
+
```
|
|
205
|
+
|
|
206
|
+
For an observed model comparison, install the optional ML extra and call `pipeline.evaluate(train_df)` or `datadoc evaluate`. Evaluation is evidence for the declared task and split strategy; it is not a promise that cleaning always improves a model.
|
|
207
|
+
|
|
208
|
+
---
|
|
209
|
+
|
|
210
|
+
## 💻 CLI Commands Reference
|
|
211
|
+
|
|
212
|
+
| Command | Description |
|
|
213
|
+
|---------|-------------|
|
|
214
|
+
| `datadoc wizard <file>` | Guided TUI: asks target/preset/scaling, writes `datadoc.toml`, runs pipeline |
|
|
215
|
+
| `datadoc init` | Writes a starter `datadoc.toml` (or `pyproject.toml [tool.datadoc]`) config |
|
|
216
|
+
| `datadoc profile <file>` | Data-quality report + roles (`--explain`, `--compare profile2.json`) |
|
|
217
|
+
| `datadoc plan <file>` | Explainable plan (`--explain`, `--diff plan2.json`) |
|
|
218
|
+
| `datadoc fit <train>` | Learns pipeline on train only (`--preset`, `--deduplicate`, `--rare-frequency`, `--cyclical`, `--no-hour`, repeatable `--identifier-column` / `--ignore-column`) |
|
|
219
|
+
| `datadoc transform <file>` | Applies saved artifact (`--validate` for schema + drift checks) |
|
|
220
|
+
| `datadoc evaluate <file>` | Candidate vs baseline (`--ablation` for per-component deltas) |
|
|
221
|
+
| `datadoc export` | Wrapper for artifact (`--format python\|joblib`) |
|
|
222
|
+
| `datadoc run <file>` | One-shot profile→plan→fit→transform + `manifest.json` (+ `--evaluate --ablation`) |
|
|
223
|
+
| `datadoc report <file>` | Generates a standalone, shareable HTML data health and preparation audit report |
|
|
224
|
+
| `datadoc compare <raw> <trans>` | Visual side-by-side dataset comparison (terminal table + HTML export) |
|
|
225
|
+
| `datadoc lint <file>` | Leakage/pitfall lint (target duplication, nulls, infinities, duplicates) |
|
|
226
|
+
| `datadoc diff <a.json> <b.json>` | Diff profile/plan/pipeline artifacts |
|
|
227
|
+
| `datadoc plugins list` | Lists 9 registered plugins (priorities, entry-points) |
|
|
228
|
+
| `datadoc ui <file>` | Local FastAPI dashboard (Ctrl+K palette, lineage panel, HTML report export) |
|
|
229
|
+
|
|
230
|
+
Short aliases: `-t/--target`, `-o/--output`, `-p/--pipeline`, `-f/--format`.
|
|
231
|
+
Presets: `--preset quick|balanced|linear|tree|time|robust`. Shell completion: `datadoc --install-completion`.
|
|
232
|
+
|
|
233
|
+
|
|
234
|
+
|
|
235
|
+
---
|
|
236
|
+
|
|
237
|
+
## 🧩 Architecture & Plugins
|
|
238
|
+
|
|
239
|
+
DATADOC operates as a fitted pipeline. Every transformation learns state only from training data, saves that state to JSON, and reuses it unchanged for later datasets.
|
|
240
|
+
|
|
241
|
+
| Priority | Plugin | Action Performed |
|
|
242
|
+
|----------|--------|-------------|
|
|
243
|
+
| 5 | **DuplicateRemoverPlugin** | Detects duplicate rows; `deduplicate=True` drops them at fit (train-only) |
|
|
244
|
+
| 10 | **MissingValuePlugin** | Imputes missing numeric values with median, categorical with mode |
|
|
245
|
+
| 20 | **OutlierPlugin** | Offers optional IQR clipping; clipping is not forced by default |
|
|
246
|
+
| 30 | **DatetimePlugin** | Detects date strings and extracts year, month, day, day_of_week (+hour when time present, optional cyclical sin/cos) |
|
|
247
|
+
| 40 | **CategoricalEncoderPlugin** | Encodes categories using training vocabularies (threshold 20) and handles unseen values |
|
|
248
|
+
| 41 | **TargetEncoderPlugin** | Empirical Bayes smoothed target encoding: `(n * cat_mean + m * global_mean) / (n + m)` |
|
|
249
|
+
| 42 | **RareCategoryPlugin** | Groups rare categories (< `rare_category_min_frequency`) into `__RARE__` |
|
|
250
|
+
| 44 | **PolynomialFeaturesPlugin** | Generates degree-2 interaction terms (`x1 * x2`) and squared terms (`x^2`) |
|
|
251
|
+
| 45 | **ScalingPlugin** | Applies configured standard or robust scaling, fit on training data only |
|
|
252
|
+
|
|
253
|
+
The fitted pipeline is the production source of truth. Plugin work should follow the lifecycle `analyze → recommend → apply`, with fitted state (`median`, `clip`, `vocabularies`, `rare maps`, `hour flags`, `center/spread`) serializable in `pipeline.json` (artifact v2 with `provenance`). External plugins auto-register via `datadoc.plugins` entry-points.
|
|
254
|
+
|
|
255
|
+
Want to build your own? See [CONTRIBUTING.md](CONTRIBUTING.md) to learn how to create and register custom plugins!
|
|
256
|
+
|
|
257
|
+
---
|
|
258
|
+
|
|
259
|
+
## 🗺️ Roadmap
|
|
260
|
+
|
|
261
|
+
- [x] Core Engine with plugin orchestration
|
|
262
|
+
- [x] 9 Built-in deterministic plugins (duplicate, missing, outlier, datetime, encoder, target encoder, rare, polynomial, scaling)
|
|
263
|
+
- [x] Stunning Rich Terminal UI (wizard, presets, `datadoc.toml`, completion)
|
|
264
|
+
- [x] Pipeline export capability (`python` + `joblib`)
|
|
265
|
+
- [x] Polars backend and local-first pipeline artifacts (v2 + provenance)
|
|
266
|
+
- [x] PyPI Release (`pip install datadoc-cli`)
|
|
267
|
+
- [x] Constrained optional AI planning path
|
|
268
|
+
- [x] Session-scoped local FastAPI dashboard (Ctrl+K palette, lineage, drift)
|
|
269
|
+
- [x] Addictive loop: `profile --compare`, `plan --explain/--diff`, `transform --validate`, `evaluate --ablation`
|
|
270
|
+
- [x] Standalone HTML audit reports (`datadoc report`)
|
|
271
|
+
- [x] Visual dataset comparison engine (`datadoc compare`)
|
|
272
|
+
- [x] Notebook widgets (`profile_to_html`, `_repr_html_`)
|
|
273
|
+
- [ ] Export targets for `dbt` and Apache Airflow
|
|
274
|
+
- [x] Local FastAPI dashboard/API companion
|
|
275
|
+
|
|
276
|
+
---
|
|
277
|
+
|
|
278
|
+
## ⚖️ License
|
|
279
|
+
|
|
280
|
+
This project is licensed under the MIT License. See [LICENSE](LICENSE) for details.
|
|
281
|
+
|
|
282
|
+
## 🤝 Contributing
|
|
283
|
+
|
|
284
|
+
We welcome contributions from the community! If you'd like to add a new plugin or improve the core engine, please see [CONTRIBUTING.md](CONTRIBUTING.md).
|
|
285
|
+
|
|
286
|
+
See [CHANGELOG.md](CHANGELOG.md) for the 0.6.0 release notes.
|