dataenginex 0.9.9__tar.gz → 0.10.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {dataenginex-0.9.9/docs → dataenginex-0.10.0/.github}/release-pr-template.md +3 -3
- {dataenginex-0.9.9 → dataenginex-0.10.0}/.github/workflows/ci.yml +2 -1
- dataenginex-0.10.0/.github/workflows/docs-notify.yml +18 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/.github/workflows/release-please.yml +1 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/.github/workflows/security.yml +1 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/.gitignore +3 -0
- dataenginex-0.10.0/.release-please-manifest.json +3 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/CHANGELOG.md +21 -0
- dataenginex-0.10.0/CLAUDE.md +59 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/PKG-INFO +1 -1
- dataenginex-0.9.9/docs/CI_CD.md → dataenginex-0.10.0/docs/ci-cd.md +2 -2
- dataenginex-0.9.9/docs/CONTRIBUTING.md → dataenginex-0.10.0/docs/contributing.md +10 -8
- dataenginex-0.9.9/docs/DEVELOPMENT.md → dataenginex-0.10.0/docs/development.md +2 -2
- {dataenginex-0.9.9 → dataenginex-0.10.0}/docs/index.md +3 -3
- dataenginex-0.9.9/docs/OBSERVABILITY.md → dataenginex-0.10.0/docs/observability.md +3 -3
- {dataenginex-0.9.9 → dataenginex-0.10.0}/docs/quickstart.md +2 -2
- dataenginex-0.9.9/docs/SDLC.md → dataenginex-0.10.0/docs/sdlc.md +4 -4
- dataenginex-0.10.0/docs/superpowers/plans/2026-03-28-unified-docs-and-tooling.md +2137 -0
- dataenginex-0.10.0/docs/superpowers/specs/2026-03-28-unified-docs-and-tooling-design.md +683 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/poe_tasks.toml +10 -23
- {dataenginex-0.9.9 → dataenginex-0.10.0}/pyproject.toml +9 -3
- dataenginex-0.10.0/release-please-config.json +23 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/__init__.py +1 -1
- dataenginex-0.10.0/src/dataenginex/data/quality/__init__.py +7 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/data/quality/gates.py +70 -2
- dataenginex-0.10.0/src/dataenginex/data/quality/spark.py +90 -0
- dataenginex-0.10.0/src/dataenginex/secops/audit.py +258 -0
- dataenginex-0.10.0/tests/unit/test_quality_gates.py +105 -0
- dataenginex-0.10.0/tests/unit/test_quality_spark.py +93 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_secops.py +53 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/uv.lock +57 -1067
- dataenginex-0.9.9/.release-please-manifest.json +0 -3
- dataenginex-0.9.9/CLAUDE.md +0 -171
- dataenginex-0.9.9/docs/docs-hub.md +0 -103
- dataenginex-0.9.9/mkdocs.yml +0 -61
- dataenginex-0.9.9/release-please-config.json +0 -18
- dataenginex-0.9.9/src/dataenginex/data/quality/__init__.py +0 -7
- dataenginex-0.9.9/src/dataenginex/secops/audit.py +0 -156
- dataenginex-0.9.9/tests/unit/test_quality_gates.py +0 -51
- {dataenginex-0.9.9 → dataenginex-0.10.0}/.claude/commands/new-feature.md +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/.claude/commands/validate.md +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/.claude/settings.json +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/.env.template +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/.github/ISSUE_TEMPLATE/bug_report.yml +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/.github/ISSUE_TEMPLATE/config.yml +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/.github/ISSUE_TEMPLATE/feature_request.yml +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/.github/PULL_REQUEST_TEMPLATE.md +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/.github/dependabot.yml +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/.github/labels.yml +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/.github/workflows/auto-pr-dev-to-main.yml +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/.github/workflows/auto-pr-to-dev.yml +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/.github/workflows/claude.yml +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/.github/workflows/docker-build-push.yml +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/.github/workflows/enforce-dev-to-main.yml +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/.github/workflows/label-sync.yml +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/.github/workflows/pypi-publish.yml +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/.github/workflows/release-dex.yml +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/.gitleaks.toml +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/.pre-commit-config.yaml +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/.python-version +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/CODEOWNERS +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/CONTRIBUTING.md +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/Dockerfile +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/LICENSE +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/README.md +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/docker-compose.test.yml +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/docs/adr/0000-template.md +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/docs/adr/0001-medallion-architecture.md +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/docs/api-reference/api.md +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/docs/api-reference/core.md +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/docs/api-reference/dashboard.md +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/docs/api-reference/data.md +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/docs/api-reference/index.md +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/docs/api-reference/lakehouse.md +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/docs/api-reference/middleware.md +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/docs/api-reference/ml.md +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/docs/api-reference/plugins.md +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/docs/api-reference/warehouse.md +0 -0
- /dataenginex-0.9.9/docs/ARCHITECTURE.md → /dataenginex-0.10.0/docs/architecture.md +0 -0
- /dataenginex-0.9.9/docs/RELEASE_NOTES.md → /dataenginex-0.10.0/docs/release-notes.md +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/docs/roadmap/project-roadmap.csv +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/docs/roadmap/project-roadmap.json +0 -0
- /dataenginex-0.9.9/docs/SECURITY_SCANNING.md → /dataenginex-0.10.0/docs/security-scanning.md +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/docs/superpowers/plans/2026-03-21-phase-0-foundation.md +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/docs/superpowers/plans/2026-03-22-dataenginex-1.0-master-plan.md +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/docs/superpowers/plans/2026-03-22-phase-1-data-layer.md +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/docs/superpowers/plans/2026-03-22-phase-6a-dex-engine-integration.md +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/docs/superpowers/plans/2026-03-22-phase-6b-dex-studio-redesign.md +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/docs/superpowers/plans/2026-03-23-careerdex-example.md +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/docs/superpowers/plans/2026-03-23-dex-studio-direct-import.md +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/docs/superpowers/plans/2026-03-24-dex-naming-architecture-dry.md +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/docs/superpowers/specs/2026-03-21-dataenginex-v2-system-redesign.md +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/docs/superpowers/specs/2026-03-22-phase-6-integration-design.md +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/docs/superpowers/specs/2026-03-23-dex-studio-direct-import-design.md +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/docs/superpowers/specs/2026-03-24-dex-naming-architecture-dry-design.md +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/examples/01_hello_pipeline.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/examples/02_api_quickstart.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/examples/03_quality_gate.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/examples/04_ml_training.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/examples/05_rag_demo.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/examples/06_llm_quickstart.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/examples/07_api_ingestion.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/examples/08_spark_ml.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/examples/09_feature_engineering.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/examples/10_model_analysis.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/examples/GUIDE.md +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/examples/dashboard/dashboard_config.yaml +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/examples/dashboard/run_dashboard.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/examples/dex.yaml +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/examples/movies.csv +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/scripts/GUIDE.md +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/scripts/localstack/create-buckets.sh +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/scripts/promote.sh +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/sonar-project.properties +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/README.md +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/RELEASE_NOTES.md +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/ai/__init__.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/ai/agents/__init__.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/ai/agents/builtin.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/ai/retrieval/__init__.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/ai/retrieval/builtin.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/ai/tools/__init__.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/ai/tools/builtin.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/api/__init__.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/api/auth.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/api/errors.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/api/factory.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/api/health.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/api/pagination.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/api/rate_limit.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/api/routers/__init__.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/api/routers/ai.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/api/routers/data.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/api/routers/health.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/api/routers/ml.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/api/routers/pipelines.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/api/routers/root.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/api/routers/system.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/api/schemas.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/cli/__init__.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/cli/main.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/cli/run.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/cli/serve.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/cli/train.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/config/__init__.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/config/defaults.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/config/loader.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/config/schema.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/core/__init__.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/core/exceptions.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/core/interfaces.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/core/medallion_architecture.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/core/quality.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/core/registry.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/core/schemas.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/core/validators.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/dashboard/__init__.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/dashboard/app.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/dashboard/panels.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/data/__init__.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/data/connectors/__init__.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/data/connectors/csv.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/data/connectors/duckdb.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/data/connectors/legacy.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/data/pipeline/__init__.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/data/pipeline/dag.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/data/pipeline/run_history.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/data/pipeline/runner.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/data/profiler.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/data/registry.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/data/transforms/__init__.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/data/transforms/sql.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/lakehouse/__init__.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/lakehouse/catalog.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/lakehouse/partitioning.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/lakehouse/storage.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/middleware/__init__.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/middleware/logging_config.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/middleware/metrics.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/middleware/metrics_middleware.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/middleware/request_logging.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/middleware/tracing.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/ml/__init__.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/ml/drift.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/ml/features/__init__.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/ml/features/builtin.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/ml/llm.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/ml/metrics.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/ml/mlflow_registry.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/ml/registry.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/ml/scheduler.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/ml/serving.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/ml/serving_engine/__init__.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/ml/serving_engine/builtin.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/ml/tracking/__init__.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/ml/tracking/builtin.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/ml/training.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/ml/vectorstore.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/orchestration/__init__.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/orchestration/builtin.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/plugins/__init__.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/plugins/registry.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/py.typed +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/secops/__init__.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/secops/gate.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/secops/masking.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/secops/pii.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/warehouse/__init__.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/warehouse/lineage.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/src/dataenginex/warehouse/transforms.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tasks/findings.md +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tasks/lessons.md +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/__init__.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/conformance/__init__.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/conformance/test_connector.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/conformance/test_feature_store.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/conformance/test_tracker.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/conformance/test_transform.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/conftest.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/fixtures/__init__.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/fixtures/sample_data.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/fixtures/sample_jobs.csv +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/fixtures/sample_jobs.json +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/integration/__init__.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/integration/test_cli_run.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/integration/test_config_cli.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/integration/test_full_app.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/integration/test_pipeline_e2e.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/integration/test_storage_real.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/load/__init__.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/__init__.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_agent_runtime.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_ai_router.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_api_auth.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_api_factory.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_api_pagination.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_api_rate_limit.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_api_schemas.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_api_validators.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_builtin_agent.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_builtin_feature_store.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_builtin_serving.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_builtin_tracker.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_cli_train.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_config_loader.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_config_schema.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_core_exceptions.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_core_interfaces.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_core_registry.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_csv_connector.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_data.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_data_router.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_drift_scheduler.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_duckdb_connector.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_errors.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_health.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_lakehouse.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_llm.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_logging.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_medallion.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_metrics.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_middleware.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_ml.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_ml_router.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_pipeline_dag.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_pipeline_runner.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_plugins.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_retriever.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_run_history.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_scheduler.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_spark_fixtures.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_sql_transforms.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_storage_abstraction.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_system_router.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_tracing.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_vectorstore.py +0 -0
- {dataenginex-0.9.9 → dataenginex-0.10.0}/tests/unit/test_warehouse.py +0 -0
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
# Release PR Template
|
|
2
2
|
|
|
3
|
-
Use this template whenever opening a release PR (`dev → main`). Reference `
|
|
3
|
+
Use this template whenever opening a release PR (`dev → main`). Reference `release-notes.md` and verify QA steps before merging.
|
|
4
4
|
|
|
5
5
|
## Summary
|
|
6
6
|
|
|
7
7
|
- Describe what changed (e.g., alerts, APIs, docs).
|
|
8
|
-
- Link to the release note: [
|
|
8
|
+
- Link to the release note: [release-notes.md](release-notes.md).
|
|
9
9
|
- Note any rollout considerations (e.g., Alertmanager rules, Docker image tag).
|
|
10
10
|
|
|
11
11
|
## Verification
|
|
@@ -13,7 +13,7 @@ Use this template whenever opening a release PR (`dev → main`). Reference `REL
|
|
|
13
13
|
- `uv run poe lint`
|
|
14
14
|
- `./.venv/bin/pytest -v`
|
|
15
15
|
- `docker compose build`
|
|
16
|
-
- Reload Prometheus/Alertmanager (see `
|
|
16
|
+
- Reload Prometheus/Alertmanager (see `observability.md` → "Reloading Alert Rules").
|
|
17
17
|
|
|
18
18
|
## QA Checklist
|
|
19
19
|
|
|
@@ -5,6 +5,7 @@ on:
|
|
|
5
5
|
branches: [main, dev]
|
|
6
6
|
pull_request:
|
|
7
7
|
branches: [main, dev]
|
|
8
|
+
workflow_dispatch:
|
|
8
9
|
|
|
9
10
|
permissions:
|
|
10
11
|
contents: read
|
|
@@ -38,7 +39,7 @@ jobs:
|
|
|
38
39
|
env:
|
|
39
40
|
UV_PROJECT_ENVIRONMENT: .venv
|
|
40
41
|
- run: uv run poe test-cov-core
|
|
41
|
-
- uses: codecov/codecov-action@
|
|
42
|
+
- uses: codecov/codecov-action@v6
|
|
42
43
|
with:
|
|
43
44
|
flags: dataenginex
|
|
44
45
|
fail_ci_if_error: false
|
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
name: Notify Docs
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
push:
|
|
5
|
+
branches: [main]
|
|
6
|
+
paths: ['docs/**', 'src/**/**.py']
|
|
7
|
+
|
|
8
|
+
jobs:
|
|
9
|
+
notify:
|
|
10
|
+
runs-on: ubuntu-latest
|
|
11
|
+
steps:
|
|
12
|
+
- uses: peter-evans/repository-dispatch@v3
|
|
13
|
+
with:
|
|
14
|
+
token: ${{ secrets.DOCS_DISPATCH_TOKEN }}
|
|
15
|
+
repository: TheDataEngineX/docs
|
|
16
|
+
event-type: docs-update
|
|
17
|
+
client-payload: >-
|
|
18
|
+
{"repo": "${{ github.repository }}", "ref": "${{ github.sha }}"}
|
|
@@ -5,6 +5,27 @@ All notable changes to `dataenginex` will be documented in this file.
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
+
## [0.10.0](https://github.com/TheDataEngineX/dex/compare/v0.9.9...v0.10.0) (2026-04-01)
|
|
9
|
+
|
|
10
|
+
|
|
11
|
+
### Features
|
|
12
|
+
|
|
13
|
+
* docs cleanup ([f4eff14](https://github.com/TheDataEngineX/dex/commit/f4eff14a1aeed657b715b182ac495c920f7b701f))
|
|
14
|
+
* docs notify ([#210](https://github.com/TheDataEngineX/dex/issues/210)) ([c36616d](https://github.com/TheDataEngineX/dex/commit/c36616d9d0ae7a97274bd40ea41f39bc26f68fe0))
|
|
15
|
+
* quality schema spark audit ([#212](https://github.com/TheDataEngineX/dex/issues/212)) ([2965801](https://github.com/TheDataEngineX/dex/commit/2965801257f6949a48f1c59be2f88e92294b4691))
|
|
16
|
+
|
|
17
|
+
|
|
18
|
+
### Bug Fixes
|
|
19
|
+
|
|
20
|
+
* add last-release-sha to anchor release-please at v0.9.9 ([7e78ca2](https://github.com/TheDataEngineX/dex/commit/7e78ca203347ff4c4000ca6ec42c335650da1b86))
|
|
21
|
+
* restructure release-please-config to packages format with last-release-sha ([46f0b69](https://github.com/TheDataEngineX/dex/commit/46f0b69b8372d54d26ba60eedd9095d7fb0f6cd3))
|
|
22
|
+
* sync dev to main ([40d5dff](https://github.com/TheDataEngineX/dex/commit/40d5dff99ae4295599badc848cd530df6eff9ffb))
|
|
23
|
+
|
|
24
|
+
|
|
25
|
+
### Documentation
|
|
26
|
+
|
|
27
|
+
* docs cleanup ([#211](https://github.com/TheDataEngineX/dex/issues/211)) ([0a6b7f7](https://github.com/TheDataEngineX/dex/commit/0a6b7f70287eb7399eb17acea2fb0ad09efd0597))
|
|
28
|
+
|
|
8
29
|
## [Unreleased]
|
|
9
30
|
|
|
10
31
|
## [0.7.1] - 2026-03-17
|
|
@@ -0,0 +1,59 @@
|
|
|
1
|
+
# CLAUDE.md — DEX (dataenginex)
|
|
2
|
+
|
|
3
|
+
Brief answers only. No explanations unless asked.
|
|
4
|
+
Goal is to save Claude code tokens for lower cost without loosing quality.
|
|
5
|
+
|
|
6
|
+
> Repo-specific context. Workspace-level rules, coding standards, and git conventions are in `../CLAUDE.md`.
|
|
7
|
+
|
|
8
|
+
## Project Overview
|
|
9
|
+
|
|
10
|
+
**DEX** — unified Data + ML + AI framework. Config-driven, self-hosted, production-ready.
|
|
11
|
+
|
|
12
|
+
| Package | Location | Purpose |
|
|
13
|
+
|---------|----------|---------|
|
|
14
|
+
| `dataenginex` | `src/dataenginex/` | Core framework — config system, backend registry, CLI, API, ML, AI agents |
|
|
15
|
+
|
|
16
|
+
**Stack:** Python 3.13+ · FastAPI · DuckDB · structlog · Pydantic · Click · Rich · uv · Ruff · mypy strict · pytest
|
|
17
|
+
|
|
18
|
+
**Version:** `uv run poe version`
|
|
19
|
+
|
|
20
|
+
**Domain:** thedataenginex.org | **Org:** github.com/TheDataEngineX
|
|
21
|
+
|
|
22
|
+
______________________________________________________________________
|
|
23
|
+
|
|
24
|
+
## Build & Run Commands
|
|
25
|
+
|
|
26
|
+
```bash
|
|
27
|
+
# Quality
|
|
28
|
+
uv run poe lint # Ruff lint
|
|
29
|
+
uv run poe lint-fix # Ruff lint + auto-fix
|
|
30
|
+
uv run poe typecheck # mypy --strict (src/dataenginex/ only)
|
|
31
|
+
uv run poe check-all # lint + typecheck + test
|
|
32
|
+
|
|
33
|
+
# Test
|
|
34
|
+
uv run poe test # All tests
|
|
35
|
+
uv run poe test-unit # Unit tests only
|
|
36
|
+
uv run poe test-integration # Integration tests only
|
|
37
|
+
uv run poe test-cov # Tests with coverage
|
|
38
|
+
|
|
39
|
+
# CLI
|
|
40
|
+
dex validate dex.yaml # Validate config file
|
|
41
|
+
dex version # Show version + environment
|
|
42
|
+
|
|
43
|
+
# Run
|
|
44
|
+
uv run poe dev # Dev server (uvicorn reload, port 17000)
|
|
45
|
+
uv run poe docker-up # Docker compose up
|
|
46
|
+
uv run poe docker-down # Docker compose down
|
|
47
|
+
|
|
48
|
+
# Deps
|
|
49
|
+
uv run poe uv-sync # Sync deps from lockfile
|
|
50
|
+
uv run poe uv-lock # Regenerate lockfile
|
|
51
|
+
uv run poe security # Audit deps for vulnerabilities
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
______________________________________________________________________
|
|
55
|
+
|
|
56
|
+
## Mandatory Validation
|
|
57
|
+
|
|
58
|
+
Run `/validate` after ANY code change. Steps 4-5 (config validation + server smoke test) are NON-NEGOTIABLE.
|
|
59
|
+
Tests passing ≠ app working.
|
|
@@ -355,8 +355,8 @@ ______________________________________________________________________
|
|
|
355
355
|
**Next Steps:**
|
|
356
356
|
|
|
357
357
|
- **Deployment Runbook** (in `infradex` repo) - Release procedures
|
|
358
|
-
- **[Observability](
|
|
359
|
-
- **[Contributing Guide](
|
|
358
|
+
- **[Observability](observability.md)** - Monitor applications built on DEX
|
|
359
|
+
- **[Contributing Guide](contributing.md)** - Development workflow
|
|
360
360
|
|
|
361
361
|
______________________________________________________________________
|
|
362
362
|
|
|
@@ -1,8 +1,10 @@
|
|
|
1
|
-
|
|
1
|
+
# Contributing to DEX
|
|
2
|
+
|
|
3
|
+
Thank you for contributing!
|
|
2
4
|
|
|
3
5
|
## Getting Started
|
|
4
6
|
|
|
5
|
-
1. Read [
|
|
7
|
+
1. Read [development.md](./development.md) for setup instructions
|
|
6
8
|
1. Fork the repository
|
|
7
9
|
1. Create a feature branch from `dev`
|
|
8
10
|
1. Make your changes
|
|
@@ -12,10 +14,10 @@ Thank you for contributing to DEX!
|
|
|
12
14
|
|
|
13
15
|
Use semantic commit format:
|
|
14
16
|
|
|
15
|
-
- `feat(#123): add
|
|
16
|
-
- `fix(#124): handle
|
|
17
|
-
- `docs: update API
|
|
18
|
-
- `test: add
|
|
17
|
+
- `feat(#123): add DuckDB backend`
|
|
18
|
+
- `fix(#124): handle missing config key in registry`
|
|
19
|
+
- `docs: update API reference`
|
|
20
|
+
- `test: add backend integration tests`
|
|
19
21
|
- `chore: update dependencies`
|
|
20
22
|
|
|
21
23
|
## Code Style
|
|
@@ -96,8 +98,8 @@ uv run poe check-all # Run all checks
|
|
|
96
98
|
|
|
97
99
|
## Questions?
|
|
98
100
|
|
|
99
|
-
- Check [
|
|
100
|
-
- Review [Architecture docs](./
|
|
101
|
+
- Check [development.md](./development.md)
|
|
102
|
+
- Review [Architecture docs](./architecture.md)
|
|
101
103
|
- Create a GitHub issue
|
|
102
104
|
- Join #dex-dev Slack channel
|
|
103
105
|
|
|
@@ -205,8 +205,8 @@ uv run poe clean # Remove caches and build artifacts
|
|
|
205
205
|
|
|
206
206
|
## Resources & Support
|
|
207
207
|
|
|
208
|
-
- **Code Style**: See [
|
|
209
|
-
- **Architecture**: See [
|
|
208
|
+
- **Code Style**: See [contributing.md](./contributing.md)
|
|
209
|
+
- **Architecture**: See [architecture.md](./architecture.md)
|
|
210
210
|
- **ADRs**: See [ADR-0001](./adr/0001-medallion-architecture.md) for architectural decisions
|
|
211
211
|
- **Deployment**: See Deployment Runbook in the `infradex` repo
|
|
212
212
|
- **Issues**: [GitHub Issues](https://github.com/TheDataEngineX/dataenginex/issues)
|
|
@@ -5,6 +5,6 @@ Welcome to the DEX docs site.
|
|
|
5
5
|
Start here:
|
|
6
6
|
|
|
7
7
|
- [Documentation Hub](docs-hub.md)
|
|
8
|
-
- [Development Setup](
|
|
9
|
-
- [Contributing](
|
|
10
|
-
- [Architecture](
|
|
8
|
+
- [Development Setup](development.md)
|
|
9
|
+
- [Contributing](contributing.md)
|
|
10
|
+
- [Architecture](architecture.md)
|
|
@@ -784,14 +784,14 @@ ______________________________________________________________________
|
|
|
784
784
|
|
|
785
785
|
**Deployment & Operations:**
|
|
786
786
|
|
|
787
|
-
- **[CI/CD Pipeline](
|
|
787
|
+
- **[CI/CD Pipeline](ci-cd.md)** - Automated deployments
|
|
788
788
|
- **Deployment Runbook** (in `infradex` repo) - Deploy procedures
|
|
789
789
|
- **Local K8s Setup** (in `infradex` repo) - Kubernetes setup
|
|
790
790
|
|
|
791
791
|
**Development:**
|
|
792
792
|
|
|
793
|
-
- **[SDLC](
|
|
794
|
-
- **[Contributing](
|
|
793
|
+
- **[SDLC](sdlc.md)** - Development workflow
|
|
794
|
+
- **[Contributing](contributing.md)** - Contribution guide
|
|
795
795
|
|
|
796
796
|
______________________________________________________________________
|
|
797
797
|
|
|
@@ -56,8 +56,8 @@ uv run poe check-all # All of the above in one command
|
|
|
56
56
|
|
|
57
57
|
## Next Steps
|
|
58
58
|
|
|
59
|
-
- [Development Guide](
|
|
60
|
-
- [Architecture](
|
|
59
|
+
- [Development Guide](development.md) — full setup, editor config, and workflow
|
|
60
|
+
- [Architecture](architecture.md) — medallion layers, API design, ML lifecycle
|
|
61
61
|
- [API Reference](api-reference/index.md) — auto-generated module docs
|
|
62
62
|
- `examples/` directory — full list of runnable examples (01–10)
|
|
63
63
|
|
|
@@ -95,7 +95,7 @@ ______________________________________________________________________
|
|
|
95
95
|
|
|
96
96
|
- CI, security, and quality checks as defined in the CI/CD pipeline.
|
|
97
97
|
|
|
98
|
-
See [CI/CD Pipeline](
|
|
98
|
+
See [CI/CD Pipeline](ci-cd.md) for the authoritative list of checks.
|
|
99
99
|
|
|
100
100
|
**Exit**: All checks pass and at least one reviewer approves.
|
|
101
101
|
|
|
@@ -185,7 +185,7 @@ flowchart TD
|
|
|
185
185
|
1. After validation in dev, open a release PR from `dev` → `main`.
|
|
186
186
|
1. Merge after all required checks pass.
|
|
187
187
|
|
|
188
|
-
See [Contributing Guide](
|
|
188
|
+
See [Contributing Guide](contributing.md) for branch naming, local checks, and PR conventions.
|
|
189
189
|
|
|
190
190
|
______________________________________________________________________
|
|
191
191
|
|
|
@@ -193,8 +193,8 @@ ______________________________________________________________________
|
|
|
193
193
|
|
|
194
194
|
**Development:**
|
|
195
195
|
|
|
196
|
-
- **[Contributing Guide](
|
|
197
|
-
- **[CI/CD Pipeline](
|
|
196
|
+
- **[Contributing Guide](contributing.md)** - Contribution workflow
|
|
197
|
+
- **[CI/CD Pipeline](ci-cd.md)** - Automated build and deploy
|
|
198
198
|
|
|
199
199
|
**Operations:**
|
|
200
200
|
|