dataenginex 0.4.3__tar.gz → 0.4.5__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {dataenginex-0.4.3 → dataenginex-0.4.5}/.claude/commands/new-feature.md +1 -1
- {dataenginex-0.4.3 → dataenginex-0.4.5}/.claude/commands/validate.md +7 -6
- {dataenginex-0.4.3 → dataenginex-0.4.5}/.github/dependabot.yml +0 -2
- {dataenginex-0.4.3 → dataenginex-0.4.5}/.github/release-pr-template.md +1 -1
- dataenginex-0.4.5/.github/workflows/auto-pr.yml +16 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/.github/workflows/ci.yml +6 -2
- {dataenginex-0.4.3 → dataenginex-0.4.5}/.github/workflows/release.yml +1 -1
- {dataenginex-0.4.3 → dataenginex-0.4.5}/.github/workflows/security.yml +3 -3
- {dataenginex-0.4.3 → dataenginex-0.4.5}/.pre-commit-config.yaml +50 -45
- {dataenginex-0.4.3 → dataenginex-0.4.5}/CLAUDE.md +6 -6
- {dataenginex-0.4.3 → dataenginex-0.4.5}/CODE_OF_CONDUCT.md +1 -1
- {dataenginex-0.4.3 → dataenginex-0.4.5}/CONTRIBUTING.md +5 -5
- {dataenginex-0.4.3 → dataenginex-0.4.5}/PKG-INFO +1 -3
- {dataenginex-0.4.3 → dataenginex-0.4.5}/README.md +3 -3
- {dataenginex-0.4.3 → dataenginex-0.4.5}/docker-compose.test.yml +1 -1
- {dataenginex-0.4.3 → dataenginex-0.4.5}/docs/adr/0001-medallion-architecture.md +3 -3
- {dataenginex-0.4.3 → dataenginex-0.4.5}/docs/architecture.md +52 -15
- dataenginex-0.4.5/docs/ci-cd.md +97 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/docs/contributing.md +4 -4
- {dataenginex-0.4.3 → dataenginex-0.4.5}/docs/development.md +62 -10
- {dataenginex-0.4.3 → dataenginex-0.4.5}/docs/quickstart.md +5 -3
- {dataenginex-0.4.3 → dataenginex-0.4.5}/docs/sdlc.md +12 -55
- {dataenginex-0.4.3 → dataenginex-0.4.5}/docs/security-scanning.md +2 -2
- {dataenginex-0.4.3 → dataenginex-0.4.5}/examples/01_hello_pipeline.py +1 -1
- {dataenginex-0.4.3 → dataenginex-0.4.5}/examples/02_api_quickstart.py +1 -1
- {dataenginex-0.4.3 → dataenginex-0.4.5}/examples/07_api_ingestion.py +1 -1
- {dataenginex-0.4.3 → dataenginex-0.4.5}/examples/08_spark_ml.py +3 -3
- {dataenginex-0.4.3 → dataenginex-0.4.5}/examples/09_feature_engineering.py +2 -2
- {dataenginex-0.4.3 → dataenginex-0.4.5}/examples/10_model_analysis.py +3 -3
- {dataenginex-0.4.3 → dataenginex-0.4.5}/examples/GUIDE.md +1 -1
- {dataenginex-0.4.3 → dataenginex-0.4.5}/examples/dashboard/dashboard_config.yaml +1 -1
- {dataenginex-0.4.3 → dataenginex-0.4.5}/examples/dashboard/run_dashboard.py +1 -1
- {dataenginex-0.4.3 → dataenginex-0.4.5}/poe_tasks.toml +2 -3
- {dataenginex-0.4.3 → dataenginex-0.4.5}/pyproject.toml +8 -4
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/__init__.py +1 -1
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/_json.py +8 -9
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ai/__init__.py +1 -35
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ai/memory/__init__.py +0 -3
- dataenginex-0.4.5/src/dataenginex/ai/observability/__init__.py +9 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ai/observability/metrics.py +6 -6
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ai/runtime/__init__.py +0 -3
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ai/runtime/sandbox.py +1 -1
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/api/pagination.py +1 -1
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/core/exceptions.py +3 -46
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/core/medallion_architecture.py +1 -1
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/core/schemas.py +1 -1
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/core/validators.py +105 -105
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/data/connectors/dbt.py +1 -1
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/data/connectors/http.py +2 -5
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/data/connectors/spark.py +2 -2
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/data/pipeline/__init__.py +0 -3
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/data/pipeline/runner.py +5 -2
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/data/registry.py +1 -1
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/engine.py +17 -18
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/lakehouse/partitioning.py +109 -109
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/lakehouse/storage.py +1 -1
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ml/mlflow_registry.py +2 -2
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/conftest.py +162 -162
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/fixtures/sample_data.py +2 -2
- dataenginex-0.4.5/tests/integration/__init__.py +1 -0
- dataenginex-0.4.5/tests/unit/__init__.py +1 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_core_schemas_extended.py +1 -16
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_llm.py +2 -2
- {dataenginex-0.4.3 → dataenginex-0.4.5}/uv.lock +389 -427
- dataenginex-0.4.3/.github/workflows/auto-pr.yml +0 -23
- dataenginex-0.4.3/.github/workflows/enforce-dev-to-main.yml +0 -11
- dataenginex-0.4.3/Dockerfile +0 -53
- dataenginex-0.4.3/docs/ci-cd.md +0 -361
- dataenginex-0.4.3/src/dataenginex/ai/observability/__init__.py +0 -15
- dataenginex-0.4.3/tests/integration/__init__.py +0 -1
- dataenginex-0.4.3/tests/unit/__init__.py +0 -1
- {dataenginex-0.4.3 → dataenginex-0.4.5}/.claude/settings.json +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/.env.template +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/.github/ISSUE_TEMPLATE/bug_report.yml +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/.github/ISSUE_TEMPLATE/config.yml +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/.github/ISSUE_TEMPLATE/feature_request.yml +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/.github/PULL_REQUEST_TEMPLATE.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/.github/labels.yml +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/.github/workflows/docs-sync.yml +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/.gitignore +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/.gitleaks.toml +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/.python-version +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/CHANGELOG.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/CODEOWNERS +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/LICENSE +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/SECURITY.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/docs/adr/0000-template.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/docs/api-reference/ai.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/docs/api-reference/api.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/docs/api-reference/config.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/docs/api-reference/core.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/docs/api-reference/data.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/docs/api-reference/engine.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/docs/api-reference/index.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/docs/api-reference/lakehouse.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/docs/api-reference/middleware.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/docs/api-reference/ml.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/docs/api-reference/orchestration.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/docs/api-reference/plugins.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/docs/api-reference/secops.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/docs/api-reference/warehouse.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/docs/index.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/docs/observability.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/docs/release-notes.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/examples/03_quality_gate.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/examples/04_ml_training.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/examples/05_rag_demo.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/examples/06_llm_quickstart.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/examples/data/events.csv +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/examples/data/users.csv +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/examples/dex.yaml +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/examples/ecommerce/data/customers.csv +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/examples/ecommerce/data/orders.csv +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/examples/ecommerce/data/products.csv +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/examples/ecommerce/dex.yaml +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/examples/ecommerce/run_all.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/examples/movies.csv +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/scripts/localstack/init.sh +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/README.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/RELEASE_NOTES.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ai/agents/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ai/agents/builtin.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ai/llm.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ai/memory/base.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ai/memory/episodic.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ai/memory/long_term.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ai/observability/audit.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ai/observability/cost.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ai/retrieval/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ai/retrieval/builtin.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ai/retrieval/graph.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ai/routing/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ai/routing/anthropic.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ai/routing/guarded.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ai/routing/ollama.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ai/routing/openai.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ai/routing/router.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ai/runtime/checkpoint.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ai/runtime/executor.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ai/tools/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ai/tools/builtin.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ai/vectorstore.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ai/workflows/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ai/workflows/conditions.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ai/workflows/dag.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ai/workflows/human_loop.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/api/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/api/errors.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/api/schemas.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/cli/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/cli/main.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/cli/run.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/cli/secops.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/cli/train.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/config/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/config/defaults.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/config/loader.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/config/schema.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/config/settings.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/core/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/core/interfaces.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/core/quality.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/core/registry.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/data/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/data/connectors/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/data/connectors/_utils.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/data/connectors/csv.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/data/connectors/delta.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/data/connectors/duckdb.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/data/connectors/parquet.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/data/connectors/rest.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/data/connectors/sse.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/data/pipeline/dag.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/data/pipeline/run_history.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/data/profiler.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/data/quality/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/data/quality/gates.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/data/quality/spark.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/data/transforms/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/data/transforms/sql.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/lakehouse/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/lakehouse/catalog.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/middleware/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/middleware/domain_metrics.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/middleware/logging_config.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/middleware/metrics.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ml/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ml/drift.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ml/features/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ml/features/builtin.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ml/metrics.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ml/registry.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ml/serving.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ml/serving_engine/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ml/serving_engine/builtin.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ml/tracking/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ml/tracking/builtin.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/ml/training.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/orchestration/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/orchestration/builtin.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/orchestration/scheduler.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/plugins/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/plugins/registry.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/py.typed +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/secops/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/secops/audit.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/secops/gate.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/secops/guard.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/secops/masking.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/secops/pii.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/store.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/warehouse/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/warehouse/lineage.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/warehouse/transforms.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/src/dataenginex/worker.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/conformance/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/conformance/test_connector.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/conformance/test_feature_store.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/conformance/test_tracker.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/conformance/test_transform.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/fixtures/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/fixtures/sample_jobs.csv +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/fixtures/sample_jobs.json +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/integration/test_ai_integration.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/integration/test_cli_run.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/integration/test_config_cli.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/integration/test_lineage_integration.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/integration/test_ml_integration.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/integration/test_pipeline_e2e.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/integration/test_secops_integration.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/integration/test_storage_real.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/load/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_agent_runtime.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_ai_modules.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_api_pagination.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_api_schemas.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_api_validators.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_builtin_agent.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_builtin_feature_store.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_builtin_serving.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_builtin_tracker.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_cli_train.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_config_loader.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_config_schema.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_config_schema_extended.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_core_exceptions.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_core_interfaces.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_core_quality.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_core_registry.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_csv_connector.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_data.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_dbt_connector.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_dex_engine.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_dex_store.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_domain_metrics.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_domain_metrics_wiring.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_drift_scheduler.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_duckdb_connector.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_guarded_provider.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_lakehouse.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_llm_extended.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_llm_litellm_vllm.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_logging.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_medallion.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_medallion_extended.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_middleware_metrics.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_ml.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_parquet_connector.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_pipeline_dag.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_pipeline_runner.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_plugins.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_privacy_guard_wiring.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_quality_gates.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_quality_spark.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_retriever.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_retriever_graph.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_run_history.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_scheduler.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_secops.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_secops_engine_and_cli.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_secops_guard.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_spark_connector.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_spark_fixtures.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_sql_transforms.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_storage_abstraction.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_vectorstore.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_vectorstore_extended.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_warehouse.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.5}/tests/unit/test_warehouse_transforms.py +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
|
-
# Validate (
|
|
1
|
+
# Validate (DataEngineX)
|
|
2
2
|
|
|
3
|
-
Run the full
|
|
3
|
+
Run the full DataEngineX validation pipeline in order. Stop and report on first failure.
|
|
4
4
|
|
|
5
5
|
1. **Lint**
|
|
6
6
|
|
|
@@ -25,10 +25,11 @@ Run the full DEX validation pipeline in order. Stop and report on first failure.
|
|
|
25
25
|
```bash
|
|
26
26
|
uv run python examples/02_api_quickstart.py &
|
|
27
27
|
sleep 2
|
|
28
|
-
|
|
29
|
-
curl -sf http://localhost:
|
|
30
|
-
curl -sf http://localhost:
|
|
31
|
-
curl -sf
|
|
28
|
+
# dataenginex has no built-in HTTP server; these endpoints apply to dex-studio
|
|
29
|
+
curl -sf http://localhost:7860/health | python -m json.tool
|
|
30
|
+
curl -sf http://localhost:7860/ | python -m json.tool
|
|
31
|
+
curl -sf http://localhost:7860/metrics | head -20
|
|
32
|
+
curl -sf -X POST http://localhost:7860/echo -H "Content-Type: application/json" -d '{"message":"test"}' | python -m json.tool
|
|
32
33
|
```
|
|
33
34
|
|
|
34
35
|
Check response **bodies** — not just status codes. Kill the server after.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Release PR Template
|
|
2
2
|
|
|
3
|
-
Use this template whenever opening a release PR
|
|
3
|
+
Use this template whenever opening a release PR to `main`. Reference `release-notes.md` and verify QA steps before merging.
|
|
4
4
|
|
|
5
5
|
## Summary
|
|
6
6
|
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
name: Auto PR
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
push:
|
|
5
|
+
branches:
|
|
6
|
+
- '*'
|
|
7
|
+
|
|
8
|
+
permissions:
|
|
9
|
+
contents: write
|
|
10
|
+
pull-requests: write
|
|
11
|
+
|
|
12
|
+
jobs:
|
|
13
|
+
auto-pr:
|
|
14
|
+
if: github.ref_name != 'main'
|
|
15
|
+
uses: TheDataEngineX/.github/.github/workflows/reusable-auto-pr-to-main.yml@main
|
|
16
|
+
secrets: inherit
|
|
@@ -2,9 +2,9 @@ name: CI
|
|
|
2
2
|
|
|
3
3
|
on:
|
|
4
4
|
push:
|
|
5
|
-
branches: [main
|
|
5
|
+
branches: [main]
|
|
6
6
|
pull_request:
|
|
7
|
-
branches: [main
|
|
7
|
+
branches: [main]
|
|
8
8
|
workflow_dispatch:
|
|
9
9
|
|
|
10
10
|
schedule:
|
|
@@ -27,8 +27,12 @@ jobs:
|
|
|
27
27
|
- run: uv sync --group ml
|
|
28
28
|
env:
|
|
29
29
|
UV_PROJECT_ENVIRONMENT: .venv
|
|
30
|
+
- uses: reviewdog/action-actionlint@v1
|
|
31
|
+
|
|
30
32
|
- run: uv run poe quality
|
|
31
33
|
|
|
34
|
+
- run: uv build
|
|
35
|
+
|
|
32
36
|
test:
|
|
33
37
|
name: Tests (Python 3.13)
|
|
34
38
|
runs-on: ubuntu-latest
|
|
@@ -2,9 +2,9 @@ name: Security Scans
|
|
|
2
2
|
|
|
3
3
|
on:
|
|
4
4
|
push:
|
|
5
|
-
branches: [main
|
|
5
|
+
branches: [main]
|
|
6
6
|
pull_request:
|
|
7
|
-
branches: [main
|
|
7
|
+
branches: [main]
|
|
8
8
|
workflow_dispatch:
|
|
9
9
|
|
|
10
10
|
permissions:
|
|
@@ -13,5 +13,5 @@ permissions:
|
|
|
13
13
|
|
|
14
14
|
jobs:
|
|
15
15
|
security:
|
|
16
|
-
uses: TheDataEngineX/.github/.github/workflows/security.yml@main
|
|
16
|
+
uses: TheDataEngineX/.github/.github/workflows/reusable-security.yml@main
|
|
17
17
|
secrets: inherit
|
|
@@ -1,45 +1,50 @@
|
|
|
1
|
-
# Pre-commit hooks for DataEngineX
|
|
2
|
-
# Install: pre-commit install
|
|
3
|
-
# Run all: pre-commit run --all-files
|
|
4
|
-
|
|
5
|
-
repos:
|
|
6
|
-
- repo: https://github.com/gitleaks/gitleaks
|
|
7
|
-
rev: v8.30.1
|
|
8
|
-
hooks:
|
|
9
|
-
- id: gitleaks
|
|
10
|
-
|
|
11
|
-
- repo: https://github.com/pre-commit/pre-commit-hooks
|
|
12
|
-
rev: v6.0.0
|
|
13
|
-
hooks:
|
|
14
|
-
- id: trailing-whitespace
|
|
15
|
-
- id: end-of-file-fixer
|
|
16
|
-
- id: check-yaml
|
|
17
|
-
args: ["--allow-multiple-documents"]
|
|
18
|
-
- id: check-toml
|
|
19
|
-
- id: check-added-large-files
|
|
20
|
-
args: ["--maxkb=500"]
|
|
21
|
-
- id: check-merge-conflict
|
|
22
|
-
- id: debug-statements
|
|
23
|
-
|
|
24
|
-
- repo: https://github.com/hukkin/mdformat
|
|
25
|
-
rev: 1.0.0
|
|
26
|
-
hooks:
|
|
27
|
-
- id: mdformat
|
|
28
|
-
exclude: '^(site/|\.github/)'
|
|
29
|
-
|
|
30
|
-
- repo: https://github.com/astral-sh/ruff-pre-commit
|
|
31
|
-
rev: v0.15.
|
|
32
|
-
hooks:
|
|
33
|
-
- id: ruff
|
|
34
|
-
args: ["--fix"]
|
|
35
|
-
- id: ruff-format
|
|
36
|
-
|
|
37
|
-
- repo: local
|
|
38
|
-
hooks:
|
|
39
|
-
- id: mypy
|
|
40
|
-
name: mypy
|
|
41
|
-
entry: uv run mypy
|
|
42
|
-
language: system
|
|
43
|
-
args:
|
|
44
|
-
|
|
45
|
-
|
|
1
|
+
# Pre-commit hooks for DataEngineX
|
|
2
|
+
# Install: pre-commit install
|
|
3
|
+
# Run all: pre-commit run --all-files
|
|
4
|
+
|
|
5
|
+
repos:
|
|
6
|
+
- repo: https://github.com/gitleaks/gitleaks
|
|
7
|
+
rev: v8.30.1
|
|
8
|
+
hooks:
|
|
9
|
+
- id: gitleaks
|
|
10
|
+
|
|
11
|
+
- repo: https://github.com/pre-commit/pre-commit-hooks
|
|
12
|
+
rev: v6.0.0
|
|
13
|
+
hooks:
|
|
14
|
+
- id: trailing-whitespace
|
|
15
|
+
- id: end-of-file-fixer
|
|
16
|
+
- id: check-yaml
|
|
17
|
+
args: ["--allow-multiple-documents"]
|
|
18
|
+
- id: check-toml
|
|
19
|
+
- id: check-added-large-files
|
|
20
|
+
args: ["--maxkb=500"]
|
|
21
|
+
- id: check-merge-conflict
|
|
22
|
+
- id: debug-statements
|
|
23
|
+
|
|
24
|
+
- repo: https://github.com/hukkin/mdformat
|
|
25
|
+
rev: 1.0.0
|
|
26
|
+
hooks:
|
|
27
|
+
- id: mdformat
|
|
28
|
+
exclude: '^(site/|\.github/|\.venv/|\.pytest_cache/|examples/)'
|
|
29
|
+
|
|
30
|
+
- repo: https://github.com/astral-sh/ruff-pre-commit
|
|
31
|
+
rev: v0.15.20
|
|
32
|
+
hooks:
|
|
33
|
+
- id: ruff
|
|
34
|
+
args: ["--fix"]
|
|
35
|
+
- id: ruff-format
|
|
36
|
+
|
|
37
|
+
- repo: local
|
|
38
|
+
hooks:
|
|
39
|
+
- id: mypy
|
|
40
|
+
name: mypy
|
|
41
|
+
entry: uv run mypy
|
|
42
|
+
language: system
|
|
43
|
+
args:
|
|
44
|
+
[
|
|
45
|
+
"--config-file=pyproject.toml",
|
|
46
|
+
"--no-incremental",
|
|
47
|
+
"src/dataenginex",
|
|
48
|
+
]
|
|
49
|
+
pass_filenames: false
|
|
50
|
+
files: ^src/dataenginex/
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
# CLAUDE.md —
|
|
1
|
+
# CLAUDE.md — DataEngineX (dataenginex)
|
|
2
2
|
|
|
3
3
|
Brief answers only. No explanations unless asked.
|
|
4
4
|
Goal is to save Claude code tokens for lower cost without losing quality.
|
|
@@ -7,7 +7,7 @@ Goal is to save Claude code tokens for lower cost without losing quality.
|
|
|
7
7
|
|
|
8
8
|
## Project Overview
|
|
9
9
|
|
|
10
|
-
**
|
|
10
|
+
**DataEngineX** — unified Data + ML + AI library. Config-driven, self-hosted, local-first. Pure Python — no bundled HTTP server.
|
|
11
11
|
|
|
12
12
|
| Package | Location | Purpose |
|
|
13
13
|
|---------|----------|---------|
|
|
@@ -41,7 +41,7 @@ dex validate dex.yaml # Validate config file
|
|
|
41
41
|
dex version # Show version + environment
|
|
42
42
|
|
|
43
43
|
# Dev
|
|
44
|
-
uv run poe dev # Dev server (uvicorn reload, port 17000) — for examples/API testing only
|
|
44
|
+
uv run poe dev # Dev server (uvicorn reload, port 17000) — for examples/API testing only (dataenginex has no built-in HTTP server; see dex-studio for web UI)
|
|
45
45
|
uv run poe docker-up # Docker compose up
|
|
46
46
|
uv run poe docker-down # Docker compose down
|
|
47
47
|
|
|
@@ -63,7 +63,7 @@ pip install 'litellm>=1.83.3' --no-deps # LLM routing (separate: pins python-dot
|
|
|
63
63
|
|
|
64
64
|
______________________________________________________________________
|
|
65
65
|
|
|
66
|
-
##
|
|
66
|
+
## Validation
|
|
67
67
|
|
|
68
|
-
|
|
69
|
-
Tests passing ≠ app working.
|
|
68
|
+
After any code change run: `uv run poe check-all` (lint + typecheck + test).
|
|
69
|
+
Tests passing ≠ app working — run `dex validate dex.yaml` to verify config.
|
|
@@ -54,4 +54,4 @@ All complaints will be reviewed and investigated promptly and fairly.
|
|
|
54
54
|
|
|
55
55
|
This Code of Conduct is adapted from the [Contributor Covenant](https://www.contributor-covenant.org),
|
|
56
56
|
version 2.1, available at
|
|
57
|
-
<https://www.contributor-covenant.org/version/2/1/code_of_conduct.html>.
|
|
57
|
+
<https://www.contributor-covenant.org/version/2/1/code_of_conduct.html>.
|
|
@@ -1,18 +1,18 @@
|
|
|
1
|
-
# Contributing to
|
|
1
|
+
# Contributing to DataEngineX
|
|
2
2
|
|
|
3
3
|
Thank you for your interest in contributing to DataEngineX!
|
|
4
4
|
|
|
5
|
-
**→ Full contributor guide: [docs/
|
|
5
|
+
**→ Full contributor guide: [docs/contributing.md](docs/contributing.md)**
|
|
6
6
|
|
|
7
7
|
Quick essentials:
|
|
8
8
|
|
|
9
|
-
- Development setup: [docs/
|
|
10
|
-
- Governance & legal: See [docs/
|
|
9
|
+
- Development setup: [docs/development.md](docs/development.md)
|
|
10
|
+
- Governance & legal: See [docs/contributing.md](docs/contributing.md)
|
|
11
11
|
- Code of Conduct: [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md)
|
|
12
12
|
|
|
13
13
|
## Quick Start
|
|
14
14
|
|
|
15
|
-
1. Fork and create a feature branch from `
|
|
15
|
+
1. Fork and create a feature branch from `main`
|
|
16
16
|
1. Run local checks: `uv run poe check-all`
|
|
17
17
|
1. Open a PR using `.github/PULL_REQUEST_TEMPLATE.md`
|
|
18
18
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: dataenginex
|
|
3
|
-
Version: 0.4.
|
|
3
|
+
Version: 0.4.5
|
|
4
4
|
Summary: DataEngineX — open-source, self-hosted, local-first Data + ML + AI workbench library
|
|
5
5
|
Author-email: Jay <jayapal.myaka99@gmail.com>
|
|
6
6
|
License: MIT License
|
|
@@ -30,7 +30,6 @@ Requires-Dist: click>=8.3.3
|
|
|
30
30
|
Requires-Dist: croniter>=6.2.2
|
|
31
31
|
Requires-Dist: duckdb>=1.5.2
|
|
32
32
|
Requires-Dist: httpx>=0.28.1
|
|
33
|
-
Requires-Dist: msgpack>=1.2.1
|
|
34
33
|
Requires-Dist: orjson>=3.11.0
|
|
35
34
|
Requires-Dist: prometheus-client>=0.25.0
|
|
36
35
|
Requires-Dist: pyarrow>=23.0.1
|
|
@@ -38,7 +37,6 @@ Requires-Dist: pydantic-settings>=2.14.2
|
|
|
38
37
|
Requires-Dist: pydantic>=2.13.4
|
|
39
38
|
Requires-Dist: pyyaml>=6.0.3
|
|
40
39
|
Requires-Dist: structlog>=25.5.0
|
|
41
|
-
Requires-Dist: zstandard>=0.25.0
|
|
42
40
|
Provides-Extra: cloud
|
|
43
41
|
Requires-Dist: boto3>=1.43.7; extra == 'cloud'
|
|
44
42
|
Requires-Dist: google-cloud-bigquery>=3.41.0; extra == 'cloud'
|
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
# dataenginex
|
|
2
2
|
|
|
3
|
-
[](https://github.com/TheDataEngineX/dataenginex/actions/workflows/ci.yml)
|
|
4
4
|
[](https://pypi.org/project/dataenginex/)
|
|
5
5
|
[](https://www.python.org/downloads/)
|
|
6
6
|
[](LICENSE)
|
|
7
7
|
|
|
8
|
-
The Python library that powers [
|
|
8
|
+
The Python library that powers [DataEngineX Studio](https://github.com/TheDataEngineX/dex-studio) — an open-source, self-hosted, local-first Data + ML + AI workbench for individuals and small teams. **Use the library directly when you want code; install DataEngineX Studio when you want a UI.**
|
|
9
9
|
|
|
10
10
|
______________________________________________________________________
|
|
11
11
|
|
|
@@ -116,7 +116,7 @@ git clone https://github.com/TheDataEngineX/dex-studio && cd dex-studio
|
|
|
116
116
|
docker compose up # http://localhost:7860
|
|
117
117
|
```
|
|
118
118
|
|
|
119
|
-
|
|
119
|
+
DataEngineX Studio imports `dataenginex` directly — no separate API server, no HTTP hop.
|
|
120
120
|
|
|
121
121
|
______________________________________________________________________
|
|
122
122
|
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
# ---------------------------------------------------------------
|
|
2
|
-
#
|
|
2
|
+
# DataEngineX — Storage Emulators for Integration Tests
|
|
3
3
|
#
|
|
4
4
|
# Spins up LocalStack (S3) and fake-gcs-server (GCS) for local
|
|
5
5
|
# integration testing without real cloud accounts.
|
|
@@ -8,18 +8,18 @@
|
|
|
8
8
|
|
|
9
9
|
## Context
|
|
10
10
|
|
|
11
|
-
|
|
11
|
+
DataEngineX needs a scalable, standardized data architecture that works across all projects (CareerDEX, Weather, etc.). The data pipeline needs to handle:
|
|
12
12
|
|
|
13
13
|
- Raw data ingestion from multiple sources
|
|
14
14
|
- Data transformation and quality validation
|
|
15
15
|
- Feature engineering for ML models
|
|
16
16
|
- Analytics-ready data consumption
|
|
17
17
|
|
|
18
|
-
Multiple projects will be built on the
|
|
18
|
+
Multiple projects will be built on the DataEngineX platform, each with different data sources and transformations. We need a consistent, reusable pattern.
|
|
19
19
|
|
|
20
20
|
## Decision
|
|
21
21
|
|
|
22
|
-
Implement the **Medallion Architecture** (Bronze → Silver → Gold layers) as the standard data architecture for all
|
|
22
|
+
Implement the **Medallion Architecture** (Bronze → Silver → Gold layers) as the standard data architecture for all DataEngineX projects.
|
|
23
23
|
|
|
24
24
|
### Architecture Layers
|
|
25
25
|
|
|
@@ -5,7 +5,7 @@
|
|
|
5
5
|
**DataEngineX** is a unified Data + ML + AI **library** that wires industry tools through a
|
|
6
6
|
single config-driven interface. One `dex.yaml` defines the entire project.
|
|
7
7
|
|
|
8
|
-
**Design principle:** Pure Python library — no HTTP server bundled. Your application (
|
|
8
|
+
**Design principle:** Pure Python library — no HTTP server bundled. Your application (DataEngineX Studio,
|
|
9
9
|
your own FastAPI/Flask app, a script) imports `dataenginex` and owns the server layer.
|
|
10
10
|
|
|
11
11
|
## Architecture
|
|
@@ -56,9 +56,9 @@ dex.yaml
|
|
|
56
56
|
Every subsystem follows the same pattern:
|
|
57
57
|
|
|
58
58
|
1. **ABC** in `core/interfaces.py` — defines the contract (e.g. `BaseConnector`)
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
59
|
+
1. **BackendRegistry[T]** in `core/registry.py` — discovers and registers implementations
|
|
60
|
+
1. **Built-in** implements the ABC with zero external deps
|
|
61
|
+
1. **Extras** implement the same ABC, swapped in via config
|
|
62
62
|
|
|
63
63
|
```python
|
|
64
64
|
from dataenginex.core.registry import BackendRegistry
|
|
@@ -108,25 +108,22 @@ DataEngineXError
|
|
|
108
108
|
├── ConfigError → ConfigValidationError
|
|
109
109
|
├── PipelineError → PipelineStepError
|
|
110
110
|
├── RegistryError
|
|
111
|
-
|
|
112
|
-
├── TrainingError
|
|
113
|
-
├── ServingError
|
|
114
|
-
└── AgentError → LLMProviderError
|
|
111
|
+
└── BackendNotInstalledError
|
|
115
112
|
```
|
|
116
113
|
|
|
117
114
|
## Module Map
|
|
118
115
|
|
|
119
116
|
| Module | Purpose |
|
|
120
|
-
|
|
117
|
+
| ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
121
118
|
| `engine.py` | `DexEngine` — application entry point |
|
|
122
119
|
| `store.py` | `DexStore` — DuckDB persistence layer |
|
|
123
120
|
| `config/` | Schema, loader, env resolution |
|
|
124
121
|
| `core/` | ABCs, registry, exceptions |
|
|
125
122
|
| `cli/` | `dex` CLI (validate, version, init) |
|
|
126
123
|
| `api/` | HTTP helpers: error types, response models |
|
|
127
|
-
| `data/connectors/` | Built-in connectors: CSV, Parquet, DuckDB, REST, Kafka, **Spark**, **dbt** |
|
|
124
|
+
| `data/connectors/` | Built-in connectors: CSV, Parquet, DuckDB, REST, Kafka, **Spark**, **dbt**, **delta**, **postgres**, **qdrant**, **sse**, **http**, **rest** |
|
|
128
125
|
| `data/pipeline/` | Pipeline runner, transforms, quality, profiler |
|
|
129
|
-
| `ml/` | Classical ML: training, registry, serving, drift |
|
|
126
|
+
| `ml/` | Classical ML: training, registry, serving, drift, **feature engines**, **mlflow registry** |
|
|
130
127
|
| `ai/` | LLM, agents, RAG, vectorstore, memory, observability |
|
|
131
128
|
| `orchestration/` | DriftScheduler, background tasks |
|
|
132
129
|
| `middleware/` | structlog config, Prometheus metrics |
|
|
@@ -138,26 +135,66 @@ DataEngineXError
|
|
|
138
135
|
## Tech Stack
|
|
139
136
|
|
|
140
137
|
| Component | Built-in | Extra |
|
|
141
|
-
|
|
138
|
+
| ----------------- | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
|
|
142
139
|
| Data Engine | DuckDB | PySpark / dbt CLI (`[data]`) |
|
|
143
140
|
| Orchestration | croniter scheduler | — |
|
|
144
141
|
| ML Tracking | JSON-based | MLflow (`[tracking]`) |
|
|
145
142
|
| Model Serving | Built-in predictor | — |
|
|
146
143
|
| LLM Provider | Ollama / vLLM | LiteLLM (install separately) |
|
|
147
|
-
| Vector Store | DuckDB VSS | Qdrant |
|
|
144
|
+
| Vector Store | DuckDB VSS | Qdrant (`[qdrant]`) |
|
|
148
145
|
| Retrieval | BM25 + Dense + Hybrid | — |
|
|
149
|
-
| Persistence | DuckDB |
|
|
146
|
+
| Persistence | DuckDB | S3/GCS/BigQuery (`[cloud]`) |
|
|
150
147
|
| Logging | structlog | — |
|
|
151
148
|
| Config | Pydantic + YAML | — |
|
|
152
149
|
| CLI | Click | — |
|
|
153
150
|
| Privacy / Audit | PrivacyGuard — PII masking + audit | — |
|
|
154
151
|
| LLM Observability | — | Langfuse (`[observability]`) |
|
|
155
152
|
| Cloud Storage | — | S3/GCS/BigQuery (`[cloud]`) |
|
|
153
|
+
| Connectors | CSV, Parquet, DuckDB, SSE, HTTP (REST, SSE), JSON | Spark, dbt, Delta Lake (`[delta]`), PostgreSQL (`[postgres]`), Qdrant (`[qdrant]`) |
|
|
154
|
+
| ML | Basic | PyTorch (`[pytorch]`), scikit-learn (`[ml]`), sentence-transformers (`[ml]`), MLflow (`[ml]` + `[tracking]`) |
|
|
155
|
+
|
|
156
|
+
## Coverage Strategy
|
|
157
|
+
|
|
158
|
+
**Current Coverage**: 81% (meets 80% threshold)
|
|
159
|
+
|
|
160
|
+
**Why Coverage is Not 100%**: Optional dependency files are excluded from coverage to keep CI fast. Tests for these run only when the optional extras are installed.
|
|
161
|
+
|
|
162
|
+
```python
|
|
163
|
+
omit = [
|
|
164
|
+
# Network & async connectors (require external deps)
|
|
165
|
+
"*/src/dataenginex/data/connectors/http.py",
|
|
166
|
+
"*/src/dataenginex/data/connectors/rest.py",
|
|
167
|
+
"*/src/dataenginex/data/connectors/sse.py",
|
|
168
|
+
|
|
169
|
+
# ML dependencies (require training packages)
|
|
170
|
+
"*/src/dataenginex/ml/mlflow_registry.py",
|
|
171
|
+
|
|
172
|
+
# Optional data connectors (require cloud tools)
|
|
173
|
+
"*/src/dataenginex/data/connectors/delta.py",
|
|
174
|
+
"*/src/dataenginex/data/connectors/postgres.py", # Note: this may not exist, but if it does
|
|
175
|
+
|
|
176
|
+
# Other excluded (from pyproject.toml)
|
|
177
|
+
"*/src/dataenginex/data/connectors/delta.py",
|
|
178
|
+
"*/src/dataenginex/lakehouse/storage.py",
|
|
179
|
+
"*/src/dataenginex/worker.py",
|
|
180
|
+
]
|
|
181
|
+
```
|
|
182
|
+
|
|
183
|
+
**To install optional dependencies and achieve >90% coverage**:
|
|
184
|
+
|
|
185
|
+
```bash
|
|
186
|
+
uv run poe uv-sync
|
|
187
|
+
pip install "dataenginex[cloud]" "dataenginex[delta]" "dataenginex[postgres]" \
|
|
188
|
+
"dataenginex[qdrant]" "dataenginex[queue]" "dataenginex[pytorch]" \
|
|
189
|
+
"dataenginex[notebook]" "dataenginex[ml]" "dataenginex[tracking]" "dataenginex[data]"
|
|
190
|
+
uv run poe uv-sync
|
|
191
|
+
uv run poe test-cov
|
|
192
|
+
```
|
|
156
193
|
|
|
157
194
|
## Key Design Decisions
|
|
158
195
|
|
|
159
196
|
| ID | Decision | Rationale |
|
|
160
|
-
|
|
197
|
+
| ---- | ---------------------------------------------- | ------------------------------------------------------------- |
|
|
161
198
|
| AD1 | Pure library — no bundled HTTP server | Applications own the server layer; library stays lean |
|
|
162
199
|
| AD2 | DexEngine as single entry point | One object to instantiate; hides wiring complexity |
|
|
163
200
|
| AD3 | DuckDB for persistence | Embedded, zero-ops, single file next to dex.yaml |
|
|
@@ -0,0 +1,97 @@
|
|
|
1
|
+
# CI/CD Pipeline
|
|
2
|
+
|
|
3
|
+
**Continuous integration and release automation for dataenginex.**
|
|
4
|
+
|
|
5
|
+
______________________________________________________________________
|
|
6
|
+
|
|
7
|
+
## Overview
|
|
8
|
+
|
|
9
|
+
dataenginex is a pure Python library published to PyPI. The pipeline is:
|
|
10
|
+
|
|
11
|
+
- **CI**: Linting, type checking, testing, and security scanning on every PR to `main`.
|
|
12
|
+
- **Release**: Push a `v{X.Y.Z}` tag to `main` → `release.yml` builds, publishes to PyPI, and creates a GitHub Release.
|
|
13
|
+
|
|
14
|
+
```mermaid
|
|
15
|
+
graph LR
|
|
16
|
+
Dev[Developer] --> PR[Open PR to main]
|
|
17
|
+
PR --> CI[CI: Lint/Test/Security]
|
|
18
|
+
CI --> Review[Code Review]
|
|
19
|
+
Review --> Merge["Merge to main"]
|
|
20
|
+
Merge --> Tag["Push tag vX.Y.Z"]
|
|
21
|
+
Tag --> Release[release.yml]
|
|
22
|
+
Release --> Build["Build wheel + sdist"]
|
|
23
|
+
Build --> PyPI["Publish to PyPI (OIDC)"]
|
|
24
|
+
Build --> GHRelease["GitHub Release + CycloneDX SBOM"]
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
______________________________________________________________________
|
|
28
|
+
|
|
29
|
+
## Continuous Integration (CI)
|
|
30
|
+
|
|
31
|
+
**Workflow**: `.github/workflows/ci.yml`
|
|
32
|
+
|
|
33
|
+
**Triggers**:
|
|
34
|
+
|
|
35
|
+
- Push to `main`
|
|
36
|
+
- Pull requests targeting `main`
|
|
37
|
+
|
|
38
|
+
**Jobs**:
|
|
39
|
+
|
|
40
|
+
### 1. Quality (`quality`)
|
|
41
|
+
|
|
42
|
+
```bash
|
|
43
|
+
uv run ruff check src/ tests/
|
|
44
|
+
uv run mypy src/dataenginex/ --strict
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
### 2. Test (`test`)
|
|
48
|
+
|
|
49
|
+
```bash
|
|
50
|
+
uv run pytest tests/ -x --tb=short --cov=src/dataenginex/
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
Coverage threshold: 80%.
|
|
54
|
+
|
|
55
|
+
### 3. Package validation
|
|
56
|
+
|
|
57
|
+
```bash
|
|
58
|
+
uv build
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
### 4. Security Scans
|
|
62
|
+
|
|
63
|
+
Runs via the shared reusable workflow at `.github/workflows/security.yml`:
|
|
64
|
+
|
|
65
|
+
- **Trivy**: Misconfig and secret scan — results uploaded to GitHub Security tab.
|
|
66
|
+
- **CodeQL**: Handled by GitHub's default setup.
|
|
67
|
+
|
|
68
|
+
______________________________________________________________________
|
|
69
|
+
|
|
70
|
+
## Release Automation
|
|
71
|
+
|
|
72
|
+
**Workflow**: `.github/workflows/release.yml`
|
|
73
|
+
|
|
74
|
+
**Trigger**: Push a tag matching `v[0-9]+.[0-9]+.[0-9]+` to `main`.
|
|
75
|
+
|
|
76
|
+
**Jobs**:
|
|
77
|
+
|
|
78
|
+
1. **build** — `uv build` → upload wheel + sdist
|
|
79
|
+
1. **publish-pypi** — `pypa/gh-action-pypi-publish` (OIDC trusted publishing)
|
|
80
|
+
1. **github-release** — CycloneDX SBOM → `gh release create`
|
|
81
|
+
|
|
82
|
+
**How to release**:
|
|
83
|
+
|
|
84
|
+
```bash
|
|
85
|
+
git tag v1.2.3
|
|
86
|
+
git push origin v1.2.3
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
______________________________________________________________________
|
|
90
|
+
|
|
91
|
+
## Workflows Overview
|
|
92
|
+
|
|
93
|
+
| Workflow | Trigger | Purpose |
|
|
94
|
+
| --- | --- | --- |
|
|
95
|
+
| **CI** | Push/PR to main | Lint + typecheck + test + security |
|
|
96
|
+
| **Security** | Push/PR to main | Trivy misconfig + secret scan |
|
|
97
|
+
| **Release** | Push tag `v*.*.*` to main | Build → PyPI → GitHub Release |
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
# Contributing to
|
|
1
|
+
# Contributing to DataEngineX
|
|
2
2
|
|
|
3
3
|
Thank you for contributing!
|
|
4
4
|
|
|
@@ -6,7 +6,7 @@ Thank you for contributing!
|
|
|
6
6
|
|
|
7
7
|
1. Read [development.md](./development.md) for setup instructions
|
|
8
8
|
1. Fork the repository
|
|
9
|
-
1. Create a feature branch from `
|
|
9
|
+
1. Create a feature branch from `main`
|
|
10
10
|
1. Make your changes
|
|
11
11
|
1. Submit a pull request
|
|
12
12
|
|
|
@@ -68,7 +68,7 @@ Use semantic commit format:
|
|
|
68
68
|
|
|
69
69
|
- This project is open source under MIT; keep license and attribution notices in redistributions.
|
|
70
70
|
- Forks are welcome, but should use a distinct public name when redistributed as a separate project.
|
|
71
|
-
- Do not present a fork as the official DataEngineX
|
|
71
|
+
- Do not present a fork as the official DataEngineX project.
|
|
72
72
|
- See the project's license and brand guidelines for brand-usage details.
|
|
73
73
|
|
|
74
74
|
## Code Reviews
|
|
@@ -94,7 +94,7 @@ uv run poe check-all # Run all checks
|
|
|
94
94
|
- `enhancement` - New feature or improvement
|
|
95
95
|
- `good first issue` - Good for newcomers
|
|
96
96
|
- `P1-high` / `P2-medium` - Priority levels
|
|
97
|
-
- `dex-module` - Core
|
|
97
|
+
- `dex-module` - Core DataEngineX infrastructure
|
|
98
98
|
|
|
99
99
|
## Questions?
|
|
100
100
|
|