dataenginex 0.4.3__tar.gz → 0.4.4__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {dataenginex-0.4.3 → dataenginex-0.4.4}/.claude/commands/new-feature.md +1 -1
- {dataenginex-0.4.3 → dataenginex-0.4.4}/.claude/commands/validate.md +7 -6
- {dataenginex-0.4.3 → dataenginex-0.4.4}/.github/dependabot.yml +0 -2
- {dataenginex-0.4.3 → dataenginex-0.4.4}/.github/release-pr-template.md +1 -1
- dataenginex-0.4.4/.github/workflows/auto-pr.yml +16 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/.github/workflows/ci.yml +6 -2
- {dataenginex-0.4.3 → dataenginex-0.4.4}/.github/workflows/release.yml +1 -1
- {dataenginex-0.4.3 → dataenginex-0.4.4}/.github/workflows/security.yml +3 -3
- {dataenginex-0.4.3 → dataenginex-0.4.4}/CLAUDE.md +6 -6
- {dataenginex-0.4.3 → dataenginex-0.4.4}/CONTRIBUTING.md +5 -5
- {dataenginex-0.4.3 → dataenginex-0.4.4}/PKG-INFO +1 -3
- {dataenginex-0.4.3 → dataenginex-0.4.4}/README.md +3 -3
- {dataenginex-0.4.3 → dataenginex-0.4.4}/docker-compose.test.yml +1 -1
- {dataenginex-0.4.3 → dataenginex-0.4.4}/docs/adr/0001-medallion-architecture.md +3 -3
- {dataenginex-0.4.3 → dataenginex-0.4.4}/docs/architecture.md +5 -8
- dataenginex-0.4.4/docs/ci-cd.md +97 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/docs/contributing.md +4 -4
- {dataenginex-0.4.3 → dataenginex-0.4.4}/docs/development.md +6 -6
- {dataenginex-0.4.3 → dataenginex-0.4.4}/docs/quickstart.md +2 -2
- {dataenginex-0.4.3 → dataenginex-0.4.4}/docs/sdlc.md +12 -55
- {dataenginex-0.4.3 → dataenginex-0.4.4}/docs/security-scanning.md +2 -2
- {dataenginex-0.4.3 → dataenginex-0.4.4}/examples/01_hello_pipeline.py +1 -1
- {dataenginex-0.4.3 → dataenginex-0.4.4}/examples/02_api_quickstart.py +1 -1
- {dataenginex-0.4.3 → dataenginex-0.4.4}/examples/07_api_ingestion.py +1 -1
- {dataenginex-0.4.3 → dataenginex-0.4.4}/examples/08_spark_ml.py +3 -3
- {dataenginex-0.4.3 → dataenginex-0.4.4}/examples/09_feature_engineering.py +2 -2
- {dataenginex-0.4.3 → dataenginex-0.4.4}/examples/10_model_analysis.py +3 -3
- {dataenginex-0.4.3 → dataenginex-0.4.4}/examples/dashboard/dashboard_config.yaml +1 -1
- {dataenginex-0.4.3 → dataenginex-0.4.4}/examples/dashboard/run_dashboard.py +1 -1
- {dataenginex-0.4.3 → dataenginex-0.4.4}/poe_tasks.toml +2 -3
- {dataenginex-0.4.3 → dataenginex-0.4.4}/pyproject.toml +3 -4
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/__init__.py +1 -1
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/_json.py +8 -9
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ai/__init__.py +1 -35
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ai/memory/__init__.py +0 -3
- dataenginex-0.4.4/src/dataenginex/ai/observability/__init__.py +9 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ai/observability/metrics.py +6 -6
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ai/runtime/__init__.py +0 -3
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ai/runtime/sandbox.py +1 -1
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/api/pagination.py +1 -1
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/core/exceptions.py +3 -46
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/core/medallion_architecture.py +1 -1
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/core/schemas.py +1 -1
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/core/validators.py +105 -105
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/data/connectors/dbt.py +1 -1
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/data/connectors/spark.py +2 -2
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/data/pipeline/__init__.py +0 -3
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/data/registry.py +1 -1
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/engine.py +1 -14
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/lakehouse/partitioning.py +109 -109
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/lakehouse/storage.py +1 -1
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ml/mlflow_registry.py +2 -2
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/conftest.py +162 -162
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/fixtures/sample_data.py +2 -2
- dataenginex-0.4.4/tests/integration/__init__.py +1 -0
- dataenginex-0.4.4/tests/unit/__init__.py +1 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_core_schemas_extended.py +1 -16
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_llm.py +2 -2
- {dataenginex-0.4.3 → dataenginex-0.4.4}/uv.lock +77 -121
- dataenginex-0.4.3/.github/workflows/auto-pr.yml +0 -23
- dataenginex-0.4.3/.github/workflows/enforce-dev-to-main.yml +0 -11
- dataenginex-0.4.3/Dockerfile +0 -53
- dataenginex-0.4.3/docs/ci-cd.md +0 -361
- dataenginex-0.4.3/src/dataenginex/ai/observability/__init__.py +0 -15
- dataenginex-0.4.3/tests/integration/__init__.py +0 -1
- dataenginex-0.4.3/tests/unit/__init__.py +0 -1
- {dataenginex-0.4.3 → dataenginex-0.4.4}/.claude/settings.json +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/.env.template +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/.github/ISSUE_TEMPLATE/bug_report.yml +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/.github/ISSUE_TEMPLATE/config.yml +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/.github/ISSUE_TEMPLATE/feature_request.yml +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/.github/PULL_REQUEST_TEMPLATE.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/.github/labels.yml +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/.github/workflows/docs-sync.yml +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/.gitignore +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/.gitleaks.toml +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/.pre-commit-config.yaml +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/.python-version +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/CHANGELOG.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/CODEOWNERS +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/CODE_OF_CONDUCT.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/LICENSE +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/SECURITY.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/docs/adr/0000-template.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/docs/api-reference/ai.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/docs/api-reference/api.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/docs/api-reference/config.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/docs/api-reference/core.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/docs/api-reference/data.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/docs/api-reference/engine.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/docs/api-reference/index.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/docs/api-reference/lakehouse.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/docs/api-reference/middleware.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/docs/api-reference/ml.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/docs/api-reference/orchestration.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/docs/api-reference/plugins.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/docs/api-reference/secops.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/docs/api-reference/warehouse.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/docs/index.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/docs/observability.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/docs/release-notes.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/examples/03_quality_gate.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/examples/04_ml_training.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/examples/05_rag_demo.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/examples/06_llm_quickstart.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/examples/GUIDE.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/examples/data/events.csv +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/examples/data/users.csv +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/examples/dex.yaml +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/examples/ecommerce/data/customers.csv +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/examples/ecommerce/data/orders.csv +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/examples/ecommerce/data/products.csv +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/examples/ecommerce/dex.yaml +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/examples/ecommerce/run_all.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/examples/movies.csv +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/scripts/localstack/init.sh +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/README.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/RELEASE_NOTES.md +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ai/agents/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ai/agents/builtin.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ai/llm.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ai/memory/base.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ai/memory/episodic.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ai/memory/long_term.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ai/observability/audit.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ai/observability/cost.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ai/retrieval/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ai/retrieval/builtin.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ai/retrieval/graph.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ai/routing/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ai/routing/anthropic.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ai/routing/guarded.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ai/routing/ollama.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ai/routing/openai.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ai/routing/router.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ai/runtime/checkpoint.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ai/runtime/executor.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ai/tools/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ai/tools/builtin.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ai/vectorstore.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ai/workflows/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ai/workflows/conditions.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ai/workflows/dag.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ai/workflows/human_loop.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/api/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/api/errors.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/api/schemas.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/cli/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/cli/main.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/cli/run.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/cli/secops.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/cli/train.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/config/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/config/defaults.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/config/loader.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/config/schema.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/config/settings.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/core/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/core/interfaces.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/core/quality.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/core/registry.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/data/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/data/connectors/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/data/connectors/_utils.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/data/connectors/csv.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/data/connectors/delta.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/data/connectors/duckdb.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/data/connectors/http.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/data/connectors/parquet.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/data/connectors/rest.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/data/connectors/sse.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/data/pipeline/dag.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/data/pipeline/run_history.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/data/pipeline/runner.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/data/profiler.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/data/quality/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/data/quality/gates.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/data/quality/spark.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/data/transforms/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/data/transforms/sql.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/lakehouse/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/lakehouse/catalog.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/middleware/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/middleware/domain_metrics.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/middleware/logging_config.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/middleware/metrics.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ml/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ml/drift.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ml/features/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ml/features/builtin.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ml/metrics.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ml/registry.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ml/serving.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ml/serving_engine/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ml/serving_engine/builtin.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ml/tracking/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ml/tracking/builtin.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/ml/training.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/orchestration/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/orchestration/builtin.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/orchestration/scheduler.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/plugins/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/plugins/registry.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/py.typed +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/secops/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/secops/audit.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/secops/gate.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/secops/guard.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/secops/masking.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/secops/pii.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/store.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/warehouse/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/warehouse/lineage.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/warehouse/transforms.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/src/dataenginex/worker.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/conformance/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/conformance/test_connector.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/conformance/test_feature_store.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/conformance/test_tracker.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/conformance/test_transform.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/fixtures/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/fixtures/sample_jobs.csv +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/fixtures/sample_jobs.json +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/integration/test_ai_integration.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/integration/test_cli_run.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/integration/test_config_cli.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/integration/test_lineage_integration.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/integration/test_ml_integration.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/integration/test_pipeline_e2e.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/integration/test_secops_integration.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/integration/test_storage_real.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/load/__init__.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_agent_runtime.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_ai_modules.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_api_pagination.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_api_schemas.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_api_validators.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_builtin_agent.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_builtin_feature_store.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_builtin_serving.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_builtin_tracker.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_cli_train.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_config_loader.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_config_schema.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_config_schema_extended.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_core_exceptions.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_core_interfaces.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_core_quality.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_core_registry.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_csv_connector.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_data.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_dbt_connector.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_dex_engine.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_dex_store.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_domain_metrics.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_domain_metrics_wiring.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_drift_scheduler.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_duckdb_connector.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_guarded_provider.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_lakehouse.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_llm_extended.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_llm_litellm_vllm.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_logging.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_medallion.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_medallion_extended.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_middleware_metrics.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_ml.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_parquet_connector.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_pipeline_dag.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_pipeline_runner.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_plugins.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_privacy_guard_wiring.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_quality_gates.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_quality_spark.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_retriever.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_retriever_graph.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_run_history.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_scheduler.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_secops.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_secops_engine_and_cli.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_secops_guard.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_spark_connector.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_spark_fixtures.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_sql_transforms.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_storage_abstraction.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_vectorstore.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_vectorstore_extended.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_warehouse.py +0 -0
- {dataenginex-0.4.3 → dataenginex-0.4.4}/tests/unit/test_warehouse_transforms.py +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
|
-
# Validate (
|
|
1
|
+
# Validate (DataEngineX)
|
|
2
2
|
|
|
3
|
-
Run the full
|
|
3
|
+
Run the full DataEngineX validation pipeline in order. Stop and report on first failure.
|
|
4
4
|
|
|
5
5
|
1. **Lint**
|
|
6
6
|
|
|
@@ -25,10 +25,11 @@ Run the full DEX validation pipeline in order. Stop and report on first failure.
|
|
|
25
25
|
```bash
|
|
26
26
|
uv run python examples/02_api_quickstart.py &
|
|
27
27
|
sleep 2
|
|
28
|
-
|
|
29
|
-
curl -sf http://localhost:
|
|
30
|
-
curl -sf http://localhost:
|
|
31
|
-
curl -sf
|
|
28
|
+
# dataenginex has no built-in HTTP server; these endpoints apply to dex-studio
|
|
29
|
+
curl -sf http://localhost:7860/health | python -m json.tool
|
|
30
|
+
curl -sf http://localhost:7860/ | python -m json.tool
|
|
31
|
+
curl -sf http://localhost:7860/metrics | head -20
|
|
32
|
+
curl -sf -X POST http://localhost:7860/echo -H "Content-Type: application/json" -d '{"message":"test"}' | python -m json.tool
|
|
32
33
|
```
|
|
33
34
|
|
|
34
35
|
Check response **bodies** — not just status codes. Kill the server after.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Release PR Template
|
|
2
2
|
|
|
3
|
-
Use this template whenever opening a release PR
|
|
3
|
+
Use this template whenever opening a release PR to `main`. Reference `release-notes.md` and verify QA steps before merging.
|
|
4
4
|
|
|
5
5
|
## Summary
|
|
6
6
|
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
name: Auto PR
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
push:
|
|
5
|
+
branches:
|
|
6
|
+
- '*'
|
|
7
|
+
|
|
8
|
+
permissions:
|
|
9
|
+
contents: write
|
|
10
|
+
pull-requests: write
|
|
11
|
+
|
|
12
|
+
jobs:
|
|
13
|
+
auto-pr:
|
|
14
|
+
if: github.ref_name != 'main'
|
|
15
|
+
uses: TheDataEngineX/.github/.github/workflows/reusable-auto-pr-to-main.yml@main
|
|
16
|
+
secrets: inherit
|
|
@@ -2,9 +2,9 @@ name: CI
|
|
|
2
2
|
|
|
3
3
|
on:
|
|
4
4
|
push:
|
|
5
|
-
branches: [main
|
|
5
|
+
branches: [main]
|
|
6
6
|
pull_request:
|
|
7
|
-
branches: [main
|
|
7
|
+
branches: [main]
|
|
8
8
|
workflow_dispatch:
|
|
9
9
|
|
|
10
10
|
schedule:
|
|
@@ -27,8 +27,12 @@ jobs:
|
|
|
27
27
|
- run: uv sync --group ml
|
|
28
28
|
env:
|
|
29
29
|
UV_PROJECT_ENVIRONMENT: .venv
|
|
30
|
+
- uses: reviewdog/action-actionlint@v1
|
|
31
|
+
|
|
30
32
|
- run: uv run poe quality
|
|
31
33
|
|
|
34
|
+
- run: uv build
|
|
35
|
+
|
|
32
36
|
test:
|
|
33
37
|
name: Tests (Python 3.13)
|
|
34
38
|
runs-on: ubuntu-latest
|
|
@@ -2,9 +2,9 @@ name: Security Scans
|
|
|
2
2
|
|
|
3
3
|
on:
|
|
4
4
|
push:
|
|
5
|
-
branches: [main
|
|
5
|
+
branches: [main]
|
|
6
6
|
pull_request:
|
|
7
|
-
branches: [main
|
|
7
|
+
branches: [main]
|
|
8
8
|
workflow_dispatch:
|
|
9
9
|
|
|
10
10
|
permissions:
|
|
@@ -13,5 +13,5 @@ permissions:
|
|
|
13
13
|
|
|
14
14
|
jobs:
|
|
15
15
|
security:
|
|
16
|
-
uses: TheDataEngineX/.github/.github/workflows/security.yml@main
|
|
16
|
+
uses: TheDataEngineX/.github/.github/workflows/reusable-security.yml@main
|
|
17
17
|
secrets: inherit
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
# CLAUDE.md —
|
|
1
|
+
# CLAUDE.md — DataEngineX (dataenginex)
|
|
2
2
|
|
|
3
3
|
Brief answers only. No explanations unless asked.
|
|
4
4
|
Goal is to save Claude code tokens for lower cost without losing quality.
|
|
@@ -7,7 +7,7 @@ Goal is to save Claude code tokens for lower cost without losing quality.
|
|
|
7
7
|
|
|
8
8
|
## Project Overview
|
|
9
9
|
|
|
10
|
-
**
|
|
10
|
+
**DataEngineX** — unified Data + ML + AI library. Config-driven, self-hosted, local-first. Pure Python — no bundled HTTP server.
|
|
11
11
|
|
|
12
12
|
| Package | Location | Purpose |
|
|
13
13
|
|---------|----------|---------|
|
|
@@ -41,7 +41,7 @@ dex validate dex.yaml # Validate config file
|
|
|
41
41
|
dex version # Show version + environment
|
|
42
42
|
|
|
43
43
|
# Dev
|
|
44
|
-
uv run poe dev # Dev server (uvicorn reload, port 17000) — for examples/API testing only
|
|
44
|
+
uv run poe dev # Dev server (uvicorn reload, port 17000) — for examples/API testing only (dataenginex has no built-in HTTP server; see dex-studio for web UI)
|
|
45
45
|
uv run poe docker-up # Docker compose up
|
|
46
46
|
uv run poe docker-down # Docker compose down
|
|
47
47
|
|
|
@@ -63,7 +63,7 @@ pip install 'litellm>=1.83.3' --no-deps # LLM routing (separate: pins python-dot
|
|
|
63
63
|
|
|
64
64
|
______________________________________________________________________
|
|
65
65
|
|
|
66
|
-
##
|
|
66
|
+
## Validation
|
|
67
67
|
|
|
68
|
-
|
|
69
|
-
Tests passing ≠ app working.
|
|
68
|
+
After any code change run: `uv run poe check-all` (lint + typecheck + test).
|
|
69
|
+
Tests passing ≠ app working — run `dex validate dex.yaml` to verify config.
|
|
@@ -1,18 +1,18 @@
|
|
|
1
|
-
# Contributing to
|
|
1
|
+
# Contributing to DataEngineX
|
|
2
2
|
|
|
3
3
|
Thank you for your interest in contributing to DataEngineX!
|
|
4
4
|
|
|
5
|
-
**→ Full contributor guide: [docs/
|
|
5
|
+
**→ Full contributor guide: [docs/contributing.md](docs/contributing.md)**
|
|
6
6
|
|
|
7
7
|
Quick essentials:
|
|
8
8
|
|
|
9
|
-
- Development setup: [docs/
|
|
10
|
-
- Governance & legal: See [docs/
|
|
9
|
+
- Development setup: [docs/development.md](docs/development.md)
|
|
10
|
+
- Governance & legal: See [docs/contributing.md](docs/contributing.md)
|
|
11
11
|
- Code of Conduct: [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md)
|
|
12
12
|
|
|
13
13
|
## Quick Start
|
|
14
14
|
|
|
15
|
-
1. Fork and create a feature branch from `
|
|
15
|
+
1. Fork and create a feature branch from `main`
|
|
16
16
|
1. Run local checks: `uv run poe check-all`
|
|
17
17
|
1. Open a PR using `.github/PULL_REQUEST_TEMPLATE.md`
|
|
18
18
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: dataenginex
|
|
3
|
-
Version: 0.4.
|
|
3
|
+
Version: 0.4.4
|
|
4
4
|
Summary: DataEngineX — open-source, self-hosted, local-first Data + ML + AI workbench library
|
|
5
5
|
Author-email: Jay <jayapal.myaka99@gmail.com>
|
|
6
6
|
License: MIT License
|
|
@@ -30,7 +30,6 @@ Requires-Dist: click>=8.3.3
|
|
|
30
30
|
Requires-Dist: croniter>=6.2.2
|
|
31
31
|
Requires-Dist: duckdb>=1.5.2
|
|
32
32
|
Requires-Dist: httpx>=0.28.1
|
|
33
|
-
Requires-Dist: msgpack>=1.2.1
|
|
34
33
|
Requires-Dist: orjson>=3.11.0
|
|
35
34
|
Requires-Dist: prometheus-client>=0.25.0
|
|
36
35
|
Requires-Dist: pyarrow>=23.0.1
|
|
@@ -38,7 +37,6 @@ Requires-Dist: pydantic-settings>=2.14.2
|
|
|
38
37
|
Requires-Dist: pydantic>=2.13.4
|
|
39
38
|
Requires-Dist: pyyaml>=6.0.3
|
|
40
39
|
Requires-Dist: structlog>=25.5.0
|
|
41
|
-
Requires-Dist: zstandard>=0.25.0
|
|
42
40
|
Provides-Extra: cloud
|
|
43
41
|
Requires-Dist: boto3>=1.43.7; extra == 'cloud'
|
|
44
42
|
Requires-Dist: google-cloud-bigquery>=3.41.0; extra == 'cloud'
|
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
# dataenginex
|
|
2
2
|
|
|
3
|
-
[](https://github.com/TheDataEngineX/dataenginex/actions/workflows/ci.yml)
|
|
4
4
|
[](https://pypi.org/project/dataenginex/)
|
|
5
5
|
[](https://www.python.org/downloads/)
|
|
6
6
|
[](LICENSE)
|
|
7
7
|
|
|
8
|
-
The Python library that powers [
|
|
8
|
+
The Python library that powers [DataEngineX Studio](https://github.com/TheDataEngineX/dex-studio) — an open-source, self-hosted, local-first Data + ML + AI workbench for individuals and small teams. **Use the library directly when you want code; install DataEngineX Studio when you want a UI.**
|
|
9
9
|
|
|
10
10
|
______________________________________________________________________
|
|
11
11
|
|
|
@@ -116,7 +116,7 @@ git clone https://github.com/TheDataEngineX/dex-studio && cd dex-studio
|
|
|
116
116
|
docker compose up # http://localhost:7860
|
|
117
117
|
```
|
|
118
118
|
|
|
119
|
-
|
|
119
|
+
DataEngineX Studio imports `dataenginex` directly — no separate API server, no HTTP hop.
|
|
120
120
|
|
|
121
121
|
______________________________________________________________________
|
|
122
122
|
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
# ---------------------------------------------------------------
|
|
2
|
-
#
|
|
2
|
+
# DataEngineX — Storage Emulators for Integration Tests
|
|
3
3
|
#
|
|
4
4
|
# Spins up LocalStack (S3) and fake-gcs-server (GCS) for local
|
|
5
5
|
# integration testing without real cloud accounts.
|
|
@@ -8,18 +8,18 @@
|
|
|
8
8
|
|
|
9
9
|
## Context
|
|
10
10
|
|
|
11
|
-
|
|
11
|
+
DataEngineX needs a scalable, standardized data architecture that works across all projects (CareerDEX, Weather, etc.). The data pipeline needs to handle:
|
|
12
12
|
|
|
13
13
|
- Raw data ingestion from multiple sources
|
|
14
14
|
- Data transformation and quality validation
|
|
15
15
|
- Feature engineering for ML models
|
|
16
16
|
- Analytics-ready data consumption
|
|
17
17
|
|
|
18
|
-
Multiple projects will be built on the
|
|
18
|
+
Multiple projects will be built on the DataEngineX platform, each with different data sources and transformations. We need a consistent, reusable pattern.
|
|
19
19
|
|
|
20
20
|
## Decision
|
|
21
21
|
|
|
22
|
-
Implement the **Medallion Architecture** (Bronze → Silver → Gold layers) as the standard data architecture for all
|
|
22
|
+
Implement the **Medallion Architecture** (Bronze → Silver → Gold layers) as the standard data architecture for all DataEngineX projects.
|
|
23
23
|
|
|
24
24
|
### Architecture Layers
|
|
25
25
|
|
|
@@ -5,7 +5,7 @@
|
|
|
5
5
|
**DataEngineX** is a unified Data + ML + AI **library** that wires industry tools through a
|
|
6
6
|
single config-driven interface. One `dex.yaml` defines the entire project.
|
|
7
7
|
|
|
8
|
-
**Design principle:** Pure Python library — no HTTP server bundled. Your application (
|
|
8
|
+
**Design principle:** Pure Python library — no HTTP server bundled. Your application (DataEngineX Studio,
|
|
9
9
|
your own FastAPI/Flask app, a script) imports `dataenginex` and owns the server layer.
|
|
10
10
|
|
|
11
11
|
## Architecture
|
|
@@ -56,9 +56,9 @@ dex.yaml
|
|
|
56
56
|
Every subsystem follows the same pattern:
|
|
57
57
|
|
|
58
58
|
1. **ABC** in `core/interfaces.py` — defines the contract (e.g. `BaseConnector`)
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
59
|
+
1. **BackendRegistry[T]** in `core/registry.py` — discovers and registers implementations
|
|
60
|
+
1. **Built-in** implements the ABC with zero external deps
|
|
61
|
+
1. **Extras** implement the same ABC, swapped in via config
|
|
62
62
|
|
|
63
63
|
```python
|
|
64
64
|
from dataenginex.core.registry import BackendRegistry
|
|
@@ -108,10 +108,7 @@ DataEngineXError
|
|
|
108
108
|
├── ConfigError → ConfigValidationError
|
|
109
109
|
├── PipelineError → PipelineStepError
|
|
110
110
|
├── RegistryError
|
|
111
|
-
|
|
112
|
-
├── TrainingError
|
|
113
|
-
├── ServingError
|
|
114
|
-
└── AgentError → LLMProviderError
|
|
111
|
+
└── BackendNotInstalledError
|
|
115
112
|
```
|
|
116
113
|
|
|
117
114
|
## Module Map
|
|
@@ -0,0 +1,97 @@
|
|
|
1
|
+
# CI/CD Pipeline
|
|
2
|
+
|
|
3
|
+
**Continuous integration and release automation for dataenginex.**
|
|
4
|
+
|
|
5
|
+
______________________________________________________________________
|
|
6
|
+
|
|
7
|
+
## Overview
|
|
8
|
+
|
|
9
|
+
dataenginex is a pure Python library published to PyPI. The pipeline is:
|
|
10
|
+
|
|
11
|
+
- **CI**: Linting, type checking, testing, and security scanning on every PR to `main`.
|
|
12
|
+
- **Release**: Push a `v{X.Y.Z}` tag to `main` → `release.yml` builds, publishes to PyPI, and creates a GitHub Release.
|
|
13
|
+
|
|
14
|
+
```mermaid
|
|
15
|
+
graph LR
|
|
16
|
+
Dev[Developer] --> PR[Open PR to main]
|
|
17
|
+
PR --> CI[CI: Lint/Test/Security]
|
|
18
|
+
CI --> Review[Code Review]
|
|
19
|
+
Review --> Merge["Merge to main"]
|
|
20
|
+
Merge --> Tag["Push tag vX.Y.Z"]
|
|
21
|
+
Tag --> Release[release.yml]
|
|
22
|
+
Release --> Build["Build wheel + sdist"]
|
|
23
|
+
Build --> PyPI["Publish to PyPI (OIDC)"]
|
|
24
|
+
Build --> GHRelease["GitHub Release + CycloneDX SBOM"]
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
______________________________________________________________________
|
|
28
|
+
|
|
29
|
+
## Continuous Integration (CI)
|
|
30
|
+
|
|
31
|
+
**Workflow**: `.github/workflows/ci.yml`
|
|
32
|
+
|
|
33
|
+
**Triggers**:
|
|
34
|
+
|
|
35
|
+
- Push to `main`
|
|
36
|
+
- Pull requests targeting `main`
|
|
37
|
+
|
|
38
|
+
**Jobs**:
|
|
39
|
+
|
|
40
|
+
### 1. Quality (`quality`)
|
|
41
|
+
|
|
42
|
+
```bash
|
|
43
|
+
uv run ruff check src/ tests/
|
|
44
|
+
uv run mypy src/dataenginex/ --strict
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
### 2. Test (`test`)
|
|
48
|
+
|
|
49
|
+
```bash
|
|
50
|
+
uv run pytest tests/ -x --tb=short --cov=src/dataenginex/
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
Coverage threshold: 80%.
|
|
54
|
+
|
|
55
|
+
### 3. Package validation
|
|
56
|
+
|
|
57
|
+
```bash
|
|
58
|
+
uv build
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
### 4. Security Scans
|
|
62
|
+
|
|
63
|
+
Runs via the shared reusable workflow at `.github/workflows/security.yml`:
|
|
64
|
+
|
|
65
|
+
- **Trivy**: Misconfig and secret scan — results uploaded to GitHub Security tab.
|
|
66
|
+
- **CodeQL**: Handled by GitHub's default setup.
|
|
67
|
+
|
|
68
|
+
______________________________________________________________________
|
|
69
|
+
|
|
70
|
+
## Release Automation
|
|
71
|
+
|
|
72
|
+
**Workflow**: `.github/workflows/release.yml`
|
|
73
|
+
|
|
74
|
+
**Trigger**: Push a tag matching `v[0-9]+.[0-9]+.[0-9]+` to `main`.
|
|
75
|
+
|
|
76
|
+
**Jobs**:
|
|
77
|
+
|
|
78
|
+
1. **build** — `uv build` → upload wheel + sdist
|
|
79
|
+
1. **publish-pypi** — `pypa/gh-action-pypi-publish` (OIDC trusted publishing)
|
|
80
|
+
1. **github-release** — CycloneDX SBOM → `gh release create`
|
|
81
|
+
|
|
82
|
+
**How to release**:
|
|
83
|
+
|
|
84
|
+
```bash
|
|
85
|
+
git tag v1.2.3
|
|
86
|
+
git push origin v1.2.3
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
______________________________________________________________________
|
|
90
|
+
|
|
91
|
+
## Workflows Overview
|
|
92
|
+
|
|
93
|
+
| Workflow | Trigger | Purpose |
|
|
94
|
+
| --- | --- | --- |
|
|
95
|
+
| **CI** | Push/PR to main | Lint + typecheck + test + security |
|
|
96
|
+
| **Security** | Push/PR to main | Trivy misconfig + secret scan |
|
|
97
|
+
| **Release** | Push tag `v*.*.*` to main | Build → PyPI → GitHub Release |
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
# Contributing to
|
|
1
|
+
# Contributing to DataEngineX
|
|
2
2
|
|
|
3
3
|
Thank you for contributing!
|
|
4
4
|
|
|
@@ -6,7 +6,7 @@ Thank you for contributing!
|
|
|
6
6
|
|
|
7
7
|
1. Read [development.md](./development.md) for setup instructions
|
|
8
8
|
1. Fork the repository
|
|
9
|
-
1. Create a feature branch from `
|
|
9
|
+
1. Create a feature branch from `main`
|
|
10
10
|
1. Make your changes
|
|
11
11
|
1. Submit a pull request
|
|
12
12
|
|
|
@@ -68,7 +68,7 @@ Use semantic commit format:
|
|
|
68
68
|
|
|
69
69
|
- This project is open source under MIT; keep license and attribution notices in redistributions.
|
|
70
70
|
- Forks are welcome, but should use a distinct public name when redistributed as a separate project.
|
|
71
|
-
- Do not present a fork as the official DataEngineX
|
|
71
|
+
- Do not present a fork as the official DataEngineX project.
|
|
72
72
|
- See the project's license and brand guidelines for brand-usage details.
|
|
73
73
|
|
|
74
74
|
## Code Reviews
|
|
@@ -94,7 +94,7 @@ uv run poe check-all # Run all checks
|
|
|
94
94
|
- `enhancement` - New feature or improvement
|
|
95
95
|
- `good first issue` - Good for newcomers
|
|
96
96
|
- `P1-high` / `P2-medium` - Priority levels
|
|
97
|
-
- `dex-module` - Core
|
|
97
|
+
- `dex-module` - Core DataEngineX infrastructure
|
|
98
98
|
|
|
99
99
|
## Questions?
|
|
100
100
|
|
|
@@ -39,7 +39,7 @@ This installs all Python dependencies and configures pre-commit hooks.
|
|
|
39
39
|
# 1. Clone repo and create feature branch
|
|
40
40
|
git clone https://github.com/TheDataEngineX/dataenginex.git
|
|
41
41
|
cd dataenginex
|
|
42
|
-
git checkout -b
|
|
42
|
+
git checkout -b my-feature-branch main
|
|
43
43
|
|
|
44
44
|
# 2. Install Python deps & pre-commit hooks
|
|
45
45
|
uv run poe setup
|
|
@@ -68,8 +68,8 @@ dataenginex/
|
|
|
68
68
|
### Branch & Commit
|
|
69
69
|
|
|
70
70
|
```bash
|
|
71
|
-
# 1. Create feature branch from
|
|
72
|
-
git checkout -b
|
|
71
|
+
# 1. Create a feature branch from main
|
|
72
|
+
git checkout -b my-feature-branch main
|
|
73
73
|
|
|
74
74
|
# 2. Make changes to src/
|
|
75
75
|
# Add tests in tests/
|
|
@@ -83,7 +83,7 @@ uv run poe test
|
|
|
83
83
|
git commit -m "feat(#XXX): description"
|
|
84
84
|
|
|
85
85
|
# 5. Push & create PR
|
|
86
|
-
git push origin
|
|
86
|
+
git push origin my-feature-branch
|
|
87
87
|
```
|
|
88
88
|
|
|
89
89
|
**PR Requirements:**
|
|
@@ -91,11 +91,11 @@ git push origin feat/issue-XXX-description
|
|
|
91
91
|
- Link to issue: `Closes #XXX`
|
|
92
92
|
- All checks pass (CI/CD ~3-5 min)
|
|
93
93
|
- 1 approval required
|
|
94
|
-
- Merge to `
|
|
94
|
+
- Merge to `main` when ready
|
|
95
95
|
|
|
96
96
|
### Version Management
|
|
97
97
|
|
|
98
|
-
|
|
98
|
+
DataEngineX has a single version source:
|
|
99
99
|
|
|
100
100
|
- **dataenginex version**: root `pyproject.toml` — managed automatically by the `release.yml` workflow
|
|
101
101
|
|
|
@@ -140,9 +140,9 @@ events = engine.secops_audit.events # list of AuditEvent
|
|
|
140
140
|
|
|
141
141
|
______________________________________________________________________
|
|
142
142
|
|
|
143
|
-
##
|
|
143
|
+
## DataEngineX Studio
|
|
144
144
|
|
|
145
|
-
|
|
145
|
+
DataEngineX Studio is the optional web UI that loads a `dex.yaml` and provides a single
|
|
146
146
|
control plane for Data / ML / AI / System. It uses `dataenginex` directly as a library
|
|
147
147
|
— no separate server process needed.
|
|
148
148
|
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
|
|
7
7
|
______________________________________________________________________
|
|
8
8
|
|
|
9
|
-
This document defines the software development lifecycle (SDLC) for
|
|
9
|
+
This document defines the software development lifecycle (SDLC) for DataEngineX, including the required stages, artifacts, and quality gates.
|
|
10
10
|
|
|
11
11
|
## Goals
|
|
12
12
|
|
|
@@ -14,28 +14,6 @@ This document defines the software development lifecycle (SDLC) for DEX, includi
|
|
|
14
14
|
- Enforce quality gates (lint, test, type checks, security scan) on every PR.
|
|
15
15
|
- Maintain a complete audit trail for promotion across environments.
|
|
16
16
|
|
|
17
|
-
## Lifecycle Overview
|
|
18
|
-
|
|
19
|
-
```mermaid
|
|
20
|
-
graph LR
|
|
21
|
-
Plan[1. Plan] --> Design[2. Design]
|
|
22
|
-
Design --> Implement[3. Implement]
|
|
23
|
-
Implement --> Verify[4. Verify CI]
|
|
24
|
-
Verify --> Release[5. Release CD]
|
|
25
|
-
Release --> Operate[6. Operate]
|
|
26
|
-
Operate -->|Issues| Plan
|
|
27
|
-
Operate -->|Incidents| Rollback[Rollback]
|
|
28
|
-
Rollback --> Operate
|
|
29
|
-
|
|
30
|
-
style Plan fill:#e1f5ff
|
|
31
|
-
style Design fill:#e1f5ff
|
|
32
|
-
style Implement fill:#fff3cd
|
|
33
|
-
style Verify fill:#fff3cd
|
|
34
|
-
style Release fill:#d4edda
|
|
35
|
-
style Operate fill:#d4edda
|
|
36
|
-
style Rollback fill:#f8d7da
|
|
37
|
-
```
|
|
38
|
-
|
|
39
17
|
## Lifecycle Stages
|
|
40
18
|
|
|
41
19
|
### 1) Plan
|
|
@@ -89,7 +67,7 @@ ______________________________________________________________________
|
|
|
89
67
|
|
|
90
68
|
### 4) Verify (CI)
|
|
91
69
|
|
|
92
|
-
**Entry**: Pull request opened against `
|
|
70
|
+
**Entry**: Pull request opened against `main`.
|
|
93
71
|
|
|
94
72
|
**Required Checks**:
|
|
95
73
|
|
|
@@ -107,8 +85,7 @@ ______________________________________________________________________
|
|
|
107
85
|
|
|
108
86
|
**Entry**:
|
|
109
87
|
|
|
110
|
-
-
|
|
111
|
-
- Stage/Prod deployment: PR merged to `main`.
|
|
88
|
+
- PR merged to `main`.
|
|
112
89
|
|
|
113
90
|
**Actions**:
|
|
114
91
|
|
|
@@ -141,49 +118,29 @@ ______________________________________________________________________
|
|
|
141
118
|
```mermaid
|
|
142
119
|
flowchart TD
|
|
143
120
|
Start([New Feature/Fix]) --> Issue[Create GitHub Issue]
|
|
144
|
-
Issue --> Branch["Create branch:
|
|
121
|
+
Issue --> Branch["Create branch: my-feature"]
|
|
145
122
|
Branch --> Code[Implement + Tests]
|
|
146
123
|
Code --> Local["Local checks: lint, test, types"]
|
|
147
124
|
Local -->|Failed| Code
|
|
148
|
-
Local -->|Passed| PR["Open PR →
|
|
125
|
+
Local -->|Passed| PR["Open PR → main"]
|
|
149
126
|
PR --> CI{CI Checks}
|
|
150
127
|
CI -->|Failed| Fix[Fix Issues]
|
|
151
128
|
Fix --> Code
|
|
152
129
|
CI -->|Passed| Review{Code Review}
|
|
153
130
|
Review -->|Changes Requested| Code
|
|
154
|
-
Review -->|Approved|
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
DevTest -->|Failed| Hotfix[Hotfix or Rollback]
|
|
158
|
-
Hotfix --> Code
|
|
159
|
-
DevTest -->|Passed| ReleasePR["Create PR: dev → main"]
|
|
160
|
-
ReleasePR --> ReleaseCI{CI Checks}
|
|
161
|
-
ReleaseCI -->|Failed| Fix
|
|
162
|
-
ReleaseCI -->|Passed| ReleaseReview{Code Review}
|
|
163
|
-
ReleaseReview -->|Changes Requested| Code
|
|
164
|
-
ReleaseReview -->|Approved| MergeMain["Merge to main"]
|
|
165
|
-
MergeMain --> ProdDeploy[Auto-deploy to stage+prod]
|
|
166
|
-
ProdDeploy --> Monitor[Monitor Production]
|
|
131
|
+
Review -->|Approved| MergeMain["Merge to main"]
|
|
132
|
+
MergeMain --> Prod[Auto-release if tagged]
|
|
133
|
+
Prod --> Monitor[Monitor]
|
|
167
134
|
Monitor --> End([✓ Complete])
|
|
168
|
-
|
|
169
|
-
style Start fill:#e1f5ff
|
|
170
|
-
style Issue fill:#e1f5ff
|
|
171
|
-
style Code fill:#fff3cd
|
|
172
|
-
style CI fill:#fff3cd
|
|
173
|
-
style Review fill:#fff3cd
|
|
174
|
-
style DevDeploy fill:#d4edda
|
|
175
|
-
style ProdDeploy fill:#d4edda
|
|
176
|
-
style End fill:#d4edda
|
|
177
|
-
style Hotfix fill:#f8d7da
|
|
178
135
|
```
|
|
179
136
|
|
|
180
137
|
**Steps**:
|
|
181
138
|
|
|
182
|
-
1. Create
|
|
183
|
-
1.
|
|
184
|
-
1. Open a PR to `
|
|
185
|
-
1. After validation in dev, open a release PR from `dev` → `main`.
|
|
139
|
+
1. Create a feature branch from `main`.
|
|
140
|
+
1. Implement changes, run local checks (`uv run poe check-all`).
|
|
141
|
+
1. Open a PR to `main` and request review.
|
|
186
142
|
1. Merge after all required checks pass.
|
|
143
|
+
1. Push a `vX.Y.Z` tag to trigger a PyPI release.
|
|
187
144
|
|
|
188
145
|
See [Contributing Guide](contributing.md) for branch naming, local checks, and PR conventions.
|
|
189
146
|
|
|
@@ -11,7 +11,7 @@ DataEngineX uses a multi-layered security scanning pipeline across CI/CD workflo
|
|
|
11
11
|
| **CodeQL** | Static analysis (Python + GitHub Actions) | `security.yml` |
|
|
12
12
|
| **pip-audit** | Python dependency vulnerability audit | `poe security` |
|
|
13
13
|
| **CycloneDX** | SBOM generation (attached to releases) | `release-*.yml` |
|
|
14
|
-
| **
|
|
14
|
+
| **Renovate** | Automated dependency updates | `renovate.json` |
|
|
15
15
|
|
|
16
16
|
## How It Works
|
|
17
17
|
|
|
@@ -30,7 +30,7 @@ DataEngineX uses a multi-layered security scanning pipeline across CI/CD workflo
|
|
|
30
30
|
trivy image dex:local
|
|
31
31
|
```
|
|
32
32
|
|
|
33
|
-
### On Release (release
|
|
33
|
+
### On Release (release.yml)
|
|
34
34
|
|
|
35
35
|
1. **CycloneDX SBOM** — generates a Software Bill of Materials in CycloneDX JSON format.
|
|
36
36
|
1. **SBOM upload** — attaches the SBOM as a release asset on GitHub Releases.
|
|
@@ -5,7 +5,7 @@ Demonstrates:
|
|
|
5
5
|
- Fetching data from an external HTTP API (OpenWeatherMap-style)
|
|
6
6
|
- Transforming raw responses into standardized records (Bronze → Silver)
|
|
7
7
|
- Validating data quality before promotion (Silver → Gold)
|
|
8
|
-
- Using
|
|
8
|
+
- Using DataEngineX QualityGate for data validation
|
|
9
9
|
|
|
10
10
|
Run:
|
|
11
11
|
uv run python examples/07_api_ingestion.py
|