dataenginex 0.3.5__tar.gz → 0.4.0__tar.gz

dataenginex-0.4.0/.claude/commands/new-feature.md

@@ -0,0 +1,22 @@
+# New Feature (DEX)
+
+Scaffold a new feature for the `dataenginex` core framework. Ask for the feature name and description if not provided as $ARGUMENTS.
+
+Steps:
+
+1. **Plan** — Write the implementation plan to `tasks/todo.md`. Check in before implementing.
+1. **Explore** — Search `src/dataenginex/` for related patterns. Use Context7 MCP for library docs.
+1. **Design** — Identify the right module:
+   - API endpoint → `src/dataenginex/api/` — `response_model=` required, type hints, auth via lifespan middleware
+   - Middleware → `src/dataenginex/middleware/` — follows request logging → metrics → auth → rate limit order
+   - Data pipeline → Medallion pattern (bronze → silver → gold) via `SchemaRegistry` / `DataCatalog`
+   - ML feature → `src/dataenginex/ml/` — `ModelRegistry` lifecycle (development → staging → production → archived)
+   - Quality gate → `src/dataenginex/quality/`
+   - Plugin hook → `src/dataenginex/plugins/` — entry-point based discovery
+   - Framework util → `src/dataenginex/core/`
+1. **Implement** — Follow existing patterns. `from __future__ import annotations` at top. `structlog` only (no loguru, no print). Use `BackendRegistry` for new pluggable subsystems.
+1. **Test** — Unit tests in `tests/unit/`, integration tests in `tests/integration/` (live uvicorn). `asyncio_mode = "auto"` — no `@pytest.mark.asyncio` needed.
+1. **Validate** — Run `/validate` (dex version — includes real server step)
+1. **Update** — Mark complete in `tasks/todo.md`; update `TODO.md` if relevant
+
+Follow all standards in `CLAUDE.md` and `../CLAUDE.md`.

dataenginex-0.4.0/.claude/commands/validate.md

@@ -0,0 +1,44 @@
+# Validate (DEX)
+
+Run the full DEX validation pipeline in order. Stop and report on first failure.
+
+1. **Lint**
+
+   ```bash
+   uv run poe lint
+   ```
+
+1. **Typecheck**
+
+   ```bash
+   uv run poe typecheck
+   ```
+
+1. **Tests**
+
+   ```bash
+   uv run poe test
+   ```
+
+1. **Real server** — start the quickstart example and verify all endpoints
+
+   ```bash
+   uv run python examples/02_api_quickstart.py &
+   sleep 2
+   curl -sf http://localhost:17000/health   | python -m json.tool
+   curl -sf http://localhost:17000/         | python -m json.tool
+   curl -sf http://localhost:17000/metrics  | head -20
+   curl -sf -X POST http://localhost:17000/echo -H "Content-Type: application/json" -d '{"message":"test"}' | python -m json.tool
+   ```
+
+   Check response **bodies** — not just status codes. Kill the server after.
+
+1. **Standalone import**
+
+   ```bash
+   uv run python -c "import dataenginex; print('OK', dataenginex.__file__)"
+   ```
+
+Tests passing ≠ app working. Step 4 is NON-NEGOTIABLE.
+
+Report pass/fail for each step with exact output. On failure, identify root cause — do not skip ahead.

dataenginex-0.4.0/.claude/settings.json

@@ -0,0 +1,71 @@
+{
+  "autoUpdatesChannel": "latest",
+  "statusLine": {
+    "type": "command",
+    "command": "b=$(git branch --show-current 2>/dev/null); [ -n \"$b\" ] && echo \"[$b]\" || echo \"\"",
+    "padding": 2
+  },
+  "permissions": {
+    "defaultMode": "default",
+    "allow": [
+      "Read",
+      "Edit",
+      "Bash(uv run poe *)",
+      "Bash(uv run uvicorn *)",
+      "Bash(uv run python *)",
+      "Bash(uv run *)",
+      "Bash(uv lock *)",
+      "Bash(uv sync *)",
+      "Bash(curl *)",
+      "Bash(git status)",
+      "Bash(git status *)",
+      "Bash(git diff *)",
+      "Bash(git log *)",
+      "Bash(git branch *)",
+      "Bash(git add *)",
+      "Bash(git fetch *)",
+      "Bash(ls *)",
+      "Bash(pwd)",
+      "Bash(which *)",
+      "Bash(wc *)",
+      "Bash(echo *)",
+      "Bash(python -c *)",
+      "Bash(python3 *)",
+      "Bash(gh pr *)",
+      "Bash(gh issue *)",
+      "Bash(gh release *)",
+      "Bash(gh run *)",
+      "Bash(gh api *)",
+      "Bash(llmfit *)",
+      "Bash(npx -y @upstash/context7-mcp*)"
+    ],
+    "deny": [
+      "Bash(git commit *)",
+      "Bash(git push *)",
+      "Bash(rm -rf *)",
+      "Bash(git push --force*)",
+      "Bash(git reset --hard*)",
+      "Bash(pip install*)",
+      "Bash(sudo *)",
+      "Bash(chmod 777*)",
+      "Bash(curl * | bash*)"
+    ]
+  },
+  "hooks": {
+    "UserPromptSubmit": [
+      {
+        "matcher": "",
+        "hooks": [
+          {
+            "type": "prompt",
+            "prompt": "Classify this task as 'simple' (data lookup, quick questions, simple edits — use haiku for speed/cost), 'moderate' (implementation, debugging, refactoring — use sonnet for balance), or 'complex' (architecture, multi-file design, research — use opus for reasoning). Based on the user message below, respond with ONLY the word: haiku, sonnet, or opus. User message: $ARGUMENTS",
+            "statusMessage": "Analyzing task complexity...",
+            "model": "claude-haiku-4-5-20251001"
+          }
+        ]
+      }
+    ]
+  },
+  "alwaysThinkingEnabled": false,
+  "cleanupPeriodDays": 7
+}

dataenginex-0.4.0/.env.template

@@ -0,0 +1,73 @@
+# =============================================================================
+# DataEngineX — Environment Configuration Template
+# =============================================================================
+# Copy to .env and fill in values for your environment.
+# NEVER commit secrets to version control — use a secret manager in production.
+# =============================================================================
+
+# -- Application -------------------------------------------------------------
+APP_NAME=dataenginex
+ENVIRONMENT=dev
+
+# -- Logging -----------------------------------------------------------------
+LOG_LEVEL=INFO
+# "json" for structured JSON output, "console" for human-readable
+LOG_FORMAT=console
+
+# -- API Server --------------------------------------------------------------
+HOST=0.0.0.0
+PORT=17000
+
+# -- Authentication (JWT) ----------------------------------------------------
+# Set to "true" to enforce Bearer-token auth on protected endpoints
+DEX_AUTH_ENABLED=false
+# HMAC-SHA256 secret — generate with: python -c "import secrets; print(secrets.token_urlsafe(64))"
+DEX_JWT_SECRET=
+DEX_JWT_ALGORITHM=HS256
+# Token TTL in minutes
+DEX_JWT_TTL_MINUTES=60
+
+# -- Rate Limiting -----------------------------------------------------------
+# Set to "true" to enable per-IP token-bucket rate limiting
+DEX_RATE_LIMIT_ENABLED=false
+DEX_RATE_LIMIT_RPM=60
+DEX_RATE_LIMIT_BURST=10
+
+# -- Observability / Tracing ------------------------------------------------
+# OpenTelemetry collector endpoint (leave blank to disable OTLP export)
+OTLP_ENDPOINT=
+ENABLE_CONSOLE_TRACES=false
+
+# -- Database ----------------------------------------------------------------
+DB_DRIVER=
+DB_SERVER=
+DB_NAME=
+DB_USER=
+DB_PASSWORD=
+
+# -- Databricks --------------------------------------------------------------
+DATABRICKS_HOST=
+DATABRICKS_CLIENT_ID=
+DATABRICKS_SECRET=
+
+# -- External APIs -----------------------------------------------------------
+OPENWEATHER_API_KEY=
+
+# -- GitHub App (CI/CD webhooks) ---------------------------------------------
+GITHUB_PRIVATE_KEY=
+GITHUB_APP_IDENTIFIER=
+GITHUB_WEBHOOK_SECRET=
+GITHUB_PERSONAL_ACCESS_TOKEN=
+
+# -- MCP Servers -------------------------------------------------------------
+# GitHub MCP uses GITHUB_PERSONAL_ACCESS_TOKEN above.
+CLOUDFLARE_API_TOKEN=
+CLOUDFLARE_ACCOUNT_ID=
+POSTGRES_CONNECTION_STRING=
+
+# -- Notifications -----------------------------------------------------------
+SLACK_WEBHOOK_URL=
+
+# -- Grafana (docker-compose local dev) --------------------------------------
+GF_SECURITY_ADMIN_USER=admin
+GF_SECURITY_ADMIN_PASSWORD=admin

dataenginex-0.4.0/.github/ISSUE_TEMPLATE/bug_report.yml

@@ -0,0 +1,107 @@
+name: Bug Report
+description: Report a bug or unexpected behavior
+title: "[Bug]: "
+labels: ["type:bug", "needs-triage", "bug"]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Thanks for taking the time to report a bug! Please fill out the sections below.
+
+  - type: textarea
+    id: description
+    attributes:
+      label: Bug Description
+      description: A clear and concise description of what the bug is.
+      placeholder: Describe what went wrong...
+    validations:
+      required: true
+
+  - type: textarea
+    id: reproduce
+    attributes:
+      label: Steps to Reproduce
+      description: How can we reproduce this issue?
+      placeholder: |
+        1. Run command '...'
+        2. Check output '...'
+        3. See error
+    validations:
+      required: true
+
+  - type: textarea
+    id: expected
+    attributes:
+      label: Expected Behavior
+      description: What did you expect to happen?
+      placeholder: Describe the expected behavior...
+    validations:
+      required: true
+
+  - type: textarea
+    id: actual
+    attributes:
+      label: Actual Behavior
+      description: What actually happened?
+      placeholder: Describe what actually happened...
+    validations:
+      required: true
+
+  - type: textarea
+    id: logs
+    attributes:
+      label: Relevant Logs/Screenshots
+      description: Add any logs, error messages, or screenshots
+      placeholder: Paste logs or drag images here...
+    validations:
+      required: false
+
+  - type: dropdown
+    id: environment
+    attributes:
+      label: Environment
+      description: Which environment(s) are affected?
+      multiple: true
+      options:
+        - Local Development
+        - Dev (dex-dev)
+        - Prod (dex)
+    validations:
+      required: true
+
+  - type: dropdown
+    id: provider_context
+    attributes:
+      label: Cloud/Provider Context
+      description: Where does this bug occur?
+      options:
+        - Local only
+        - Cloud-agnostic path
+        - AWS-specific path
+        - GCP-specific path
+        - Azure-specific path
+        - Unsure
+    validations:
+      required: true
+
+  - type: dropdown
+    id: cost_impact
+    attributes:
+      label: Cost Impact
+      description: Does this bug increase runtime or infra cost?
+      options:
+        - No noticeable cost impact
+        - Minor cost impact
+        - Major cost impact
+        - Unknown
+    validations:
+      required: false
+
+  - type: textarea
+    id: additional
+    attributes:
+      label: Additional Context
+      description: Any other context about the problem
+      placeholder: Add any other context...
+    validations:
+      required: false

dataenginex-0.4.0/.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,20 @@
+blank_issues_enabled: false
+contact_links:
+  - name: Security Policy
+    url: https://github.com/TheDataEngineX/dataenginex/blob/main/SECURITY.md
+    about: Report vulnerabilities responsibly using the security policy
+  - name: Support Guide
+    url: https://github.com/TheDataEngineX/dataenginex/blob/main/SUPPORT.md
+    about: Find the right channel for help, bugs, and feature proposals
+  - name: Questions and Ideas (GitHub Discussions)
+    url: https://github.com/TheDataEngineX/dataenginex/discussions
+    about: Use Discussions for Q&A, design brainstorming, and early proposals
+  - name: Organization Projects
+    url: https://github.com/orgs/TheDataEngineX/projects
+    about: Track roadmap status, milestones, and planning boards
+  - name: Documentation
+    url: https://github.com/TheDataEngineX/dataenginex/blob/main/docs/docs-hub.md
+    about: Check the documentation for guides and runbooks
+  - name: SDLC Guide
+    url: https://github.com/TheDataEngineX/dataenginex/blob/main/docs/SDLC.md
+    about: Review the development lifecycle and workflow

dataenginex-0.4.0/.github/ISSUE_TEMPLATE/feature_request.yml

@@ -0,0 +1,110 @@
+name: Feature Request
+description: Suggest a new feature or enhancement
+title: "[Feature]: "
+labels: ["type:feature", "needs-triage", "enhancement"]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Thanks for suggesting a feature! Please describe your idea below.
+
+  - type: textarea
+    id: problem
+    attributes:
+      label: Problem Statement
+      description: Is your feature request related to a problem? Please describe.
+      placeholder: "I'm frustrated when..."
+    validations:
+      required: true
+
+  - type: textarea
+    id: solution
+    attributes:
+      label: Proposed Solution
+      description: Describe the solution you'd like
+      placeholder: Describe your proposed solution...
+    validations:
+      required: true
+
+  - type: textarea
+    id: alternatives
+    attributes:
+      label: Alternatives Considered
+      description: Describe any alternative solutions or features you've considered
+      placeholder: Other approaches you've thought about...
+    validations:
+      required: false
+
+  - type: dropdown
+    id: priority
+    attributes:
+      label: Priority
+      description: How important is this feature?
+      options:
+        - P0 - Critical (blocks release)
+        - P1 - High (needed soon)
+        - P2 - Medium (nice to have)
+        - P3 - Low (future consideration)
+    validations:
+      required: true
+
+  - type: dropdown
+    id: component
+    attributes:
+      label: Component
+      description: Which part of the system does this affect?
+      multiple: true
+      options:
+        - API (FastAPI)
+        - Data Pipelines
+        - ML Models
+        - Infrastructure (K8s/ArgoCD)
+        - CI/CD
+        - Documentation
+        - Other
+    validations:
+      required: true
+
+  - type: dropdown
+    id: cloud_neutrality
+    attributes:
+      label: Cloud Neutrality
+      description: How will this stay vendor-neutral?
+      options:
+        - No cloud dependency (local/open-source default)
+        - Optional provider adapter (AWS/GCP/Azure)
+        - Provider-specific (must justify)
+    validations:
+      required: true
+
+  - type: dropdown
+    id: cost_profile
+    attributes:
+      label: Cost Profile
+      description: Expected baseline operating cost for default usage
+      options:
+        - Free / local-first
+        - Free tier compatible
+        - Paid service required (must justify)
+    validations:
+      required: true
+
+  - type: textarea
+    id: acceptance
+    attributes:
+      label: Acceptance Criteria
+      description: What defines "done" for this feature?
+      placeholder: |
+        - [ ] Criterion 1
+        - [ ] Criterion 2
+    validations:
+      required: false
+
+  - type: textarea
+    id: additional
+    attributes:
+      label: Additional Context
+      description: Add any other context, mockups, or examples
+      placeholder: Links, diagrams, screenshots...
+    validations:
+      required: false

dataenginex-0.4.0/.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,75 @@
+<!-- Describe the change and motivation in a few sentences -->
+
+## Description
+
+**Related Issue**: Closes #XXX
+
+## Suggested PR Title
+
+<!-- Example: chore: org/domain foundation with pages + label/project automation -->
+
+## Suggested Squash Commit Message
+
+<!--
+chore: establish org/domain foundation (pages, labels, project automation)
+
+- migrate platform/docs references to thedataenginex.org
+- add GitHub Pages docs deployment workflow
+- add label sync and org project intake automation
+- align issue templates/support/security/workflow docs
+-->
+
+## Keep / Change / Remove
+
+- **Keep**:
+- **Change**:
+- **Remove**:
+
+## Type of Change
+
+- [ ] Bug fix (non-breaking change which fixes an issue)
+- [ ] New feature (non-breaking change which adds functionality)
+- [ ] Breaking change (fix or feature that would cause existing functionality to change)
+- [ ] Documentation update
+- [ ] Infrastructure/DevOps change
+
+## Changes Made
+
+- Change 1
+- Change 2
+- Change 3
+
+## Testing
+
+- [ ] Unit tests added/updated
+- [ ] Integration tests added/updated
+- [ ] Tests pass locally (`uv run poe test`)
+- [ ] Coverage maintained/improved (target: 80%+)
+
+## Architecture Impact
+
+- [ ] Cloud-neutral by default (no hard dependency on a single provider)
+- [ ] Cost-aware default path (works locally or free/open-source tier)
+- [ ] Extension points documented for future adapters/plugins
+
+## Deprecation / Migration
+
+- [ ] No deprecation impact
+- [ ] If behavior changed, deprecation/migration notes were added to docs/release notes
+
+## Checklist
+
+- [ ] Linting passes (`uv run poe lint` ✓)
+- [ ] Type checking passes (`uv run poe typecheck` ✓)
+- [ ] Documentation updated (if applicable)
+- [ ] Pre-commit hooks pass
+
+## External Setup (if applicable)
+
+- [ ] GitHub Pages source set to **GitHub Actions**
+- [ ] Org/Repo variable set: `ORG_PROJECT_URL`
+- [ ] Org/Repo secret set: `ORG_PROJECT_TOKEN`
+- [ ] Cloudflare DNS updated for docs/api/apex domains
+- [ ] Post-cutover checks completed (see `docs/DEPLOY_RUNBOOK.md` → Org + Domain Rollout)
+
+## Notes for Reviewers

dataenginex-0.4.0/.github/dependabot.yml

@@ -0,0 +1,12 @@
+version: 2
+updates:
+  - package-ecosystem: pip
+    directory: "/"
+    schedule:
+      interval: weekly
+    target-branch: dev
+  - package-ecosystem: github-actions
+    directory: "/"
+    schedule:
+      interval: weekly
+    target-branch: dev

dataenginex-0.4.0/.github/labels.yml

@@ -0,0 +1,107 @@
+- name: type:bug
+  color: "D73A4A"
+  description: Defect or unintended behavior
+
+- name: type:feature
+  color: "0E8A16"
+  description: New capability or enhancement request
+
+- name: type:docs
+  color: "1D76DB"
+  description: Documentation improvements
+
+- name: type:chore
+  color: "BFD4F2"
+  description: Maintenance and housekeeping work
+
+- name: type:refactor
+  color: "C2E0C6"
+  description: Code restructuring without behavior changes
+
+- name: area:api
+  color: "5319E7"
+  description: API layer and contracts
+
+- name: area:data
+  color: "FBCA04"
+  description: Data pipelines, storage, and quality
+
+- name: area:ml
+  color: "C5DEF5"
+  description: ML modeling, training, and inference
+
+- name: area:infra
+  color: "0052CC"
+  description: Infrastructure, Kubernetes, and platform
+
+- name: area:docs
+  color: "0E8A16"
+  description: Documentation structure and content
+
+- name: area:ci
+  color: "0366D6"
+  description: CI/CD workflows and automation
+
+- name: priority:p0
+  color: "B60205"
+  description: Critical priority
+
+- name: priority:p1
+  color: "D93F0B"
+  description: High priority
+
+- name: priority:p2
+  color: "FBCA04"
+  description: Medium priority
+
+- name: priority:p3
+  color: "0E8A16"
+  description: Low priority
+
+- name: needs-triage
+  color: "EDEDED"
+  description: Awaiting maintainer triage
+
+- name: needs-info
+  color: "F9D0C4"
+  description: More information required
+
+- name: blocked
+  color: "000000"
+  description: Blocked by external dependency
+
+- name: duplicate
+  color: "CFD3D7"
+  description: Duplicate of an existing issue
+
+- name: wontfix
+  color: "FFFFFF"
+  description: Intentionally not planned for implementation
+
+- name: good first issue
+  color: "7057FF"
+  description: Good starter task for first-time contributors
+
+- name: help wanted
+  color: "008672"
+  description: Community help welcomed
+
+- name: incident
+  color: "B60205"
+  description: Incident issue generated from deployment failures
+
+- name: deployment
+  color: "1D76DB"
+  description: Deployment and environment operations
+
+- name: cd
+  color: "5319E7"
+  description: Continuous deployment automation
+
+- name: bug
+  color: "D73A4A"
+  description: Backward-compatible alias for existing bug reports
+
+- name: enhancement
+  color: "A2EEEF"
+  description: Backward-compatible alias for feature requests

dataenginex-0.4.0/.github/release-pr-template.md

@@ -0,0 +1,23 @@
+# Release PR Template
+
+Use this template whenever opening a release PR (`dev → main`). Reference `release-notes.md` and verify QA steps before merging.
+
+## Summary
+
+- Describe what changed (e.g., alerts, APIs, docs).
+- Link to the release note: [release-notes.md](release-notes.md).
+- Note any rollout considerations (e.g., Alertmanager rules, Docker image tag).
+
+## Verification
+
+- `uv run poe lint`
+- `./.venv/bin/pytest -v`
+- `docker compose build`
+- Reload Prometheus/Alertmanager (see `observability.md` → "Reloading Alert Rules").
+
+## QA Checklist
+
+- [ ] Release note referenced above
+- [ ] Alert rules reloaded in target environment
+- [ ] Docker image tagged and pushed (`ghcr.io/thedataenginex/dex:sha-<8-char-sha>`)
+- [ ] Observability and QA docs updated as needed

dataenginex-0.4.0/.github/workflows/auto-pr.yml

@@ -0,0 +1,23 @@
+name: Auto PR
+
+on:
+  push:
+    branches:
+      - dev
+      - 'feature/**'
+      - 'fix/**'
+
+permissions:
+  contents: write
+  pull-requests: write
+
+jobs:
+  feature-to-dev:
+    if: startsWith(github.ref, 'refs/heads/feature/') || startsWith(github.ref, 'refs/heads/fix/')
+    uses: TheDataEngineX/.github/.github/workflows/auto-pr-to-dev.yml@main
+    secrets: inherit
+
+  dev-to-main:
+    if: github.ref == 'refs/heads/dev'
+    uses: TheDataEngineX/.github/.github/workflows/auto-pr-dev-to-main.yml@main
+    secrets: inherit