dataenginex 0.9.9__tar.gz → 1.0.0__tar.gz

{dataenginex-0.9.9/docs → dataenginex-1.0.0/.github}/release-pr-template.md

@@ -1,11 +1,11 @@
 # Release PR Template
 
-Use this template whenever opening a release PR (`dev → main`). Reference `RELEASE_NOTES.md` and verify QA steps before merging.
+Use this template whenever opening a release PR (`dev → main`). Reference `release-notes.md` and verify QA steps before merging.
 
 ## Summary
 
 - Describe what changed (e.g., alerts, APIs, docs).
-- Link to the release note: [RELEASE_NOTES.md](RELEASE_NOTES.md).
+- Link to the release note: [release-notes.md](release-notes.md).
 - Note any rollout considerations (e.g., Alertmanager rules, Docker image tag).
 
 ## Verification
@@ -13,7 +13,7 @@ Use this template whenever opening a release PR (`dev → main`). Reference `REL
 - `uv run poe lint`
 - `./.venv/bin/pytest -v`
 - `docker compose build`
-- Reload Prometheus/Alertmanager (see `OBSERVABILITY.md` → "Reloading Alert Rules").
+- Reload Prometheus/Alertmanager (see `observability.md` → "Reloading Alert Rules").
 
 ## QA Checklist
 

{dataenginex-0.9.9 → dataenginex-1.0.0}/.github/workflows/ci.yml

@@ -5,6 +5,11 @@ on:
     branches: [main, dev]
   pull_request:
     branches: [main, dev]
+  workflow_dispatch:
+
+  schedule:
+    # Weekly Python version compatibility check
+    - cron: '0 0 * * 0'
 
 permissions:
   contents: read
@@ -25,7 +30,7 @@ jobs:
       - run: uv run poe quality
 
   test:
-    name: Tests
+    name: Tests (Python 3.13)
     runs-on: ubuntu-latest
     needs: quality
     steps:
@@ -38,9 +43,27 @@ jobs:
         env:
           UV_PROJECT_ENVIRONMENT: .venv
       - run: uv run poe test-cov-core
-      - uses: codecov/codecov-action@v5
+      - uses: codecov/codecov-action@v6
         with:
           flags: dataenginex
           fail_ci_if_error: false
         env:
           CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
+
+  test-compat:
+    name: Python Compatibility
+    runs-on: ubuntu-latest
+    if: github.event_name == 'schedule'
+    strategy:
+      matrix:
+        python-version: ["3.11", "3.12"]
+    steps:
+      - uses: actions/checkout@v6
+      - uses: astral-sh/setup-uv@v7
+        with:
+          version: "latest"
+          python-version: ${{ matrix.python-version }}
+      - run: uv sync --group ml
+        env:
+          UV_PROJECT_ENVIRONMENT: .venv
+      - run: uv run poe check-all

dataenginex-1.0.0/.github/workflows/docs-notify.yml

@@ -0,0 +1,18 @@
+name: Notify Docs
+
+on:
+  push:
+    branches: [main]
+    paths: ['docs/**', 'src/**/**.py']
+
+jobs:
+  notify:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: peter-evans/repository-dispatch@v4
+        with:
+          token: ${{ secrets.DOCS_DISPATCH_TOKEN }}
+          repository: TheDataEngineX/docs
+          event-type: docs-update
+          client-payload: >-
+            {"repo": "${{ github.repository }}", "ref": "${{ github.sha }}"}

{dataenginex-0.9.9 → dataenginex-1.0.0}/.github/workflows/release-please.yml

@@ -4,6 +4,7 @@ on:
   push:
     branches:
       - main
+  workflow_dispatch:
 
 permissions:
   contents: write

{dataenginex-0.9.9 → dataenginex-1.0.0}/.github/workflows/security.yml

@@ -5,6 +5,7 @@ on:
     branches: [main, dev]
   pull_request:
     branches: [main, dev]
+  workflow_dispatch:
 
 permissions:
   contents: read

{dataenginex-0.9.9 → dataenginex-1.0.0}/.gitignore

@@ -87,3 +87,6 @@ Notes.txt
 # Claude Code — local overrides (personal, never commit)
 .claude/settings.local.json
 .superpowers/
+
+# AI Coding Assistant
+.continue/

dataenginex-1.0.0/.release-please-manifest.json

@@ -0,0 +1,3 @@
+{
+  ".": "0.10.0"
+}

{dataenginex-0.9.9 → dataenginex-1.0.0}/CHANGELOG.md

@@ -5,8 +5,68 @@ All notable changes to `dataenginex` will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.10.0](https://github.com/TheDataEngineX/dex/compare/v0.9.9...v0.10.0) (2026-04-01)
+
+
+### Features
+
+* docs cleanup ([f4eff14](https://github.com/TheDataEngineX/dex/commit/f4eff14a1aeed657b715b182ac495c920f7b701f))
+* docs notify ([#210](https://github.com/TheDataEngineX/dex/issues/210)) ([c36616d](https://github.com/TheDataEngineX/dex/commit/c36616d9d0ae7a97274bd40ea41f39bc26f68fe0))
+* quality schema spark audit ([#212](https://github.com/TheDataEngineX/dex/issues/212)) ([2965801](https://github.com/TheDataEngineX/dex/commit/2965801257f6949a48f1c59be2f88e92294b4691))
+
+
+### Bug Fixes
+
+* add last-release-sha to anchor release-please at v0.9.9 ([7e78ca2](https://github.com/TheDataEngineX/dex/commit/7e78ca203347ff4c4000ca6ec42c335650da1b86))
+* restructure release-please-config to packages format with last-release-sha ([46f0b69](https://github.com/TheDataEngineX/dex/commit/46f0b69b8372d54d26ba60eedd9095d7fb0f6cd3))
+* sync dev to main ([40d5dff](https://github.com/TheDataEngineX/dex/commit/40d5dff99ae4295599badc848cd530df6eff9ffb))
+
+
+### Documentation
+
+* docs cleanup ([#211](https://github.com/TheDataEngineX/dex/issues/211)) ([0a6b7f7](https://github.com/TheDataEngineX/dex/commit/0a6b7f70287eb7399eb17acea2fb0ad09efd0597))
+
 ## [Unreleased]
 
+## [1.0.0] - 2026-04-07
+
+### Highlights
+
+- **Complete Data + ML + AI Framework**: All phases from the v1.0 master plan implemented — config-driven pipeline via `dex.yaml`, BackendRegistry pattern for swappable backends, unified CLI.
+- **Data Layer**: DuckDB connector (default), CSV connector, PipelineRunner with DAG resolution, transforms (filter, derive, cast, deduplicate), quality gates (completeness, uniqueness), column-level lineage tracking, built-in cron scheduler.
+- **ML Layer**: SQLite-backed experiment tracker, DuckDB feature store, sklearn/xgboost training integration, model registry with versioning (dev → staging → production), built-in model serving via FastAPI, PSI drift detection.
+- **AI Layer**: Built-in ReAct agent runtime, Ollama LLM provider (default), tool registry (sql_query, predict, search), BM25 sparse retrieval (DuckDB FTS), dense vector retrieval (DuckDB VSS HNSW), hybrid retrieval with RRF fusion, agent memory (short-term + episodic).
+- **CLI Commands**: `dex init`, `dex validate`, `dex version`, `dex serve`, `dex run`, `dex train`, `dex agent`, `dex query`.
+- **API**: FastAPI app factory, JWT auth, rate limiting, health endpoints, project CRUD, pipeline run/status, data explorer, ML experiments/models, agent chat/manage, WebSocket for live logs and streaming.
+- **Backend Registry Pattern**: Every subsystem follows ABC + BackendRegistry[T] pattern with built-in implementations and optional extras (Dagster, MLflow, Qdrant, LanceDB, sentence-transformers, PySpark).
+
+### Breaking Changes
+
+- **FastAPI now optional**: Core install (`pip install dataenginex`) includes only lightweight deps. Install `[api]` extra for FastAPI/uvicorn: `pip install dataenginex[api]`
+- **Cloud SDKs now optional**: Core install no longer requires boto3/google-cloud-storage/google-cloud-bigquery. Install `[cloud]` extra: `pip install dataenginex[cloud]`
+- **Routers moved**: API routers moved to application packages. Use `from dataenginex.api import ...` directly (requires `[api]` extra)
+- **Root `__init__.py` slimmed**: Re-exports removed. Import from submodules directly: `from dataenginex.api import HealthChecker` etc.
+
+### Added
+
+- **Full project templates**: `dex init --template [minimal|data-pipeline|ml-project|ai-agent|full-stack|career-intelligence]`
+- **Docker support**: Multi-stage Dockerfile (`ghcr.io/thedataenginex/dex`), docker-compose.yml for full stack
+- **SecOps**: PII scanning in pipelines, masking, audit trail
+- **Quality schema**: Spark audit integration for data quality validation
+- **Examples**: 5 runnable examples in `examples/` directory
+
+### Verification checklist
+
+1. `uv run poe lint` — Ruff checks clean
+2. `uv run poe typecheck` — mypy strict (all modules)
+3. `uv run poe test` — 663 passed, 36 skipped
+4. `pip install dataenginex` — installs successfully
+5. `dex validate dex.yaml` — validates config
+6. `dex version` — shows version
+
+[Unreleased]: https://github.com/TheDataEngineX/DEX/compare/v1.0.0...HEAD
+[1.0.0]: https://github.com/TheDataEngineX/DEX/releases/tag/v1.0.0
+
 ## [0.7.1] - 2026-03-17
 
 ### Fixed

dataenginex-1.0.0/CLAUDE.md

@@ -0,0 +1,59 @@
+# CLAUDE.md — DEX (dataenginex)
+
+Brief answers only. No explanations unless asked.
+Goal is to save Claude code tokens for lower cost without loosing quality.
+
+> Repo-specific context. Workspace-level rules, coding standards, and git conventions are in `../CLAUDE.md`.
+
+## Project Overview
+
+**DEX** — unified Data + ML + AI framework. Config-driven, self-hosted, production-ready.
+
+| Package | Location | Purpose |
+|---------|----------|---------|
+| `dataenginex` | `src/dataenginex/` | Core framework — config, registry, CLI, API, ML, AI (routing, runtime, memory, observability, workflows) |
+
+**Stack:** Python 3.13+ · FastAPI · DuckDB · structlog · Pydantic · Click · Rich · uv · Ruff · mypy strict · pytest
+
+**Version:** `uv run poe version`
+
+**Domain:** thedataenginex.org | **Org:** github.com/TheDataEngineX
+
+______________________________________________________________________
+
+## Build & Run Commands
+
+```bash
+# Quality
+uv run poe lint           # Ruff lint
+uv run poe lint-fix       # Ruff lint + auto-fix
+uv run poe typecheck      # mypy --strict (src/dataenginex/ only)
+uv run poe check-all      # lint + typecheck + test
+
+# Test
+uv run poe test           # All tests
+uv run poe test-unit      # Unit tests only
+uv run poe test-integration  # Integration tests only
+uv run poe test-cov       # Tests with coverage
+
+# CLI
+dex validate dex.yaml     # Validate config file
+dex version               # Show version + environment
+
+# Run
+uv run poe dev            # Dev server (uvicorn reload, port 17000)
+uv run poe docker-up      # Docker compose up
+uv run poe docker-down    # Docker compose down
+
+# Deps
+uv run poe uv-sync        # Sync deps from lockfile
+uv run poe uv-lock        # Regenerate lockfile
+uv run poe security       # Audit deps for vulnerabilities
+```
+
+______________________________________________________________________
+
+## Mandatory Validation
+
+Run `/validate` after ANY code change. Steps 4-5 (config validation + server smoke test) are NON-NEGOTIABLE.
+Tests passing ≠ app working.

dataenginex-1.0.0/CODE_OF_CONDUCT.md

@@ -0,0 +1,57 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our
+community a harassment-free experience for everyone, regardless of age, body
+size, visible or invisible disability, ethnicity, sex characteristics, gender
+identity and expression, level of experience, education, socio-economic status,
+nationality, personal appearance, race, religion, or sexual identity and
+orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming,
+diverse, inclusive, and healthy community.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment:
+
+- Using welcoming and inclusive language
+- Being respectful of differing viewpoints and experiences
+- Gracefully accepting constructive criticism
+- Focusing on what is best for the community
+- Showing empathy towards other community members
+
+Examples of unacceptable behavior:
+
+- The use of sexualized language or imagery, and sexual attention or advances
+- Trolling, insulting or derogatory comments, and personal or political attacks
+- Public or private harassment
+- Publishing others' private information without explicit permission
+- Other conduct which could reasonably be considered inappropriate
+
+## Enforcement Responsibilities
+
+Community leaders are responsible for clarifying and enforcing our standards of
+acceptable behavior and will take appropriate and fair corrective action in
+response to any behavior that they deem inappropriate, threatening, offensive,
+or harmful.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies when
+an individual is officially representing the community in public spaces.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported to the community leaders responsible for enforcement at
+**conduct@thedataenginex.dev**.
+
+All complaints will be reviewed and investigated promptly and fairly.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant](https://www.contributor-covenant.org),
+version 2.1, available at
+<https://www.contributor-covenant.org/version/2/1/code_of_conduct.html>.

{dataenginex-0.9.9 → dataenginex-1.0.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dataenginex
-Version: 0.9.9
+Version: 1.0.0
 Summary: DataEngineX - Core framework for AI/ML/Data engineering projects
 Author-email: Jay <jayapal.myaka99@gmail.com>
 License: MIT License
@@ -40,7 +40,6 @@ Requires-Dist: prometheus-client>=0.24.1
 Requires-Dist: pyarrow>=23.0.1
 Requires-Dist: pydantic>=2.10.0
 Requires-Dist: python-dotenv>=1.2.1
-Requires-Dist: python-json-logger>=4.0.0
 Requires-Dist: pyyaml>=6.0.3
 Requires-Dist: rich>=14.3.3
 Requires-Dist: structlog>=25.5.0

{dataenginex-0.9.9 → dataenginex-1.0.0}/README.md

@@ -112,7 +112,7 @@ ______________________________________________________________________
 
 ## Development
 
-See [docs/DEVELOPMENT.md](docs/DEVELOPMENT.md) for full setup.
+See [docs/development.md](docs/development.md) for full setup.
 
 ```bash
 uv run poe check-all         # lint + typecheck + tests

dataenginex-1.0.0/SECURITY.md

@@ -0,0 +1,71 @@
+# Security Policy
+
+## Supported Versions
+
+| Version | Supported |
+|---------|-----------|
+| Latest minor release (1.0.x) | ✅ |
+| Previous minor release | ✅ (security fixes only) |
+| Older versions | ❌ |
+
+## Reporting a Vulnerability
+
+**Do NOT open a public issue for security vulnerabilities.**
+
+Instead, please report them via one of these channels:
+
+1. **Email**: security@thedataenginex.dev
+2. **GitHub Security Advisories**: Use the "Report a vulnerability" button on the Security tab
+
+### What to Include
+
+- Description of the vulnerability
+- Steps to reproduce
+- Potential impact assessment
+- Suggested fix (if any)
+
+### Response Timeline
+
+| Stage | Timeline |
+|-------|----------|
+| Acknowledgment | Within 48 hours |
+| Initial assessment | Within 5 business days |
+| Fix development | Within 30 days (critical), 90 days (non-critical) |
+| Public disclosure | After fix is released |
+
+## Disclosure Policy
+
+We follow [coordinated disclosure](https://en.wikipedia.org/wiki/Coordinated_vulnerability_disclosure).
+We will credit reporters in the security advisory unless they prefer to remain anonymous.
+
+## Security Practices
+
+DataEngineX follows these security practices:
+
+- **No hardcoded secrets** — all credentials via environment variables
+- **Parameterized queries** — never SQL concatenation
+- **Input validation** — Pydantic models at API boundaries
+- **Dependency auditing** — automated via `uv run poe security`
+- **Pickle safety** — SafeUnpickler with HMAC verification for model loading
+- **Container security** — non-root users, minimal base images
+- **HTTPS only** — all production traffic encrypted
+- **Least privilege** — minimal permissions for service accounts
+
+## Security-Related Dependencies
+
+| Dependency | Purpose | Security Note |
+|------------|---------|---------------|
+| pydantic | Config validation | Validates all inputs |
+| python-dotenv | Env var loading | Never commit .env files |
+| httpx | HTTP client | Timeout configured |
+| structlog | Logging | No PII in logs by default |
+
+## Auditing
+
+Run security audits locally:
+
+```bash
+uv run poe security  # pip-audit for vulnerabilities
+```
+
+CI runs `pip-audit` and dependency scanning on every PR.

dataenginex-0.9.9/docs/CI_CD.md → dataenginex-1.0.0/docs/ci-cd.md

@@ -355,8 +355,8 @@ ______________________________________________________________________
 **Next Steps:**
 
 - **Deployment Runbook** (in `infradex` repo) - Release procedures
-- **[Observability](OBSERVABILITY.md)** - Monitor applications built on DEX
-- **[Contributing Guide](CONTRIBUTING.md)** - Development workflow
+- **[Observability](observability.md)** - Monitor applications built on DEX
+- **[Contributing Guide](contributing.md)** - Development workflow
 
 ______________________________________________________________________
 
@@ -407,4 +407,4 @@ gh workflow run pypi-publish.yml -f tag=v<version>
 
 ______________________________________________________________________
 
-**[← Back to Documentation Hub](docs-hub.md)**
+**[← Back to Documentation](index.md)**

dataenginex-0.9.9/docs/CONTRIBUTING.md → dataenginex-1.0.0/docs/contributing.md

@@ -1,8 +1,10 @@
-Thank you for contributing to DEX!
+# Contributing to DEX
+
+Thank you for contributing!
 
 ## Getting Started
 
-1. Read [DEVELOPMENT.md](./DEVELOPMENT.md) for setup instructions
+1. Read [development.md](./development.md) for setup instructions
 1. Fork the repository
 1. Create a feature branch from `dev`
 1. Make your changes
@@ -12,10 +14,10 @@ Thank you for contributing to DEX!
 
 Use semantic commit format:
 
-- `feat(#123): add resume parser`
-- `fix(#124): handle edge case in matching`
-- `docs: update API docs`
-- `test: add parsing tests`
+- `feat(#123): add DuckDB backend`
+- `fix(#124): handle missing config key in registry`
+- `docs: update API reference`
+- `test: add backend integration tests`
 - `chore: update dependencies`
 
 ## Code Style
@@ -96,8 +98,8 @@ uv run poe check-all   # Run all checks
 
 ## Questions?
 
-- Check [DEVELOPMENT.md](./DEVELOPMENT.md)
-- Review [Architecture docs](./ARCHITECTURE.md)
+- Check [development.md](./development.md)
+- Review [Architecture docs](./architecture.md)
 - Create a GitHub issue
 - Join #dex-dev Slack channel
 

dataenginex-0.9.9/docs/DEVELOPMENT.md → dataenginex-1.0.0/docs/development.md

@@ -205,8 +205,8 @@ uv run poe clean              # Remove caches and build artifacts
 
 ## Resources & Support
 
-- **Code Style**: See [CONTRIBUTING.md](./CONTRIBUTING.md)
-- **Architecture**: See [ARCHITECTURE.md](./ARCHITECTURE.md)
+- **Code Style**: See [contributing.md](./contributing.md)
+- **Architecture**: See [architecture.md](./architecture.md)
 - **ADRs**: See [ADR-0001](./adr/0001-medallion-architecture.md) for architectural decisions
 - **Deployment**: See Deployment Runbook in the `infradex` repo
 - **Issues**: [GitHub Issues](https://github.com/TheDataEngineX/dataenginex/issues)

dataenginex-1.0.0/docs/index.md

@@ -0,0 +1,10 @@
+# DataEngineX Documentation
+
+Welcome to the DEX docs site.
+
+Start here:
+
+- [Development Setup](development.md)
+- [Contributing](contributing.md)
+- [Architecture](architecture.md)
+- [Quickstart](quickstart.md)

dataenginex-0.9.9/docs/OBSERVABILITY.md → dataenginex-1.0.0/docs/observability.md

@@ -784,15 +784,15 @@ ______________________________________________________________________
 
 **Deployment & Operations:**
 
-- **[CI/CD Pipeline](CI_CD.md)** - Automated deployments
+- **[CI/CD Pipeline](ci-cd.md)** - Automated deployments
 - **Deployment Runbook** (in `infradex` repo) - Deploy procedures
 - **Local K8s Setup** (in `infradex` repo) - Kubernetes setup
 
 **Development:**
 
-- **[SDLC](SDLC.md)** - Development workflow
-- **[Contributing](CONTRIBUTING.md)** - Contribution guide
+- **[SDLC](sdlc.md)** - Development workflow
+- **[Contributing](contributing.md)** - Contribution guide
 
 ______________________________________________________________________
 
-**[← Back to Documentation Hub](docs-hub.md)**
+**[← Back to Documentation](index.md)**

{dataenginex-0.9.9 → dataenginex-1.0.0}/docs/quickstart.md

@@ -56,8 +56,8 @@ uv run poe check-all  # All of the above in one command
 
 ## Next Steps
 
-- [Development Guide](DEVELOPMENT.md) — full setup, editor config, and workflow
-- [Architecture](ARCHITECTURE.md) — medallion layers, API design, ML lifecycle
+- [Development Guide](development.md) — full setup, editor config, and workflow
+- [Architecture](architecture.md) — medallion layers, API design, ML lifecycle
 - [API Reference](api-reference/index.md) — auto-generated module docs
 - `examples/` directory — full list of runnable examples (01–10)
 

dataenginex-0.9.9/docs/SDLC.md → dataenginex-1.0.0/docs/sdlc.md

@@ -95,7 +95,7 @@ ______________________________________________________________________
 
 - CI, security, and quality checks as defined in the CI/CD pipeline.
 
-See [CI/CD Pipeline](CI_CD.md) for the authoritative list of checks.
+See [CI/CD Pipeline](ci-cd.md) for the authoritative list of checks.
 
 **Exit**: All checks pass and at least one reviewer approves.
 
@@ -185,7 +185,7 @@ flowchart TD
 1. After validation in dev, open a release PR from `dev` → `main`.
 1. Merge after all required checks pass.
 
-See [Contributing Guide](CONTRIBUTING.md) for branch naming, local checks, and PR conventions.
+See [Contributing Guide](contributing.md) for branch naming, local checks, and PR conventions.
 
 ______________________________________________________________________
 
@@ -193,8 +193,8 @@ ______________________________________________________________________
 
 **Development:**
 
-- **[Contributing Guide](CONTRIBUTING.md)** - Contribution workflow
-- **[CI/CD Pipeline](CI_CD.md)** - Automated build and deploy
+- **[Contributing Guide](contributing.md)** - Contribution workflow
+- **[CI/CD Pipeline](ci-cd.md)** - Automated build and deploy
 
 **Operations:**
 
@@ -203,4 +203,4 @@ ______________________________________________________________________
 
 ______________________________________________________________________
 
-**[← Back to Documentation Hub](docs-hub.md)**
+**[← Back to Documentation](index.md)**