dataenginex 0.3.4__tar.gz → 0.4.11__tar.gz

dataenginex-0.4.11/.gitignore

@@ -0,0 +1,81 @@
+# Byte-compiled / cache
+__pycache__/
+*.py[cod]
+*$py.class
+
+# Distribution / packaging
+build/
+dist/
+*.egg-info/
+.eggs/
+
+# Virtual environments
+.env
+.venv/
+venv/
+ENV/
+env/
+
+# TODO
+TODO.md
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage
+.coverage
+coverage.xml
+htmlcov/
+.tox/
+.nox/
+.pytest_cache/
+
+# PyInstaller
+*.manifest
+*.spec
+
+# Type checkers
+.mypy_cache/
+.pyre/
+.pytype/
+
+# Linter cache
+.ruff_cache/
+
+# Tool / build cache
+.cache/
+
+# IDEs and editors
+.vscode/
+.idea/
+*.sublime-project
+*.sublime-workspace
+
+# Track VS Code MCP config only
+!.vscode/
+.vscode/*
+!.vscode/mcp.json
+
+# OS files
+.DS_Store
+Thumbs.db
+desktop.ini
+
+# Logs and databases
+*.log
+*.sqlite3
+logs/
+
+# Wheel metadata
+pip-wheel-metadata/
+
+# Python egg
+*.egg
+*.egg-info/
+
+# Temporary
+*.tmp
+
+# Generated site (Zensical / MkDocs)
+site/

dataenginex-0.4.11/CHANGELOG.md

@@ -0,0 +1,138 @@
+# Changelog
+
+All notable changes to `dataenginex` will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.4.11] - 2026-02-27
+
+### Changed
+
+- Added `environment` label support across HTTP metrics counters/histograms/gauges and middleware emission.
+- Aligned alert rule histogram quantile expressions with explicit bucket aggregation by `le` and `environment`.
+- Standardized docs and release prep metadata for CSV-canonical roadmap and setup workflow updates.
+
+## [0.4.10] - 2026-02-21
+
+### Added
+
+- `examples/` directory with 4 runnable quickstart scripts
+- `01_hello_pipeline.py` — profiler, schema validation, medallion config
+- `02_api_quickstart.py` — FastAPI app with health, v1 router, metrics
+- `03_quality_gate.py` — QualityGate evaluations against layer thresholds
+- `04_ml_training.py` — SklearnTrainer, ModelRegistry, DriftDetector demo
+- `examples/README.md` with table of examples and run instructions
+
+## [0.4.8] - 2026-02-21
+
+### Added
+
+- PySpark local-mode test fixtures in `tests/conftest.py` (session-scoped `spark` session)
+- Sample DataFrame fixtures: `spark_df_jobs`, `spark_df_weather`, `spark_df_empty`
+- `requires_pyspark` skip marker — tests auto-skip when PySpark is not installed
+- `tests/fixtures/sample_data.py` — factory helpers for job, user, and weather records
+- `tests/unit/test_spark_fixtures.py` — validates PySpark fixture behaviour
+
+## [0.4.6] - 2026-02-21
+
+### Added
+
+- `QualityGate` — orchestrates quality checks at medallion layer transitions
+- `QualityStore` — in-memory store accumulating per-layer quality metrics
+- `QualityResult` — immutable dataclass capturing evaluation outcomes
+- `QualityDimension` — StrEnum for named quality dimensions
+- `/api/v1/data/quality/{layer}` endpoint for per-layer quality history
+- `set_quality_store()` / `get_quality_store()` for wiring quality at app startup
+- New exports in `dataenginex.core` and `dataenginex.api`
+
+### Changed
+
+- `/api/v1/data/quality` now returns live metrics from `QualityStore` (was placeholder zeros)
+- Wired `DataProfiler`, `DataQualityChecks`, and `QualityScorer` into `QualityGate` pipeline
+
+## [0.4.5] - 2026-02-21
+
+### Added
+
+- `StorageBackend` ABC with proper `@abstractmethod` contracts
+- `S3Storage` backend for AWS S3 (requires `boto3`)
+- `GCSStorage` backend for Google Cloud Storage (requires `google-cloud-storage`)
+- Re-exported `StorageBackend` from `dataenginex.lakehouse`
+
+### Changed
+
+- Refactored `StorageBackend` from plain class to proper `ABC` subclass
+- Updated `lakehouse.__init__` to export all 4 storage backends + ABC
+
+## [0.4.3] - 2026-02-21
+
+### Added
+
+- Comprehensive attribute-level docstrings on all public dataclasses
+- `from __future__ import annotations` in all source modules
+- Module-level class/function inventory docstrings
+- mkdocs API reference configuration with `mkdocstrings` plugin
+- API reference pages for all 7 subpackages under `docs/api-reference/`
+
+### Changed
+
+- Upgraded mkdocs theme from `mkdocs` to `material`
+- Enhanced module docstrings in middleware, core, and validators
+
+## [0.4.1] - 2026-02-21
+
+### Added
+
+- CHANGELOG.md with Keep a Changelog format
+- Release workflow extracts changelog notes for GitHub Releases automatically
+
+### Changed
+
+- `release.yml` now reads `packages/dataenginex/CHANGELOG.md` for release notes
+
+## [0.4.0] - 2026-02-21
+
+### Added
+
+- Stable `__all__` exports in every subpackage `__init__.py`
+- `from __future__ import annotations` in all public modules
+- Comprehensive module-level docstrings with usage examples
+- New public API exports: `ComponentHealth`, `AuthMiddleware`, `AuthUser`,
+  `create_token`, `decode_token`, `BadRequestError`, `NotFoundError`,
+  `PaginationMeta`, `RateLimiter`, `RateLimitMiddleware`,
+  `ConnectorStatus`, `FetchResult`, `ColumnProfile`, `get_logger`, `get_tracer`
+
+### Changed
+
+- Reorganized `__all__` in all subpackages for logical grouping
+- Updated package version to 0.4.0
+
+## [0.3.5] - 2026-02-13
+
+### Added
+
+- Production hardening: structured logging, Prometheus/OTel, health probes
+- Data connectors: `RestConnector`, `FileConnector` with async interface
+- Schema registry with versioned schema management
+- Data profiler with automated dataset statistics
+- Lakehouse catalog, partitioning, and storage backends
+- ML framework: trainer, model registry, drift detection, serving
+- Warehouse transforms and persistent lineage tracking
+- JWT authentication middleware
+- Rate limiting middleware
+- Cursor-based pagination utilities
+- Versioned API router (`/api/v1/`)
+
+[Unreleased]: https://github.com/TheDataEngineX/DEX/compare/v0.4.11...HEAD
+[0.4.11]: https://github.com/TheDataEngineX/DEX/compare/v0.4.10...v0.4.11
+[0.4.10]: https://github.com/TheDataEngineX/DEX/compare/v0.4.8...v0.4.10
+[0.4.8]: https://github.com/TheDataEngineX/DEX/compare/v0.4.6...v0.4.8
+[0.4.6]: https://github.com/TheDataEngineX/DEX/compare/v0.4.5...v0.4.6
+[0.4.5]: https://github.com/TheDataEngineX/DEX/compare/v0.4.3...v0.4.5
+[0.4.3]: https://github.com/TheDataEngineX/DEX/compare/v0.4.1...v0.4.3
+[0.4.1]: https://github.com/TheDataEngineX/DEX/compare/v0.4.0...v0.4.1
+[0.4.0]: https://github.com/TheDataEngineX/DEX/compare/v0.3.5...v0.4.0
+[0.3.5]: https://github.com/TheDataEngineX/DEX/releases/tag/v0.3.5

dataenginex-0.4.11/PKG-INFO

@@ -0,0 +1,71 @@
+Metadata-Version: 2.4
+Name: dataenginex
+Version: 0.4.11
+Summary: DataEngineX - Core framework for data engineering projects
+Author-email: Jay <jayapal.myaka99@gmail.com>
+License: MIT
+Requires-Python: >=3.11
+Requires-Dist: email-validator>=2.0.0
+Requires-Dist: fastapi>=0.128.4
+Requires-Dist: httpx>=0.28.0
+Requires-Dist: loguru>=0.7.3
+Requires-Dist: opentelemetry-api>=1.39.0
+Requires-Dist: opentelemetry-exporter-otlp>=1.39.0
+Requires-Dist: opentelemetry-instrumentation-fastapi>=0.60b1
+Requires-Dist: opentelemetry-sdk>=1.39.0
+Requires-Dist: prometheus-client>=0.24.0
+Requires-Dist: python-dotenv>=1.2.0
+Requires-Dist: python-json-logger>=4.0.0
+Requires-Dist: pyyaml>=6.0.2
+Requires-Dist: structlog>=25.5.0
+Requires-Dist: uvicorn>=0.40.0
+Provides-Extra: all
+Requires-Dist: boto3>=1.35.0; extra == 'all'
+Requires-Dist: google-cloud-storage>=2.18.0; extra == 'all'
+Requires-Dist: pyarrow>=18.0.0; extra == 'all'
+Provides-Extra: cloud
+Requires-Dist: boto3>=1.35.0; extra == 'cloud'
+Requires-Dist: google-cloud-storage>=2.18.0; extra == 'cloud'
+Provides-Extra: gcs
+Requires-Dist: google-cloud-storage>=2.18.0; extra == 'gcs'
+Provides-Extra: parquet
+Requires-Dist: pyarrow>=18.0.0; extra == 'parquet'
+Provides-Extra: s3
+Requires-Dist: boto3>=1.35.0; extra == 's3'
+Description-Content-Type: text/markdown
+
+# dataenginex
+
+`dataenginex` is the core DataEngineX framework package for building observable, production-ready data and API services.
+
+It provides:
+- FastAPI application primitives and API extensions
+- Middleware for structured logging, metrics, and tracing
+- Data quality and validation utilities
+- Lakehouse and warehouse building blocks
+- Reusable ML support modules for model-serving workflows
+
+## Install
+
+```bash
+pip install dataenginex
+```
+
+## Package Scope
+
+This package is the core library from the DEX monorepo.
+`careerdex` and `weatherdex` are maintained in the same repository but are not part of this package release flow.
+
+## Quick Usage
+
+```python
+from dataenginex import __version__
+
+print(__version__)
+```
+
+## Source and Docs
+
+- Repository: https://github.com/TheDataEngineX/DEX
+- CI/CD guide: `docs/CI_CD.md`
+- Release notes: `packages/dataenginex/src/dataenginex/RELEASE_NOTES.md`

{dataenginex-0.3.4 → dataenginex-0.4.11}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "dataenginex"
-version = "0.3.4"
+version = "0.4.11"
 description = "DataEngineX - Core framework for data engineering projects"
 authors = [
     {name = "Jay", email = "jayapal.myaka99@gmail.com"}
@@ -25,13 +25,23 @@ dependencies = [
     "httpx>=0.28.0",
 ]
 
-[build-system]
-requires = ["poetry-core>=2.0.0,<3.0.0"]
-build-backend = "poetry.core.masonry.api"
+[project.optional-dependencies]
+s3 = ["boto3>=1.35.0"]
+gcs = ["google-cloud-storage>=2.18.0"]
+cloud = [
+    "boto3>=1.35.0",
+    "google-cloud-storage>=2.18.0",
+]
+parquet = ["pyarrow>=18.0.0"]
+all = [
+    "boto3>=1.35.0",
+    "google-cloud-storage>=2.18.0",
+    "pyarrow>=18.0.0",
+]
 
-[tool.poetry]
-package-mode = true
+[build-system]
+requires = ["hatchling>=1.25.0"]
+build-backend = "hatchling.build"
 
-[[tool.poetry.packages]]
-include = "dataenginex"
-from = "src"
+[tool.hatch.build.targets.wheel]
+packages = ["src/dataenginex"]

{dataenginex-0.3.4 → dataenginex-0.4.11}/src/dataenginex/README.md

@@ -30,6 +30,6 @@ print(__version__)
 
 ## Source and Docs
 
-- Repository: https://github.com/data-literate/DEX
+- Repository: https://github.com/TheDataEngineX/DEX
 - CI/CD guide: `docs/CI_CD.md`
 - Release notes: `packages/dataenginex/src/dataenginex/RELEASE_NOTES.md`

{dataenginex-0.3.4 → dataenginex-0.4.11}/src/dataenginex/RELEASE_NOTES.md

@@ -3,6 +3,16 @@
 This document tracks published package releases for `dataenginex` only.
 Only include changes that modify files under `packages/dataenginex/src/dataenginex/**`.
 
+## v0.3.5 - 2026-02-20
+
+- Released package version `0.3.5`.
+- Tag: `v0.3.5`
+- Release title: `Release v0.3.5`
+- Changes in this release:
+	- Hardened `PyPI Publish` workflow for trusted publishing.
+	- Added release-only publish gating and build-only behavior for manual dispatch.
+	- Switched publish job environment names to repo vars for stricter workflow validation compatibility.
+
 ## v0.3.4 - 2026-02-20
 
 - Released package version `0.3.4`.

dataenginex-0.4.11/src/dataenginex/__init__.py

@@ -0,0 +1,34 @@
+"""
+DataEngineX (DEX) — Core framework for data engineering projects.
+
+Public API surface. Import from top-level or from subpackages:
+
+    from dataenginex import __version__
+    from dataenginex.api import HealthChecker, HealthStatus
+    from dataenginex.core import MedallionArchitecture, DataLayer
+    from dataenginex.data import DataConnector, DataProfiler, SchemaRegistry
+    from dataenginex.lakehouse import DataCatalog, ParquetStorage
+    from dataenginex.middleware import configure_logging, configure_tracing
+    from dataenginex.ml import ModelRegistry, SklearnTrainer, DriftDetector
+    from dataenginex.warehouse import PersistentLineage, TransformPipeline
+
+Submodules:
+    api        – FastAPI application, health checks, error handling, pagination
+    core       – Schemas, validators, medallion architecture, pipeline config
+    data       – Data connectors, profiler, schema registry
+    lakehouse  – Storage backends, data catalog, partitioning
+    middleware – Logging, metrics, tracing, request middleware
+    ml         – ML training, model registry, drift detection, serving
+    warehouse  – Transforms, persistent lineage tracking
+"""
+
+from __future__ import annotations
+
+from importlib.metadata import PackageNotFoundError, version
+
+try:
+    __version__ = version("dataenginex")
+except PackageNotFoundError:
+    __version__ = "0.4.11"
+
+__all__ = ["__version__"]

dataenginex-0.4.11/src/dataenginex/api/__init__.py

@@ -0,0 +1,54 @@
+"""Reusable API components — auth, health, errors, pagination, rate limiting, quality.
+
+Public API::
+
+    from dataenginex.api import (
+        HealthChecker, HealthStatus, ComponentHealth,
+        APIHTTPException, BadRequestError, NotFoundError, ServiceUnavailableError,
+        PaginatedResponse, paginate,
+        AuthMiddleware, AuthUser, create_token, decode_token,
+        RateLimiter, RateLimitMiddleware,
+        get_quality_store, set_quality_store,
+    )
+"""
+
+from __future__ import annotations
+
+from .auth import AuthMiddleware, AuthUser, create_token, decode_token
+from .errors import (
+    APIHTTPException,
+    BadRequestError,
+    NotFoundError,
+    ServiceUnavailableError,
+)
+from .health import ComponentHealth, HealthChecker, HealthStatus
+from .pagination import PaginatedResponse, PaginationMeta, paginate
+from .rate_limit import RateLimiter, RateLimitMiddleware
+from .routers.v1 import get_quality_store, set_quality_store
+
+__all__ = [
+    # Auth
+    "AuthMiddleware",
+    "AuthUser",
+    "create_token",
+    "decode_token",
+    # Errors
+    "APIHTTPException",
+    "BadRequestError",
+    "NotFoundError",
+    "ServiceUnavailableError",
+    # Health
+    "ComponentHealth",
+    "HealthChecker",
+    "HealthStatus",
+    # Pagination
+    "PaginatedResponse",
+    "PaginationMeta",
+    "paginate",
+    # Quality store
+    "get_quality_store",
+    "set_quality_store",
+    # Rate limiting
+    "RateLimiter",
+    "RateLimitMiddleware",
+]

{dataenginex-0.3.4 → dataenginex-0.4.11}/src/dataenginex/api/health.py

@@ -22,6 +22,15 @@ class HealthStatus(StrEnum):
 
 @dataclass(frozen=True)
 class ComponentHealth:
+    """Health status of a single dependency component.
+
+    Attributes:
+        name: Component identifier (e.g. ``"database"``, ``"cache"``).
+        status: Current health status.
+        message: Optional human-readable message.
+        duration_ms: Time taken for the health check in milliseconds.
+    """
+
     name: str
     status: HealthStatus
     message: str | None = None
@@ -129,5 +138,5 @@ class HealthChecker:
                 asyncio.open_connection(host, port), timeout=self.timeout_seconds
             )
             return True, "reachable"
-        except (TimeoutError, OSError) as exc:
+        except (TimeoutError, ConnectionRefusedError, OSError) as exc:
             return False, f"error={exc.__class__.__name__}"

dataenginex-0.4.11/src/dataenginex/api/routers/__init__.py

@@ -0,0 +1,3 @@
+"""API routers package."""
+
+from __future__ import annotations

{dataenginex-0.3.4 → dataenginex-0.4.11}/src/dataenginex/api/routers/v1.py

@@ -12,9 +12,25 @@ from typing import Any
 from fastapi import APIRouter
 
 from dataenginex.api.pagination import PaginatedResponse, paginate
+from dataenginex.core.quality import QualityStore
 
 router = APIRouter(prefix="/api/v1", tags=["v1"])
 
+# Module-level quality store — shared across requests.
+# Populate via ``set_quality_store()`` from application startup.
+_quality_store: QualityStore = QualityStore()
+
+
+def set_quality_store(store: QualityStore) -> None:
+    """Replace the module-level quality store (call at app startup)."""
+    global _quality_store  # noqa: PLW0603
+    _quality_store = store
+
+
+def get_quality_store() -> QualityStore:
+    """Return the active quality store."""
+    return _quality_store
+
 
 # ---------------------------------------------------------------------------
 # Data pipeline endpoints
@@ -35,28 +51,28 @@ def list_data_sources(cursor: str | None = None, limit: int = 20) -> PaginatedRe
 
 @router.get("/data/quality")
 def data_quality_summary() -> dict[str, Any]:
-    """Return a summary of data quality metrics.
+    """Return a summary of data quality metrics from the quality store.
+
+    Returns live metrics when a ``QualityStore`` has been populated via
+    ``QualityGate.evaluate()``.  Falls back to zeros when no evaluations
+    have been recorded yet.
+    """
+    return _quality_store.summary()
+
+
+@router.get("/data/quality/{layer}")
+def data_quality_layer(layer: str, limit: int = 10) -> dict[str, Any]:
+    """Return quality history for a specific medallion layer.
 
-    .. note::
-        Scores are **placeholder values** until the quality-tracking
-        subsystem is wired in.  See Issue backlog for live metrics epic.
+    Args:
+        layer: One of ``bronze``, ``silver``, ``gold``.
+        limit: Maximum number of history entries to return.
     """
-    # TODO(#future): Replace with live metrics from DataProfiler / quality store
+    latest = _quality_store.latest(layer)
     return {
-        "overall_score": 0.0,
-        "dimensions": {
-            "completeness": 0.0,
-            "accuracy": 0.0,
-            "consistency": 0.0,
-            "timeliness": 0.0,
-            "uniqueness": 0.0,
-        },
-        "layer_scores": {
-            "bronze": 0.0,
-            "silver": 0.0,
-            "gold": 0.0,
-        },
-        "_note": "Placeholder — connect quality store for live data",
+        "layer": layer,
+        "latest": latest.to_dict() if latest else None,
+        "history": _quality_store.history(layer, limit=limit),
     }
 
 

dataenginex-0.4.11/src/dataenginex/core/__init__.py

@@ -0,0 +1,81 @@
+"""Core framework — schemas, validators, medallion architecture, pipeline config, quality.
+
+Public API::
+
+    from dataenginex.core import (
+        # Medallion
+        MedallionArchitecture, DataLayer, StorageFormat, LayerConfiguration,
+        # Pipeline
+        PipelineConfig, PipelineMetrics,
+        # Quality
+        QualityGate, QualityStore, QualityResult, QualityDimension,
+        # Schemas
+        JobPosting, JobSourceEnum, UserProfile,
+        ErrorDetail, ErrorResponse, RootResponse, HealthResponse,
+        DataQualityReport, PipelineExecutionMetadata,
+        # Validators
+        SchemaValidator, DataQualityChecks, DataHash,
+        QualityScorer, ValidationReport,
+    )
+"""
+
+from __future__ import annotations
+
+from .medallion_architecture import (
+    DataLayer,
+    LayerConfiguration,
+    MedallionArchitecture,
+    StorageFormat,
+)
+from .pipeline_config import PipelineConfig, PipelineMetrics
+from .quality import QualityDimension, QualityGate, QualityResult, QualityStore
+from .schemas import (
+    DataQualityReport,
+    ErrorDetail,
+    ErrorResponse,
+    HealthResponse,
+    JobPosting,
+    JobSourceEnum,
+    PipelineExecutionMetadata,
+    RootResponse,
+    UserProfile,
+)
+from .validators import (
+    DataHash,
+    DataQualityChecks,
+    QualityScorer,
+    SchemaValidator,
+    ValidationReport,
+)
+
+__all__ = [
+    # Medallion architecture
+    "DataLayer",
+    "LayerConfiguration",
+    "MedallionArchitecture",
+    "StorageFormat",
+    # Pipeline
+    "PipelineConfig",
+    "PipelineMetrics",
+    # Quality gate
+    "QualityDimension",
+    "QualityGate",
+    "QualityResult",
+    "QualityStore",
+    # Schemas
+    "DataQualityReport",
+    "ErrorDetail",
+    "ErrorResponse",
+    "HealthResponse",
+    "JobPosting",
+    "JobSourceEnum",
+    "PipelineExecutionMetadata",
+    "RootResponse",
+    "UserProfile",
+    # Validators
+    "DataHash",
+    "DataQualityChecks",
+    "QualityScorer",
+    "SchemaValidator",
+    "ValidationReport",
+]