dataenginex 0.4.1__tar.gz → 0.4.2__tar.gz

{dataenginex-0.4.1 → dataenginex-0.4.2}/CHANGELOG.md

@@ -7,6 +7,18 @@ All notable changes to `dataenginex` will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.4.2] - 2026-06-23
+
+### Added
+
+- Example scripts refreshed: `08_spark_ml.py`, `09_feature_engineering.py`, `10_model_analysis.py` — demonstrate PySpark ML, feature transforms, and drift detection
+- `docs/release-notes.md` rewritten — cleaned up pre-reset 1.x entries, corrected version history to start from 0.3.5
+- `docs/observability.md` trimmed to library-level content only; HTTP/K8s content moved to `dex-studio/docs/observability.md`
+
+### Changed
+
+- Version bumped to 0.4.2
+
 ## [0.4.1] - 2026-06-12
 
 ### Added

{dataenginex-0.4.1 → dataenginex-0.4.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dataenginex
-Version: 0.4.1
+Version: 0.4.2
 Summary: DataEngineX — open-source, self-hosted, local-first Data + ML + AI workbench library
 Author-email: Jay <jayapal.myaka99@gmail.com>
 License: MIT License
@@ -30,9 +30,11 @@ Requires-Dist: click>=8.3.3
 Requires-Dist: croniter>=6.2.2
 Requires-Dist: duckdb>=1.5.2
 Requires-Dist: httpx>=0.28.1
+Requires-Dist: msgpack>=1.2.1
 Requires-Dist: orjson>=3.11.0
 Requires-Dist: prometheus-client>=0.25.0
 Requires-Dist: pyarrow>=23.0.1
+Requires-Dist: pydantic-settings>=2.14.2
 Requires-Dist: pydantic>=2.13.4
 Requires-Dist: pyyaml>=6.0.3
 Requires-Dist: structlog>=25.5.0

{dataenginex-0.4.1 → dataenginex-0.4.2}/docs/development.md

@@ -38,13 +38,13 @@ This installs all Python dependencies and configures pre-commit hooks.
 ```bash
 # 1. Clone repo and create feature branch
 git clone https://github.com/TheDataEngineX/dataenginex.git
-cd dex
+cd dataenginex
 git checkout -b feat/issue-XXX-description dev
 
-# 3. Install Python deps & pre-commit hooks
+# 2. Install Python deps & pre-commit hooks
 uv run poe setup
 
-# 4. Verify setup
+# 3. Verify setup
 uv run poe check-all
 ```
 
@@ -53,12 +53,11 @@ All tests and linting should pass. You're ready to develop!
 ## Project Structure
 
 ```
-DEX/
+dataenginex/
 ├── src/dataenginex/        # Core framework package
 ├── examples/               # Runnable example scripts (01–10)
 ├── tests/                  # Test suite
 ├── docs/                   # Documentation
-├── monitoring/             # Local observability stack configs
 ├── .github/workflows/      # CI/CD pipelines
 ├── pyproject.toml          # Project config
 └── poe_tasks.toml          # Task definitions
@@ -160,24 +159,7 @@ uv run poe test-unit
 uv run poe check-all
 ```
 
-### Monitoring & Debugging
-
-```bash
-# View application logs
-tail -f logs/app.log
-
-# Enable debug logging
-export LOG_LEVEL=DEBUG
-uv run poe dev
-
-# Use Python debugger
-python -m pdb examples/02_api_quickstart.py
-
-# Prometheus metrics (if running)
-open http://localhost:9090
-```
-
-## Troubleshooting
+### Troubleshooting
 
 | Issue | Solution |
 |-------|----------|
@@ -198,7 +180,7 @@ uv run poe test               # Run all tests
 uv run poe test-cov           # Tests with coverage report
 uv run poe security           # pip-audit vulnerability scan
 uv run poe pre-commit         # Run all pre-commit hooks
-uv run poe dev                # Run dev server (localhost:17000)
+uv run poe docker-up          # Run Docker Compose stack
 uv run poe clean              # Remove caches and build artifacts
 ```
 

{dataenginex-0.4.1 → dataenginex-0.4.2}/docs/index.md

@@ -23,4 +23,6 @@ See [Quickstart](quickstart.md) for a full walkthrough with a sample project.
 - [Quickstart](quickstart.md) — Get up and running in 5 minutes
 - [Architecture](architecture.md) — Core patterns, module map, design decisions
 - [Development Setup](development.md) — Prerequisites, workflow, troubleshooting
+- [Release Notes](release-notes.md) — Full version history
+- [Observability](observability.md) — Metrics, logging, tracing
 - [Contributing](contributing.md) — How to contribute to the project

dataenginex-0.4.2/docs/observability.md

@@ -0,0 +1,61 @@
+# Observability: Metrics, Logging & Tracing
+
+**Library-level observability for `dataenginex`.** For application-level monitoring (HTTP middleware, health endpoints, Grafana dashboards), see [dex-studio/docs/observability.md](https://github.com/TheDataEngineX/dex-studio/blob/main/docs/observability.md).
+
+## Logging
+
+`dataenginex` uses `structlog` for structured logging throughout the library. All loggers are configured by the host application (e.g., dex-studio). The library does not configure handlers itself.
+
+### Logging in your application
+
+```python
+import structlog
+from dataenginex.engine import DexEngine
+
+structlog.configure(
+    processors=[structlog.dev.ConsoleRenderer()],
+    wrapper_class=structlog.make_filtering_bound_logger(20),  # INFO
+)
+engine = DexEngine("dex.yaml")
+```
+
+## Metrics
+
+Prometheus metrics are exposed via `dataenginex.observability.metrics`:
+
+```python
+from dataenginex.observability.metrics import (
+    HTTP_REQUESTS_TOTAL,
+    HTTP_REQUEST_DURATION_SECONDS,
+    PIPELINE_RUN_DURATION,
+)
+```
+
+### Available library-level metrics
+
+| Metric | Type | Description | Labels |
+|--------|------|-------------|--------|
+| `pipeline_run_duration_seconds` | Histogram | Pipeline execution time | pipeline_name, status |
+| `model_prediction_latency_seconds` | Histogram | Model inference time | model_name |
+| `llm_request_duration_seconds` | Histogram | LLM call time | provider, model |
+| `data_connector_rows_read` | Counter | Rows read by source connectors | connector_type |
+
+## Tracing
+
+OpenTelemetry tracing is available via `dataenginex.tracing`:
+
+```python
+from dataenginex.tracing import get_tracer
+
+tracer = get_tracer(__name__)
+
+with tracer.start_as_current_span("pipeline_run") as span:
+    span.set_attribute("pipeline.name", "ingest")
+    engine.run_pipeline("ingest")
+```
+
+Enable OTLP export:
+
+```bash
+export OTLP_ENDPOINT="http://localhost:4317"
+```

dataenginex-0.4.2/docs/release-notes.md

@@ -0,0 +1,94 @@
+# Release Notes
+
+## [0.4.2] — 2026-06-23
+
+### Added
+
+- Example scripts refreshed: `08_spark_ml.py`, `09_feature_engineering.py`, `10_model_analysis.py` — demonstrate PySpark ML, feature transforms, and drift detection
+- Documentation cleanup across all docs
+
+### Changed
+
+- Version bumped to 0.4.2
+
+### Verification checklist
+
+1. `uv run poe lint` — Ruff checks clean
+1. `uv run poe typecheck` — mypy strict, 0 errors
+1. `uv run poe test` — all tests pass
+
+______________________________________________________________________
+
+## [0.4.1] - 2026-06-12
+
+### Added
+
+- `dataenginex._json` — drop-in `orjson`-backed JSON shim (`dumps`, `loads`, `JSONResponse`) replacing stdlib `json` across the library for ~3–5× serialization throughput
+- `DeltaConnector` — native Delta Lake read/write via `deltalake` (new `delta` optional extra: `pip install "dataenginex[delta]"`)
+- `ml.features.builtin` — built-in feature transformers: `StandardScalerTransform`, `MinMaxScalerTransform`, `OneHotEncoderTransform`, `PolynomialFeaturesTransform`
+- `core.interfaces` — new `Closeable` and `AsyncCloseable` protocols for uniform resource lifecycle
+- `orjson>=3.11.0` and `zstandard>=0.25.0` promoted to core runtime dependencies
+
+### Changed
+
+- **Lakehouse storage** (`lakehouse/storage.py`) — full rewrite: unified `LakehouseStorage` with pluggable backends (local, S3, GCS), Zstandard compression throughout, columnar partition pruning
+- **Lakehouse catalog** (`lakehouse/catalog.py`) — catalog entries now carry partition stats and schema fingerprints; `register` / `resolve` API stabilised
+- **ML registry** (`ml/registry.py`) — artifact versioning with aliasing (`promote_alias`), stage transitions (`development` → `staging` → `production`), and metadata search
+- **ML training** (`ml/training.py`) — `TrainingJob` lifecycle management, early-stopping callbacks, cross-validation harness, experiment comparison utilities
+- **Pagination** (`api/pagination.py`) — cursor-based and page-number strategies unified under `PaginationResult`; `paginate_query` helper works with any iterable
+- **Store** (`store.py`) — async-safe DuckDB connection pool, `get_pipeline_runs` and `list_model_artifacts` now return typed dataclasses
+- **SecOps audit** (`secops/audit.py`) — structured audit events with severity levels, retention policy enforcement, export to JSONL
+- **AI runtime** (`ai/runtime/executor.py`) — tool call concurrency limit, timeout per tool, structured error envelopes
+- **Config loader** (`config/loader.py`) — environment variable interpolation (`${VAR}`) and `include:` directive for config composition
+- `zstandard` used for pipeline run history compression reducing on-disk footprint by ~60%
+
+### Fixed
+
+- `mypy --strict` passes cleanly across all modules after strict type annotation pass
+- `DeltaConnector` and `LakehouseStorage` excluded from coverage thresholds (require live filesystems); coverage gate unchanged for all other modules
+
+______________________________________________________________________
+
+## [0.4.0] - 2026-02-21
+
+> **Scope reset from 1.x.** Versions 1.0.0–1.1.2 were prematurely tagged stable. Resetting to `0.4.0` to honestly reflect pre-1.0 maturity. See [ADR-0007](https://github.com/TheDataEngineX/docs/blob/main/adr/0007-local-first-scope-reset.md) for rationale. The 1.x versions on PyPI are yanked but remain installable by exact pin (`pip install 'dataenginex==1.1.2'`); plain `pip install dataenginex` now resolves to `0.4.0`.
+
+### Added
+
+- Stable `__all__` exports in every subpackage `__init__.py`
+- `from __future__ import annotations` in all public modules
+- Comprehensive module-level docstrings with usage examples
+- New public API exports: `ComponentHealth`, `AuthMiddleware`, `AuthUser`,
+  `create_token`, `decode_token`, `BadRequestError`, `NotFoundError`,
+  `PaginationMeta`, `RateLimiter`, `RateLimitMiddleware`,
+  `ConnectorStatus`, `FetchResult`, `ColumnProfile`, `get_logger`, `get_tracer`
+
+### Changed
+
+- Reorganized `__all__` in all subpackages for logical grouping
+- Updated package version to 0.4.0
+
+______________________________________________________________________
+
+## [0.3.5] - 2026-02-13
+
+### Added
+
+- Production hardening: structured logging, Prometheus/OTel, health probes
+- Data connectors: `RestConnector`, `FileConnector` with async interface
+- Schema registry with versioned schema management
+- Data profiler with automated dataset statistics
+- Lakehouse catalog, partitioning, and storage backends
+- ML framework: trainer, model registry, drift detection, serving
+- Warehouse transforms and persistent lineage tracking
+- JWT authentication middleware
+- Rate limiting middleware
+- Cursor-based pagination utilities
+- Versioned API router (`/api/v1/`)
+
+______________________________________________________________________
+
+[0.3.5]: https://github.com/TheDataEngineX/dataenginex/releases/tag/v0.3.5
+[0.4.0]: https://github.com/TheDataEngineX/dataenginex/compare/v0.3.5...v0.4.0
+[0.4.1]: https://github.com/TheDataEngineX/dataenginex/compare/v0.4.0...v0.4.1
+[0.4.2]: https://github.com/TheDataEngineX/dataenginex/compare/v0.4.1...v0.4.2