dataenginex 0.4.1__tar.gz → 0.4.3__tar.gz

{dataenginex-0.4.1 → dataenginex-0.4.3}/.github/workflows/ci.yml

@@ -19,7 +19,7 @@ jobs:
     name: Code Quality
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v7
       - uses: astral-sh/setup-uv@v8.2.0
         with:
           version: "latest"
@@ -34,7 +34,7 @@ jobs:
     runs-on: ubuntu-latest
     needs: quality
     steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v7
       - uses: astral-sh/setup-uv@v8.2.0
         with:
           version: "latest"
@@ -58,7 +58,7 @@ jobs:
       matrix:
         python-version: ["3.11", "3.12"]
     steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v7
       - uses: astral-sh/setup-uv@v8.2.0
         with:
           version: "latest"

{dataenginex-0.4.1 → dataenginex-0.4.3}/.github/workflows/release.yml

@@ -23,7 +23,7 @@ jobs:
     permissions:
       contents: read
     steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v7
         with:
           ref: ${{ inputs.ref || github.ref }}
       - uses: astral-sh/setup-uv@v8.2.0
@@ -58,7 +58,7 @@ jobs:
     permissions:
       contents: write
     steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v7
         with:
           ref: ${{ inputs.ref || github.ref }}
       - uses: astral-sh/setup-uv@v8.2.0

{dataenginex-0.4.1 → dataenginex-0.4.3}/CHANGELOG.md

@@ -7,6 +7,18 @@ All notable changes to `dataenginex` will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.4.2] - 2026-06-23
+
+### Added
+
+- Example scripts refreshed: `08_spark_ml.py`, `09_feature_engineering.py`, `10_model_analysis.py` — demonstrate PySpark ML, feature transforms, and drift detection
+- `docs/release-notes.md` rewritten — cleaned up pre-reset 1.x entries, corrected version history to start from 0.3.5
+- `docs/observability.md` trimmed to library-level content only; HTTP/K8s content moved to `dex-studio/docs/observability.md`
+
+### Changed
+
+- Version bumped to 0.4.2
+
 ## [0.4.1] - 2026-06-12
 
 ### Added

{dataenginex-0.4.1 → dataenginex-0.4.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dataenginex
-Version: 0.4.1
+Version: 0.4.3
 Summary: DataEngineX — open-source, self-hosted, local-first Data + ML + AI workbench library
 Author-email: Jay <jayapal.myaka99@gmail.com>
 License: MIT License
@@ -30,9 +30,11 @@ Requires-Dist: click>=8.3.3
 Requires-Dist: croniter>=6.2.2
 Requires-Dist: duckdb>=1.5.2
 Requires-Dist: httpx>=0.28.1
+Requires-Dist: msgpack>=1.2.1
 Requires-Dist: orjson>=3.11.0
 Requires-Dist: prometheus-client>=0.25.0
 Requires-Dist: pyarrow>=23.0.1
+Requires-Dist: pydantic-settings>=2.14.2
 Requires-Dist: pydantic>=2.13.4
 Requires-Dist: pyyaml>=6.0.3
 Requires-Dist: structlog>=25.5.0

{dataenginex-0.4.1 → dataenginex-0.4.3}/README.md

@@ -7,8 +7,6 @@
 
 The Python library that powers [DEX Studio](https://github.com/TheDataEngineX/dex-studio) — an open-source, self-hosted, local-first Data + ML + AI workbench for individuals and small teams. **Use the library directly when you want code; install DEX Studio when you want a UI.**
 
-> **Pre-1.0 status.** `0.4.0` is honest about it. See the [CHANGELOG scope-reset entry](CHANGELOG.md) for the rationale.
-
 ______________________________________________________________________
 
 ## Install

dataenginex-0.4.3/docs/api-reference/ai.md

@@ -0,0 +1,191 @@
+# dataenginex.ai
+
+LLM routing, agent runtimes, vector stores, memory, retrieval, observability, and workflow orchestration.
+
+## Quick import
+
+```python
+from dataenginex.ai import (
+    ModelRouter, BaseProvider,
+    LLMProvider, LLMResponse,
+    BuiltinAgentRuntime,
+    BuiltinRetriever,
+    VectorStore,
+    SandboxConfig,
+)
+```
+
+______________________________________________________________________
+
+## LLM Interface
+
+`dataenginex.ai.llm`
+
+Unified LLM request/response interface across providers. Handles streaming, tool calls, retries, and token counting.
+
+::: dataenginex.ai.llm
+
+**Key classes:** `LLMProvider`, `LLMResponse`, `LLMMessage`, `ToolCall`
+
+```python
+from dataenginex.ai.llm import LLMMessage
+
+response = provider.complete([
+    LLMMessage(role="user", content="Summarize this dataset."),
+])
+print(response.content)
+```
+
+______________________________________________________________________
+
+## Model Router
+
+`dataenginex.ai.routing.router`
+
+Routes LLM requests to the appropriate provider based on cost, latency, capability, and fallback rules.
+
+::: dataenginex.ai.routing.router
+
+**Key class:** `ModelRouter`
+
+```python
+from dataenginex.ai.routing.router import ModelRouter
+
+router = ModelRouter.from_config(engine.config)
+response = router.complete("Explain this error.", model_hint="fast")
+```
+
+### Providers
+
+`dataenginex.ai.routing.anthropic` · `dataenginex.ai.routing.openai` · `dataenginex.ai.routing.ollama` · `dataenginex.ai.routing.guarded`
+
+::: dataenginex.ai.routing.anthropic
+::: dataenginex.ai.routing.openai
+::: dataenginex.ai.routing.ollama
+::: dataenginex.ai.routing.guarded
+
+______________________________________________________________________
+
+## Agents
+
+`dataenginex.ai.agents.builtin`
+
+Built-in agent runtime — tool-use loop, memory injection, step tracing, and structured output parsing.
+
+::: dataenginex.ai.agents.builtin
+
+**Key class:** `BuiltinAgentRuntime`
+
+```python
+from dataenginex.ai.agents.builtin import BuiltinAgentRuntime
+
+agent = BuiltinAgentRuntime(router=router, tools=[search_tool, sql_tool])
+result = agent.run("Find the top 10 customers by revenue last quarter.")
+print(result.output)
+```
+
+______________________________________________________________________
+
+## Vector Store
+
+`dataenginex.ai.vectorstore`
+
+Embedding storage and similarity search. Defaults to in-process DuckDB VSS; swap for Qdrant via `dataenginex[qdrant]`.
+
+::: dataenginex.ai.vectorstore
+
+**Key class:** `VectorStore`
+
+```python
+from dataenginex.ai.vectorstore import VectorStore
+
+store = VectorStore(db_path=".dex/store.duckdb")
+store.upsert("doc-1", embedding=[0.1, 0.2, ...], metadata={"source": "wiki"})
+results = store.search(query_embedding, top_k=5)
+```
+
+______________________________________________________________________
+
+## Memory
+
+`dataenginex.ai.memory.base` — abstract memory interface
+
+::: dataenginex.ai.memory.base
+
+`dataenginex.ai.memory.episodic` — short-term conversation memory scoped to a single agent session
+
+::: dataenginex.ai.memory.episodic
+
+`dataenginex.ai.memory.long_term` — persistent memory backed by the vector store, survives across sessions
+
+::: dataenginex.ai.memory.long_term
+
+______________________________________________________________________
+
+## Retrieval
+
+`dataenginex.ai.retrieval.builtin` — RAG retriever: embeds query, searches vector store, returns ranked chunks
+
+::: dataenginex.ai.retrieval.builtin
+
+`dataenginex.ai.retrieval.graph` — graph-based retrieval for structured knowledge graphs
+
+::: dataenginex.ai.retrieval.graph
+
+______________________________________________________________________
+
+## Runtime
+
+`dataenginex.ai.runtime.executor` — async execution engine with concurrency, timeout, and step-level error handling
+
+::: dataenginex.ai.runtime.executor
+
+`dataenginex.ai.runtime.checkpoint` — saves and restores agent run state for long-running or resumable workflows
+
+::: dataenginex.ai.runtime.checkpoint
+
+`dataenginex.ai.runtime.sandbox` — isolated code execution sandbox for agent-generated Python with configurable resource limits
+
+::: dataenginex.ai.runtime.sandbox
+
+______________________________________________________________________
+
+## Tools
+
+`dataenginex.ai.tools.builtin`
+
+Built-in agent tools: `sql_query`, `web_search`, `file_read`, `python_exec`, `vector_search`.
+
+::: dataenginex.ai.tools.builtin
+
+______________________________________________________________________
+
+## Workflows
+
+`dataenginex.ai.workflows.dag` — multi-step agent workflows as DAGs; steps branch, merge, and pass structured outputs
+
+::: dataenginex.ai.workflows.dag
+
+`dataenginex.ai.workflows.conditions` — conditional branching logic for DAG workflows
+
+::: dataenginex.ai.workflows.conditions
+
+`dataenginex.ai.workflows.human_loop` — pause a workflow at a step requiring human review or approval
+
+::: dataenginex.ai.workflows.human_loop
+
+______________________________________________________________________
+
+## Observability
+
+`dataenginex.ai.observability.audit` — logs every LLM request/response, tool call, and agent step for compliance
+
+::: dataenginex.ai.observability.audit
+
+`dataenginex.ai.observability.cost` — tracks token usage and estimated cost per provider, model, and agent run
+
+::: dataenginex.ai.observability.cost
+
+`dataenginex.ai.observability.metrics` — Prometheus metrics for LLM latency, token throughput, error rate
+
+::: dataenginex.ai.observability.metrics

dataenginex-0.4.3/docs/api-reference/config.md

@@ -0,0 +1,84 @@
+# dataenginex.config
+
+Config schema, loader, settings, and defaults. The `dex.yaml` file is parsed into a typed `DexConfig` object that drives the engine.
+
+## Quick import
+
+```python
+from dataenginex.config import DexConfig, load_config, DexSettings
+```
+
+______________________________________________________________________
+
+## Schema
+
+`dataenginex.config.schema`
+
+Pydantic models for `dex.yaml`. Covers pipelines, connectors, transforms, ML, AI, secops, and lakehouse config blocks.
+
+::: dataenginex.config.schema
+
+**Key model:** `DexConfig`
+
+```python
+from dataenginex.config.schema import DexConfig
+
+config = DexConfig.model_validate({
+    "pipelines": {
+        "ingest_events": {
+            "source": {"type": "csv", "path": "data/raw/events.csv"},
+            "destination": {"type": "parquet", "path": "data/bronze/events"},
+        }
+    }
+})
+```
+
+______________________________________________________________________
+
+## Loader
+
+`dataenginex.config.loader`
+
+Loads and validates `dex.yaml` (or a custom path) with environment variable interpolation and secret resolution.
+
+::: dataenginex.config.loader
+
+**Key function:** `load_config`
+
+```python
+from dataenginex.config.loader import load_config
+
+config = load_config("dex.yaml")
+```
+
+______________________________________________________________________
+
+## Settings
+
+`dataenginex.config.settings`
+
+Runtime settings sourced from environment variables (`.env` or system env). Covers API keys, storage paths, log level, and feature flags.
+
+::: dataenginex.config.settings
+
+**Key class:** `DexSettings`
+
+```python
+from dataenginex.config.settings import DexSettings
+
+settings = DexSettings()
+print(settings.log_level)       # "INFO"
+print(settings.dex_store_path)  # ".dex/store.duckdb"
+```
+
+See `.env.template` in the project root for all available settings.
+
+______________________________________________________________________
+
+## Defaults
+
+`dataenginex.config.defaults`
+
+Default values applied when config keys are omitted.
+
+::: dataenginex.config.defaults

dataenginex-0.4.3/docs/api-reference/core.md

@@ -0,0 +1,143 @@
+# dataenginex.core
+
+Foundation layer — interfaces, medallion architecture, quality gates, registries, exceptions, and shared schemas.
+
+## Quick import
+
+```python
+from dataenginex.core import (
+    MedallionConfig, Layer,
+    QualityGate, QualityCheck, QualityResult, Severity,
+    BaseConnector, BaseTransform, BaseRunner,
+    NotFoundError, ValidationError, PipelineError,
+    ComponentRegistry,
+)
+```
+
+______________________________________________________________________
+
+## Medallion Architecture
+
+`dataenginex.core.medallion_architecture`
+
+Defines the Bronze → Silver → Gold layer model and per-layer configuration.
+
+::: dataenginex.core.medallion_architecture
+
+**Key classes:** `MedallionConfig`, `Layer`, `LayerConfig`, `MedallionPipeline`
+
+```python
+from dataenginex.core.medallion_architecture import Layer, MedallionConfig, LayerConfig
+
+cfg = MedallionConfig(
+    bronze=LayerConfig(path="data/bronze"),
+    silver=LayerConfig(path="data/silver", quality_threshold=0.95),
+    gold=LayerConfig(path="data/gold", quality_threshold=0.99),
+)
+```
+
+______________________________________________________________________
+
+## Quality Gates
+
+`dataenginex.core.quality`
+
+Declarative data quality checks that run at layer promotion boundaries.
+
+::: dataenginex.core.quality
+
+**Key classes:** `QualityCheck`, `QualityGate`, `QualityResult`, `Severity`
+
+```python
+from dataenginex.core.quality import QualityCheck, QualityGate, Severity
+
+gate = QualityGate(
+    checks=[
+        QualityCheck(name="no_nulls", column="user_id", check_type="not_null"),
+        QualityCheck(
+            name="email_format",
+            column="email",
+            check_type="regex",
+            pattern=r"^[^@]+@[^@]+\.[^@]+$",
+            severity=Severity.ERROR,
+        ),
+    ]
+)
+result = gate.run(df)
+assert result.passed
+```
+
+______________________________________________________________________
+
+## Interfaces
+
+`dataenginex.core.interfaces`
+
+Abstract base classes for connectors, transforms, runners, and storage backends. Implement these to extend DEX with custom components.
+
+::: dataenginex.core.interfaces
+
+**Key classes:** `BaseConnector`, `BaseTransform`, `BaseRunner`, `BaseStorage`, `BaseProfiler`
+
+______________________________________________________________________
+
+## Registry
+
+`dataenginex.core.registry`
+
+Generic component registry used by connectors, transforms, models, and agents.
+
+::: dataenginex.core.registry
+
+**Key class:** `ComponentRegistry`
+
+```python
+from dataenginex.core.registry import ComponentRegistry
+
+registry: ComponentRegistry[MyPlugin] = ComponentRegistry()
+registry.register("my_plugin", MyPlugin)
+plugin = registry.get("my_plugin")
+```
+
+______________________________________________________________________
+
+## Schemas
+
+`dataenginex.core.schemas`
+
+Shared Pydantic response models used across the engine, CLI, and API layers.
+
+::: dataenginex.core.schemas
+
+**Key models:** `HealthResponse`, `RootResponse`, `EchoRequest`, `EchoResponse`, `ErrorResponse`, `ComponentStatus`
+
+______________________________________________________________________
+
+## Validators
+
+`dataenginex.core.validators`
+
+Reusable Pydantic validators and field helpers for config schema enforcement.
+
+::: dataenginex.core.validators
+
+______________________________________________________________________
+
+## Exceptions
+
+`dataenginex.core.exceptions`
+
+Typed exception hierarchy for pipeline, validation, and resource errors.
+
+::: dataenginex.core.exceptions
+
+**Key exceptions:** `DexError`, `NotFoundError`, `ValidationError`, `PipelineError`, `RegistryError`, `ConfigError`
+
+```python
+from dataenginex.core.exceptions import NotFoundError, PipelineError
+
+try:
+    engine.run_pipeline("missing")
+except NotFoundError as e:
+    print(e.detail)
+```