aptdata 0.0.2__tar.gz

aptdata-0.0.2/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 strondata
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

aptdata-0.0.2/PKG-INFO

@@ -0,0 +1,330 @@
+Metadata-Version: 2.4
+Name: aptdata
+Version: 0.0.2
+Summary: A declarative, extensible framework for building smart data pipelines in Python
+License: MIT
+License-File: LICENSE
+Keywords: data-pipeline,framework,etl,pydantic,data-engineering
+Author: strondata
+Requires-Python: >=3.10,<4.0
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Software Development :: Libraries :: Application Frameworks
+Provides-Extra: all
+Provides-Extra: pandas
+Provides-Extra: plugins
+Provides-Extra: spark
+Requires-Dist: httpx (>=0.27,<0.28) ; extra == "plugins" or extra == "all"
+Requires-Dist: mcp (>=1.26.0,<2.0.0)
+Requires-Dist: opentelemetry-api (>=1.40.0,<2.0.0)
+Requires-Dist: opentelemetry-sdk (>=1.40.0,<2.0.0)
+Requires-Dist: pandas (>=2.2,<3.0) ; extra == "pandas" or extra == "all"
+Requires-Dist: psycopg2-binary (>=2.9,<3.0) ; extra == "plugins" or extra == "all"
+Requires-Dist: pyarrow (>=15.0,<16.0) ; extra == "plugins" or extra == "all"
+Requires-Dist: pydantic (>=2.0,<3.0)
+Requires-Dist: pyspark (>=3.5,<4.0) ; extra == "spark" or extra == "all"
+Requires-Dist: python-dotenv (>=1.0,<2.0)
+Requires-Dist: pyyaml (>=6.0,<7.0)
+Requires-Dist: questionary (>=2.0)
+Requires-Dist: rich (>=13.0,<14.0)
+Requires-Dist: sqlalchemy (>=2.0,<3.0) ; extra == "plugins" or extra == "all"
+Requires-Dist: textual (>=0.60,<0.61)
+Requires-Dist: typer[all] (>=0.15,<0.16)
+Project-URL: Documentation, https://strondata.github.io/smart-data
+Project-URL: Homepage, https://strondata.github.io/smart-data
+Project-URL: Repository, https://github.com/strondata/smart-data
+Description-Content-Type: text/markdown
+
+# aptdata
+
+> **v0.0.2** · A declarative, extensible framework for building smart data pipelines in Python.
+
+[![Python](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/)
+[![License](https://img.shields.io/badge/license-MIT-green)](LICENSE)
+[![Version](https://img.shields.io/badge/version-0.0.2-orange)](CHANGELOG.md)
+
+---
+
+## Overview
+
+**aptdata** is built around three universal abstractions — **System**,
+**Flow**, and **Component** — that cover every data-processing paradigm in a
+single, coherent model:
+
+```mermaid
+flowchart TD
+    I["IComponent / IFlow / ISystem\n@dataclass + ABC — pure interfaces"]
+    B["BaseComponent / BaseFlow / BaseSystem\n@pydantic_dataclass — validated fields"]
+    Y["Your concrete implementations"]
+
+    I --> B --> Y
+```
+
+Datasets remain the fundamental data-exchange contract (`IDataset` /
+`BaseDataset`).  Every outcome from the CLI is emitted as a machine-readable
+JSON line, making aptdata a natural fit for AI orchestrators, CI/CD
+pipelines and scripted workflows.
+
+---
+
+## Requirements
+
+- Python ≥ 3.10
+- [Poetry](https://python-poetry.org/) (for development)
+
+---
+
+## Installation
+
+### From PyPI
+
+```bash
+pip install aptdata
+```
+
+### Optional extras
+
+```bash
+pip install aptdata[pandas]   # pandas support
+pip install aptdata[spark]    # PySpark support
+pip install aptdata[plugins]  # REST, PostgreSQL, Parquet I/O
+pip install aptdata[all]      # everything
+```
+
+### From source (development)
+
+```bash
+git clone https://github.com/strondata/smart-data.git
+cd aptdata
+poetry install
+```
+
+---
+
+## Quick start
+
+```python
+from pydantic.dataclasses import dataclass as pydantic_dataclass
+from aptdata.core import (
+    BaseDataset, IDataset,
+    BaseComponent, ComponentMeta, ComponentKind,
+    BaseFlow, IFlow,
+    BaseSystem,
+)
+
+@pydantic_dataclass
+class MemoryDataset(BaseDataset):
+    def __post_init__(self): self._data = None
+    def read(self): return self._data
+    def write(self, data): self._data = data
+
+@pydantic_dataclass
+class DoubleComponent(BaseComponent):
+    def validate_inputs(self, inputs: list[IDataset]) -> bool:
+        return len(inputs) == 1
+    def execute(self, inputs: list[IDataset]) -> list[IDataset]:
+        out = MemoryDataset(uri="memory://out")
+        out.write([x * 2 for x in inputs[0].read()])
+        return [out]
+
+@pydantic_dataclass
+class ETLFlow(BaseFlow):
+    def __post_init__(self):
+        self._nodes = {}
+        self._edges = []
+        self._compiled = False
+    def add_component(self, c): self._nodes[c.component_id] = c
+    def connect(self, src, tgt, condition=None): ...
+    def compile(self): self._compiled = True
+    def run(self, inputs): return inputs  # wire your logic here
+
+@pydantic_dataclass
+class MySystem(BaseSystem):
+    def __post_init__(self): self._flows: list[IFlow] = []
+    def register_flow(self, flow): self._flows.append(flow)
+    def run(self):
+        for flow in self._flows:
+            flow.run([])
+
+# Register and run via CLI
+from aptdata.plugins import registry
+registry.register("my_system", MySystem)
+```
+
+```bash
+aptdata run my_system
+# {"event": "pipeline.started", "pipeline": "my_system", "env": "dev", "dry_run": false, "trace_id": null}
+# {"event": "pipeline.completed", "pipeline": "my_system", "env": "dev", "dry_run": false, "elapsed_seconds": 0.001, "trace_id": null}
+```
+
+---
+
+## CLI reference
+
+```
+aptdata run SYSTEM_NAME [--env ENV] [--dry-run]
+aptdata monitor [--refresh SECONDS]
+aptdata scaffold PROJECT_NAME [--template TEMPLATE] [--output PATH]
+aptdata schema export --output schema.json
+aptdata system list [--json]
+aptdata system info NAME [--json]
+aptdata system validate NAME
+aptdata plugin list [--json]
+aptdata plugin inspect NAME [--json]
+aptdata plugin preview READER [--limit N]
+aptdata plugin load MODULE_PATH
+aptdata config validate PATH
+aptdata config init [--output PATH]
+aptdata config show PATH
+aptdata config run PATH [--env ENV]
+aptdata telemetry status [--json]
+aptdata telemetry export [--format json]
+aptdata mesh list [--dir DIR] [--json]
+aptdata mesh run COMPONENT [--dir DIR] [--dry-run] [--json]
+aptdata mesh build COMPONENT [--dir DIR] [--json]
+aptdata mcp-start [--transport TRANSPORT]
+aptdata interactive
+```
+
+Every static command supports `--json` for machine-readable JSON line output
+(backward compatible). Without `--json`, commands render Rich tables, panels,
+and syntax-highlighted output.
+
+### Scaffold templates
+
+| Template              | Description                                         |
+|-----------------------|-----------------------------------------------------|
+| `hello-world`         | Minimal pandas pipeline (default)                   |
+| `medallion`           | Bronze → Silver → Gold data lakehouse               |
+| `rag-ingestion`       | RAG pipeline: extract → chunk → embed → load        |
+| `data-quality-test`   | Schema contract + expectation suite                  |
+| `job-wheel`           | Python wheel executor for portable job packaging     |
+| `docker-compose-app`  | Multi-service Docker Compose application             |
+
+```bash
+aptdata scaffold my_lakehouse --template medallion
+aptdata scaffold my_job --template job-wheel
+aptdata scaffold my_service --template docker-compose-app
+```
+
+---
+
+## Processing Engines
+
+Engine-agnostic transformation wrappers for pandas and PySpark:
+
+```python
+from aptdata.plugins.transform import PandasTransformer
+
+def clean(df):
+    return df.dropna().drop_duplicates()
+
+transformer = PandasTransformer("clean", clean)
+result = transformer.transform(my_dataset)
+```
+
+See [Transform Engines docs](docs/transform-engines.md) for PySpark usage.
+
+---
+
+## Data Quality & Contracts
+
+```python
+from aptdata.plugins.quality import (
+    EnforcementMode, ExpectColumnToNotBeNull,
+    QualityValidator, SchemaContract,
+)
+
+validator = QualityValidator(
+    expectations=[ExpectColumnToNotBeNull("id")],
+    enforcement=EnforcementMode.ABORT,
+)
+clean_data = validator.validate(raw_df)
+```
+
+See [Quality docs](docs/quality.md) for all built-in expectations.
+
+---
+
+## Data Governance
+
+```python
+from aptdata.plugins.governance import (
+    BusinessRule, DatasetCatalog, DatasetCatalogEntry, LineageStore,
+)
+from aptdata.core.lineage import LineageGraph, LineageNode, LineageEventType
+
+# Lineage tracking
+graph = LineageGraph(run_id="run-1", workflow_name="etl")
+graph.add_node(LineageNode(dataset_uri="s3://raw/data", event_type=LineageEventType.READ))
+
+store = LineageStore()
+store.save(graph)
+```
+
+See [Governance docs](docs/governance.md) for the full API.
+
+---
+
+## Release process
+
+Releases are automated via the [Release workflow](.github/workflows/release.yml).
+After a PR is merged into `main`, the CI reads its labels and bumps the version
+accordingly.
+
+| Label | Effect |
+|---|---|
+| `release:patch` | `0.0.1 → 0.0.2` |
+| `release:minor` | `0.0.1 → 0.1.0` |
+| `release:major` | `0.0.1 → 1.0.0` |
+| `release:skip` | no release (explicit opt-out) |
+| *(no label)* | no release (silent skip) |
+
+The workflow will:
+1. Detect the merged PR and its labels.
+2. Run `bump-my-version bump <part>` to update `pyproject.toml` and
+   `aptdata/__init__.py`.
+3. Create a `chore(release): bump version to X.Y.Z` commit and a `vX.Y.Z` tag.
+4. Push the commit and tag to `main`.
+5. The tag push automatically triggers the **Publish to PyPI** workflow.
+
+> **Branch protection note:** GitHub Actions must have *read and write
+> permissions* (Settings → Actions → General → Workflow permissions) and, if
+> branch protection is enabled on `main`, the rule must allow GitHub Actions
+> to bypass it.
+
+---
+
+## Development
+
+```bash
+make install   # install all dependencies
+make test      # run the test suite
+make lint      # lint with ruff
+make docs      # build the documentation
+```
+
+---
+
+## Documentation
+
+Full documentation is available in the [`docs/`](docs/) directory and can be
+served locally with:
+
+```bash
+mkdocs serve
+```
+
+---
+
+## License
+
+[MIT](LICENSE)
+

aptdata-0.0.2/README.md

@@ -0,0 +1,285 @@
+# aptdata
+
+> **v0.0.2** · A declarative, extensible framework for building smart data pipelines in Python.
+
+[![Python](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/)
+[![License](https://img.shields.io/badge/license-MIT-green)](LICENSE)
+[![Version](https://img.shields.io/badge/version-0.0.2-orange)](CHANGELOG.md)
+
+---
+
+## Overview
+
+**aptdata** is built around three universal abstractions — **System**,
+**Flow**, and **Component** — that cover every data-processing paradigm in a
+single, coherent model:
+
+```mermaid
+flowchart TD
+    I["IComponent / IFlow / ISystem\n@dataclass + ABC — pure interfaces"]
+    B["BaseComponent / BaseFlow / BaseSystem\n@pydantic_dataclass — validated fields"]
+    Y["Your concrete implementations"]
+
+    I --> B --> Y
+```
+
+Datasets remain the fundamental data-exchange contract (`IDataset` /
+`BaseDataset`).  Every outcome from the CLI is emitted as a machine-readable
+JSON line, making aptdata a natural fit for AI orchestrators, CI/CD
+pipelines and scripted workflows.
+
+---
+
+## Requirements
+
+- Python ≥ 3.10
+- [Poetry](https://python-poetry.org/) (for development)
+
+---
+
+## Installation
+
+### From PyPI
+
+```bash
+pip install aptdata
+```
+
+### Optional extras
+
+```bash
+pip install aptdata[pandas]   # pandas support
+pip install aptdata[spark]    # PySpark support
+pip install aptdata[plugins]  # REST, PostgreSQL, Parquet I/O
+pip install aptdata[all]      # everything
+```
+
+### From source (development)
+
+```bash
+git clone https://github.com/strondata/smart-data.git
+cd aptdata
+poetry install
+```
+
+---
+
+## Quick start
+
+```python
+from pydantic.dataclasses import dataclass as pydantic_dataclass
+from aptdata.core import (
+    BaseDataset, IDataset,
+    BaseComponent, ComponentMeta, ComponentKind,
+    BaseFlow, IFlow,
+    BaseSystem,
+)
+
+@pydantic_dataclass
+class MemoryDataset(BaseDataset):
+    def __post_init__(self): self._data = None
+    def read(self): return self._data
+    def write(self, data): self._data = data
+
+@pydantic_dataclass
+class DoubleComponent(BaseComponent):
+    def validate_inputs(self, inputs: list[IDataset]) -> bool:
+        return len(inputs) == 1
+    def execute(self, inputs: list[IDataset]) -> list[IDataset]:
+        out = MemoryDataset(uri="memory://out")
+        out.write([x * 2 for x in inputs[0].read()])
+        return [out]
+
+@pydantic_dataclass
+class ETLFlow(BaseFlow):
+    def __post_init__(self):
+        self._nodes = {}
+        self._edges = []
+        self._compiled = False
+    def add_component(self, c): self._nodes[c.component_id] = c
+    def connect(self, src, tgt, condition=None): ...
+    def compile(self): self._compiled = True
+    def run(self, inputs): return inputs  # wire your logic here
+
+@pydantic_dataclass
+class MySystem(BaseSystem):
+    def __post_init__(self): self._flows: list[IFlow] = []
+    def register_flow(self, flow): self._flows.append(flow)
+    def run(self):
+        for flow in self._flows:
+            flow.run([])
+
+# Register and run via CLI
+from aptdata.plugins import registry
+registry.register("my_system", MySystem)
+```
+
+```bash
+aptdata run my_system
+# {"event": "pipeline.started", "pipeline": "my_system", "env": "dev", "dry_run": false, "trace_id": null}
+# {"event": "pipeline.completed", "pipeline": "my_system", "env": "dev", "dry_run": false, "elapsed_seconds": 0.001, "trace_id": null}
+```
+
+---
+
+## CLI reference
+
+```
+aptdata run SYSTEM_NAME [--env ENV] [--dry-run]
+aptdata monitor [--refresh SECONDS]
+aptdata scaffold PROJECT_NAME [--template TEMPLATE] [--output PATH]
+aptdata schema export --output schema.json
+aptdata system list [--json]
+aptdata system info NAME [--json]
+aptdata system validate NAME
+aptdata plugin list [--json]
+aptdata plugin inspect NAME [--json]
+aptdata plugin preview READER [--limit N]
+aptdata plugin load MODULE_PATH
+aptdata config validate PATH
+aptdata config init [--output PATH]
+aptdata config show PATH
+aptdata config run PATH [--env ENV]
+aptdata telemetry status [--json]
+aptdata telemetry export [--format json]
+aptdata mesh list [--dir DIR] [--json]
+aptdata mesh run COMPONENT [--dir DIR] [--dry-run] [--json]
+aptdata mesh build COMPONENT [--dir DIR] [--json]
+aptdata mcp-start [--transport TRANSPORT]
+aptdata interactive
+```
+
+Every static command supports `--json` for machine-readable JSON line output
+(backward compatible). Without `--json`, commands render Rich tables, panels,
+and syntax-highlighted output.
+
+### Scaffold templates
+
+| Template              | Description                                         |
+|-----------------------|-----------------------------------------------------|
+| `hello-world`         | Minimal pandas pipeline (default)                   |
+| `medallion`           | Bronze → Silver → Gold data lakehouse               |
+| `rag-ingestion`       | RAG pipeline: extract → chunk → embed → load        |
+| `data-quality-test`   | Schema contract + expectation suite                  |
+| `job-wheel`           | Python wheel executor for portable job packaging     |
+| `docker-compose-app`  | Multi-service Docker Compose application             |
+
+```bash
+aptdata scaffold my_lakehouse --template medallion
+aptdata scaffold my_job --template job-wheel
+aptdata scaffold my_service --template docker-compose-app
+```
+
+---
+
+## Processing Engines
+
+Engine-agnostic transformation wrappers for pandas and PySpark:
+
+```python
+from aptdata.plugins.transform import PandasTransformer
+
+def clean(df):
+    return df.dropna().drop_duplicates()
+
+transformer = PandasTransformer("clean", clean)
+result = transformer.transform(my_dataset)
+```
+
+See [Transform Engines docs](docs/transform-engines.md) for PySpark usage.
+
+---
+
+## Data Quality & Contracts
+
+```python
+from aptdata.plugins.quality import (
+    EnforcementMode, ExpectColumnToNotBeNull,
+    QualityValidator, SchemaContract,
+)
+
+validator = QualityValidator(
+    expectations=[ExpectColumnToNotBeNull("id")],
+    enforcement=EnforcementMode.ABORT,
+)
+clean_data = validator.validate(raw_df)
+```
+
+See [Quality docs](docs/quality.md) for all built-in expectations.
+
+---
+
+## Data Governance
+
+```python
+from aptdata.plugins.governance import (
+    BusinessRule, DatasetCatalog, DatasetCatalogEntry, LineageStore,
+)
+from aptdata.core.lineage import LineageGraph, LineageNode, LineageEventType
+
+# Lineage tracking
+graph = LineageGraph(run_id="run-1", workflow_name="etl")
+graph.add_node(LineageNode(dataset_uri="s3://raw/data", event_type=LineageEventType.READ))
+
+store = LineageStore()
+store.save(graph)
+```
+
+See [Governance docs](docs/governance.md) for the full API.
+
+---
+
+## Release process
+
+Releases are automated via the [Release workflow](.github/workflows/release.yml).
+After a PR is merged into `main`, the CI reads its labels and bumps the version
+accordingly.
+
+| Label | Effect |
+|---|---|
+| `release:patch` | `0.0.1 → 0.0.2` |
+| `release:minor` | `0.0.1 → 0.1.0` |
+| `release:major` | `0.0.1 → 1.0.0` |
+| `release:skip` | no release (explicit opt-out) |
+| *(no label)* | no release (silent skip) |
+
+The workflow will:
+1. Detect the merged PR and its labels.
+2. Run `bump-my-version bump <part>` to update `pyproject.toml` and
+   `aptdata/__init__.py`.
+3. Create a `chore(release): bump version to X.Y.Z` commit and a `vX.Y.Z` tag.
+4. Push the commit and tag to `main`.
+5. The tag push automatically triggers the **Publish to PyPI** workflow.
+
+> **Branch protection note:** GitHub Actions must have *read and write
+> permissions* (Settings → Actions → General → Workflow permissions) and, if
+> branch protection is enabled on `main`, the rule must allow GitHub Actions
+> to bypass it.
+
+---
+
+## Development
+
+```bash
+make install   # install all dependencies
+make test      # run the test suite
+make lint      # lint with ruff
+make docs      # build the documentation
+```
+
+---
+
+## Documentation
+
+Full documentation is available in the [`docs/`](docs/) directory and can be
+served locally with:
+
+```bash
+mkdocs serve
+```
+
+---
+
+## License
+
+[MIT](LICENSE)

aptdata-0.0.2/aptdata/__init__.py

@@ -0,0 +1,3 @@
+"""aptdata: A framework for smart data pipelines."""
+
+__version__ = "0.0.2"

aptdata-0.0.2/aptdata/cli/__init__.py

@@ -0,0 +1,5 @@
+"""Static CLI entry-point for aptdata (Typer-based)."""
+
+from aptdata.cli.app import app
+
+__all__ = ["app"]