aptdata 0.0.2__py3-none-any.whl

aptdata-0.0.2.dist-info/METADATA

@@ -0,0 +1,330 @@
+Metadata-Version: 2.4
+Name: aptdata
+Version: 0.0.2
+Summary: A declarative, extensible framework for building smart data pipelines in Python
+License: MIT
+License-File: LICENSE
+Keywords: data-pipeline,framework,etl,pydantic,data-engineering
+Author: strondata
+Requires-Python: >=3.10,<4.0
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Software Development :: Libraries :: Application Frameworks
+Provides-Extra: all
+Provides-Extra: pandas
+Provides-Extra: plugins
+Provides-Extra: spark
+Requires-Dist: httpx (>=0.27,<0.28) ; extra == "plugins" or extra == "all"
+Requires-Dist: mcp (>=1.26.0,<2.0.0)
+Requires-Dist: opentelemetry-api (>=1.40.0,<2.0.0)
+Requires-Dist: opentelemetry-sdk (>=1.40.0,<2.0.0)
+Requires-Dist: pandas (>=2.2,<3.0) ; extra == "pandas" or extra == "all"
+Requires-Dist: psycopg2-binary (>=2.9,<3.0) ; extra == "plugins" or extra == "all"
+Requires-Dist: pyarrow (>=15.0,<16.0) ; extra == "plugins" or extra == "all"
+Requires-Dist: pydantic (>=2.0,<3.0)
+Requires-Dist: pyspark (>=3.5,<4.0) ; extra == "spark" or extra == "all"
+Requires-Dist: python-dotenv (>=1.0,<2.0)
+Requires-Dist: pyyaml (>=6.0,<7.0)
+Requires-Dist: questionary (>=2.0)
+Requires-Dist: rich (>=13.0,<14.0)
+Requires-Dist: sqlalchemy (>=2.0,<3.0) ; extra == "plugins" or extra == "all"
+Requires-Dist: textual (>=0.60,<0.61)
+Requires-Dist: typer[all] (>=0.15,<0.16)
+Project-URL: Documentation, https://strondata.github.io/smart-data
+Project-URL: Homepage, https://strondata.github.io/smart-data
+Project-URL: Repository, https://github.com/strondata/smart-data
+Description-Content-Type: text/markdown
+
+# aptdata
+
+> **v0.0.2** · A declarative, extensible framework for building smart data pipelines in Python.
+
+[![Python](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/)
+[![License](https://img.shields.io/badge/license-MIT-green)](LICENSE)
+[![Version](https://img.shields.io/badge/version-0.0.2-orange)](CHANGELOG.md)
+
+---
+
+## Overview
+
+**aptdata** is built around three universal abstractions — **System**,
+**Flow**, and **Component** — that cover every data-processing paradigm in a
+single, coherent model:
+
+```mermaid
+flowchart TD
+    I["IComponent / IFlow / ISystem\n@dataclass + ABC — pure interfaces"]
+    B["BaseComponent / BaseFlow / BaseSystem\n@pydantic_dataclass — validated fields"]
+    Y["Your concrete implementations"]
+
+    I --> B --> Y
+```
+
+Datasets remain the fundamental data-exchange contract (`IDataset` /
+`BaseDataset`).  Every outcome from the CLI is emitted as a machine-readable
+JSON line, making aptdata a natural fit for AI orchestrators, CI/CD
+pipelines and scripted workflows.
+
+---
+
+## Requirements
+
+- Python ≥ 3.10
+- [Poetry](https://python-poetry.org/) (for development)
+
+---
+
+## Installation
+
+### From PyPI
+
+```bash
+pip install aptdata
+```
+
+### Optional extras
+
+```bash
+pip install aptdata[pandas]   # pandas support
+pip install aptdata[spark]    # PySpark support
+pip install aptdata[plugins]  # REST, PostgreSQL, Parquet I/O
+pip install aptdata[all]      # everything
+```
+
+### From source (development)
+
+```bash
+git clone https://github.com/strondata/smart-data.git
+cd aptdata
+poetry install
+```
+
+---
+
+## Quick start
+
+```python
+from pydantic.dataclasses import dataclass as pydantic_dataclass
+from aptdata.core import (
+    BaseDataset, IDataset,
+    BaseComponent, ComponentMeta, ComponentKind,
+    BaseFlow, IFlow,
+    BaseSystem,
+)
+
+@pydantic_dataclass
+class MemoryDataset(BaseDataset):
+    def __post_init__(self): self._data = None
+    def read(self): return self._data
+    def write(self, data): self._data = data
+
+@pydantic_dataclass
+class DoubleComponent(BaseComponent):
+    def validate_inputs(self, inputs: list[IDataset]) -> bool:
+        return len(inputs) == 1
+    def execute(self, inputs: list[IDataset]) -> list[IDataset]:
+        out = MemoryDataset(uri="memory://out")
+        out.write([x * 2 for x in inputs[0].read()])
+        return [out]
+
+@pydantic_dataclass
+class ETLFlow(BaseFlow):
+    def __post_init__(self):
+        self._nodes = {}
+        self._edges = []
+        self._compiled = False
+    def add_component(self, c): self._nodes[c.component_id] = c
+    def connect(self, src, tgt, condition=None): ...
+    def compile(self): self._compiled = True
+    def run(self, inputs): return inputs  # wire your logic here
+
+@pydantic_dataclass
+class MySystem(BaseSystem):
+    def __post_init__(self): self._flows: list[IFlow] = []
+    def register_flow(self, flow): self._flows.append(flow)
+    def run(self):
+        for flow in self._flows:
+            flow.run([])
+
+# Register and run via CLI
+from aptdata.plugins import registry
+registry.register("my_system", MySystem)
+```
+
+```bash
+aptdata run my_system
+# {"event": "pipeline.started", "pipeline": "my_system", "env": "dev", "dry_run": false, "trace_id": null}
+# {"event": "pipeline.completed", "pipeline": "my_system", "env": "dev", "dry_run": false, "elapsed_seconds": 0.001, "trace_id": null}
+```
+
+---
+
+## CLI reference
+
+```
+aptdata run SYSTEM_NAME [--env ENV] [--dry-run]
+aptdata monitor [--refresh SECONDS]
+aptdata scaffold PROJECT_NAME [--template TEMPLATE] [--output PATH]
+aptdata schema export --output schema.json
+aptdata system list [--json]
+aptdata system info NAME [--json]
+aptdata system validate NAME
+aptdata plugin list [--json]
+aptdata plugin inspect NAME [--json]
+aptdata plugin preview READER [--limit N]
+aptdata plugin load MODULE_PATH
+aptdata config validate PATH
+aptdata config init [--output PATH]
+aptdata config show PATH
+aptdata config run PATH [--env ENV]
+aptdata telemetry status [--json]
+aptdata telemetry export [--format json]
+aptdata mesh list [--dir DIR] [--json]
+aptdata mesh run COMPONENT [--dir DIR] [--dry-run] [--json]
+aptdata mesh build COMPONENT [--dir DIR] [--json]
+aptdata mcp-start [--transport TRANSPORT]
+aptdata interactive
+```
+
+Every static command supports `--json` for machine-readable JSON line output
+(backward compatible). Without `--json`, commands render Rich tables, panels,
+and syntax-highlighted output.
+
+### Scaffold templates
+
+| Template              | Description                                         |
+|-----------------------|-----------------------------------------------------|
+| `hello-world`         | Minimal pandas pipeline (default)                   |
+| `medallion`           | Bronze → Silver → Gold data lakehouse               |
+| `rag-ingestion`       | RAG pipeline: extract → chunk → embed → load        |
+| `data-quality-test`   | Schema contract + expectation suite                  |
+| `job-wheel`           | Python wheel executor for portable job packaging     |
+| `docker-compose-app`  | Multi-service Docker Compose application             |
+
+```bash
+aptdata scaffold my_lakehouse --template medallion
+aptdata scaffold my_job --template job-wheel
+aptdata scaffold my_service --template docker-compose-app
+```
+
+---
+
+## Processing Engines
+
+Engine-agnostic transformation wrappers for pandas and PySpark:
+
+```python
+from aptdata.plugins.transform import PandasTransformer
+
+def clean(df):
+    return df.dropna().drop_duplicates()
+
+transformer = PandasTransformer("clean", clean)
+result = transformer.transform(my_dataset)
+```
+
+See [Transform Engines docs](docs/transform-engines.md) for PySpark usage.
+
+---
+
+## Data Quality & Contracts
+
+```python
+from aptdata.plugins.quality import (
+    EnforcementMode, ExpectColumnToNotBeNull,
+    QualityValidator, SchemaContract,
+)
+
+validator = QualityValidator(
+    expectations=[ExpectColumnToNotBeNull("id")],
+    enforcement=EnforcementMode.ABORT,
+)
+clean_data = validator.validate(raw_df)
+```
+
+See [Quality docs](docs/quality.md) for all built-in expectations.
+
+---
+
+## Data Governance
+
+```python
+from aptdata.plugins.governance import (
+    BusinessRule, DatasetCatalog, DatasetCatalogEntry, LineageStore,
+)
+from aptdata.core.lineage import LineageGraph, LineageNode, LineageEventType
+
+# Lineage tracking
+graph = LineageGraph(run_id="run-1", workflow_name="etl")
+graph.add_node(LineageNode(dataset_uri="s3://raw/data", event_type=LineageEventType.READ))
+
+store = LineageStore()
+store.save(graph)
+```
+
+See [Governance docs](docs/governance.md) for the full API.
+
+---
+
+## Release process
+
+Releases are automated via the [Release workflow](.github/workflows/release.yml).
+After a PR is merged into `main`, the CI reads its labels and bumps the version
+accordingly.
+
+| Label | Effect |
+|---|---|
+| `release:patch` | `0.0.1 → 0.0.2` |
+| `release:minor` | `0.0.1 → 0.1.0` |
+| `release:major` | `0.0.1 → 1.0.0` |
+| `release:skip` | no release (explicit opt-out) |
+| *(no label)* | no release (silent skip) |
+
+The workflow will:
+1. Detect the merged PR and its labels.
+2. Run `bump-my-version bump <part>` to update `pyproject.toml` and
+   `aptdata/__init__.py`.
+3. Create a `chore(release): bump version to X.Y.Z` commit and a `vX.Y.Z` tag.
+4. Push the commit and tag to `main`.
+5. The tag push automatically triggers the **Publish to PyPI** workflow.
+
+> **Branch protection note:** GitHub Actions must have *read and write
+> permissions* (Settings → Actions → General → Workflow permissions) and, if
+> branch protection is enabled on `main`, the rule must allow GitHub Actions
+> to bypass it.
+
+---
+
+## Development
+
+```bash
+make install   # install all dependencies
+make test      # run the test suite
+make lint      # lint with ruff
+make docs      # build the documentation
+```
+
+---
+
+## Documentation
+
+Full documentation is available in the [`docs/`](docs/) directory and can be
+served locally with:
+
+```bash
+mkdocs serve
+```
+
+---
+
+## License
+
+[MIT](LICENSE)
+

aptdata-0.0.2.dist-info/RECORD

@@ -0,0 +1,65 @@
+aptdata/__init__.py,sha256=43uh9j6aW_deNzgRRifT51dT-vzGaLxtaes1_-DJ4Js,76
+aptdata/cli/__init__.py,sha256=B_VlxEiqwYks_5g0Pi_2-uXNBlBeaNq9tWXxUMco680,108
+aptdata/cli/app.py,sha256=PUK3HgSlUDxQwysmeAyseypUNYiHIB3cOdqgKGC1OkI,6767
+aptdata/cli/commands/__init__.py,sha256=YPF32aBSuAy7GnaIrW2hbAeKfFnxSJVhPN8Qp2sLHog,405
+aptdata/cli/commands/config_cmd.py,sha256=OUA5TBNMMXWOPEGd-BwAURASo-7Vp3KVQy_6VMtZXLE,3930
+aptdata/cli/commands/mesh_cmd.py,sha256=pD8i-cm2rs8jvXGPeL8HObj7jUJ9borAlioISg9jBvQ,13587
+aptdata/cli/commands/plugin_cmd.py,sha256=ZRbGHuzYQa4tvuwxRXi5p5kmORoU-mhoxEZyIDBn2Pk,3589
+aptdata/cli/commands/system_cmd.py,sha256=kXtCB-GwW-5fYm2DK5lgeIELOimqaTWESiXtqoPfn4g,2881
+aptdata/cli/commands/telemetry_cmd.py,sha256=riHUqrB9Yw8PK-8_JRAvhxdDwZM1B4cCHR1ZRWRgxYQ,1647
+aptdata/cli/completions.py,sha256=bbKhf2kvvfWDdd1zIPwRmethyAPazGxxtfwsqVJygAw,1960
+aptdata/cli/interactive.py,sha256=ZsnmRhk09xvFcKHhBOkaqRHdVCgU-pZe0mBVIucfLH0,8852
+aptdata/cli/rendering/__init__.py,sha256=XpQucueN7TGk4fJFYki7EO-BWao-4YvXXh-uRrSVFtQ,721
+aptdata/cli/rendering/console.py,sha256=e8QTzCQe1he-9e-TJtv0j4og33pWPaYBc96YeUsgEfY,3903
+aptdata/cli/rendering/logger.py,sha256=a6TH9_5w8CJ4oWFGcYB_euYQq1JyLM3b1ouyhqDgnMY,657
+aptdata/cli/rendering/panels.py,sha256=hfQQ40rJ5_hczJ2eY9Rjgkq6IJpxhFBiQBGPqHXu5xk,2881
+aptdata/cli/rendering/tables.py,sha256=6nnyIKs0rfqYB-UCd4qR1tbUmgObOW6C8vyMjJn7LAs,2763
+aptdata/cli/scaffold.py,sha256=764TCNhd9DX0kiNWLv_N68imvGfzVjkGo_itvXO9gzk,28406
+aptdata/config/__init__.py,sha256=Eujz6UwTFQgmLq6aGZqtCK3fM7HQq6mNObhqSdDsDnA,379
+aptdata/config/parser.py,sha256=YBwYIOuDaNHiewIZMv0Ot9DSuAJs4TzjkrTm3PYDnB4,4488
+aptdata/config/schema.py,sha256=KyCraUMC_hESbwbX95XaBZRf7ngK4wrIGUjtnWBfBag,761
+aptdata/config/secrets.py,sha256=1xzXNrgIHtPQxB6EVrML1LkA5dG54w2Mv9WYuSw_Z30,2129
+aptdata/core/__init__.py,sha256=tRdbM_p8p4ZYJyejrUSLngFCq87uoAPGh7rY3Pqdldg,876
+aptdata/core/context.py,sha256=gIHAmZ9WSJCB4VSqS3GxAWXeJq8FuStTHkm1q1MVKhg,900
+aptdata/core/dataset.py,sha256=HfXkthp5cqBD0wedyP-GBTbb8uvn4ZJNxLpUaAhbW5Q,1121
+aptdata/core/lineage.py,sha256=FpxJx900kTNnVM_N8HFD7YXerTCurudbvsaZYloNtrE,6819
+aptdata/core/state.py,sha256=VTkAmxu9QiMcK49BPC88iiYED-Jf85kI3KbcK1k1GnI,889
+aptdata/core/system.py,sha256=APGIQZFsmzCDRgOAWmkCKFZ6qyqg58M_sQje4xkBkwI,10553
+aptdata/core/workflow.py,sha256=0ycB0AGyvX1zEX2hT3To461Nwom-wlEyrhQnPK6XBJI,13292
+aptdata/mcp/__init__.py,sha256=rQNyFCA-3I8EQRVux7Ong9A1lNfpAU4Tr8AZGwgluEc,122
+aptdata/mcp/server.py,sha256=SGJsmZb8Z6PNdAWmtCkG3QFioiN_2lNBtFcF0YILDnA,5916
+aptdata/plugins/__init__.py,sha256=IEfsgx_3xSgOE_Td4JhJvLDGoFZoJyMgOZqF_OsEEf0,2173
+aptdata/plugins/ai/__init__.py,sha256=IwiETgjfStDRaYG8Hkuboo3Tv82nRtIAD-QvNagRhOY,228
+aptdata/plugins/ai/chunking.py,sha256=ChkLaLO5uAC-_hg11NohdVBgUPOiRL_57m_oUhk2KJQ,2613
+aptdata/plugins/ai/embeddings.py,sha256=QE1NskhOBBhqx0U3sjyIq7pULfT-N4JZmriKd3pYN1Y,2017
+aptdata/plugins/base.py,sha256=r5JW-2ho01YB8GsBMKHw0PL8kafV1w3YQthPKg1znZg,1695
+aptdata/plugins/dataset.py,sha256=bOOcQt1TfvPSvr9dfEe6mxDwu3VQYejGpQd3uxqyV2U,1695
+aptdata/plugins/governance/__init__.py,sha256=HACQ-3fDFHQbsm7p_FoLXNkMcpfYI31Nqba5zpk0uHQ,780
+aptdata/plugins/governance/catalog.py,sha256=HvLVmNOAtT2z8qZnxWXEe5D5JubKkYchnW0Ee2saII4,3641
+aptdata/plugins/governance/classification.py,sha256=gbBJLLIqk9NGdUXRTlRxMj0Ko-RDkWfebgPTJT2goxw,1336
+aptdata/plugins/governance/lineage_store.py,sha256=0wnTKJwrDiDV7ethNoG_oEXc7iTThzeR7go6agvqwQk,1505
+aptdata/plugins/governance/rules.py,sha256=6N_6AMOdsUGUj3J8RQLNETDWn9zBf_Z1Url_CBBbndU,5082
+aptdata/plugins/local_fs.py,sha256=RXxr2y-EsEyKTAa2TR-uuD-JNfJ4ub39f9UGJkisrMs,7219
+aptdata/plugins/manager.py,sha256=mqfNzknV8Tz0MSbBXDSSIBq_MEx3B8qMxY-r7ShLMwA,5260
+aptdata/plugins/postgres.py,sha256=nxizsIFQrJpL3mY3xW7NfyLaJqbCDQ8XrIWyQaP8Uzk,3593
+aptdata/plugins/quality/__init__.py,sha256=x2mDXbUwYSnt9SUUVPuimx6ARsUyvmLiJiaxiDTAB_A,1014
+aptdata/plugins/quality/contract.py,sha256=HVYElig2qDuoOdEtHFGssiTz8YUt4BeSb4nYALORzDQ,3411
+aptdata/plugins/quality/expectations.py,sha256=flDd1BoeWWkmp51YcjqDD29nGrZfw5ioe2AYQSpdt48,10878
+aptdata/plugins/quality/report.py,sha256=jYuVaWj3i1cnja3FYaR4YRAflpSaSu7VfdxjIDh2uTM,2556
+aptdata/plugins/quality/validator.py,sha256=CPu--KS4NFlckCZdP_lNzTITbPXjHmwT-1TxwV8ReyA,4806
+aptdata/plugins/rest.py,sha256=MKEyEgIioXt-yaVMIBDT14cjRKFrCVNunPm27Pdq1i4,4728
+aptdata/plugins/transform/__init__.py,sha256=NF5fu226i6vNNOFE-R1OgDozbtzLldsqMg4t5i-KSqA,527
+aptdata/plugins/transform/pandas.py,sha256=f-lXt6TVHMypaoM5TlOUSkoBNT0qx16Yteud2qu9lIE,4083
+aptdata/plugins/transform/spark.py,sha256=QN1lRiG8MnrLdsEPXscPF7dx17zmFmYcycwt836ehRg,4268
+aptdata/plugins/vector/__init__.py,sha256=1aWCh7WfJxBPuY-B1Evt23X6vISd2ZKRF8rSSw_u6w8,184
+aptdata/plugins/vector/base.py,sha256=Plpcm25Y8GPsuJpxyPOstmybpxIeOQKz2Ce1qqWlCa4,524
+aptdata/plugins/vector/qdrant.py,sha256=XBozfTwmXcucTfb-DeEvFoEj0B0aJssCricZv9cOpr4,1442
+aptdata/telemetry/__init__.py,sha256=RE_hUkxHiwvCt2gWcWeBzAPUGcvzfromyebvDAJTtrk,189
+aptdata/telemetry/instrumentation.py,sha256=QogZ2uIDYA-Amg_lxgiOeUGjrg-SNIlYp5dJotVrp8U,5753
+aptdata/tui/__init__.py,sha256=U8ScSoGZM8xNVtWBTF__QwvcUItaL2sJiJiReHwiNB4,122
+aptdata/tui/monitor.py,sha256=XRji6dTfXthN06B3Sepabp9f-Tk_5J5e-WIwYHbzuoU,8381
+aptdata-0.0.2.dist-info/METADATA,sha256=oABnmGsQrRt8beFNUb4srfG4lul9oPJd1PpE6PXda8Y,9941
+aptdata-0.0.2.dist-info/WHEEL,sha256=kJCRJT_g0adfAJzTx2GUMmS80rTJIVHRCfG0DQgLq3o,88
+aptdata-0.0.2.dist-info/entry_points.txt,sha256=JpKcSxK1iFyy3-tSYOP0yutq8glTTXdmhD1qXONGlM8,47
+aptdata-0.0.2.dist-info/licenses/LICENSE,sha256=cPddJTJgWfKbOopnk6L2SWEo-PRRAr5TYIJgDPQF9BU,1066
+aptdata-0.0.2.dist-info/RECORD,,

aptdata-0.0.2.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: poetry-core 2.3.1
+Root-Is-Purelib: true
+Tag: py3-none-any

aptdata-0.0.2.dist-info/entry_points.txt

@@ -0,0 +1,3 @@
+[console_scripts]
+aptdata=aptdata.cli.app:app
+

aptdata-0.0.2.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 strondata
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.