serverless-data-mesh 0.2.0__tar.gz

serverless_data_mesh-0.2.0/.gitignore

@@ -0,0 +1,61 @@
+# Byte-compiled / cache
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+.Python
+
+# Virtual environments
+.venv/
+venv/
+env/
+
+# Packaging
+dist/
+build/
+*.egg-info/
+.eggs/
+*.whl
+
+# Test / coverage
+.pytest_cache/
+.coverage
+htmlcov/
+.mypy_cache/
+.ruff_cache/
+
+# IDE / editor
+.idea/
+.vscode/
+.cursor/
+*.swp
+
+# AWS SAM
+.aws-sam/
+samconfig.toml
+
+# Terraform
+infrastructure/terraform/build/*
+!infrastructure/terraform/build/.gitkeep
+.terraform/
+*.tfplan
+*.tfstate
+*.tfstate.*
+.terraform.lock.hcl
+terraform.tfvars
+**/terraform.tfvars
+!**/terraform.tfvars.example
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Secrets / keys
+.env
+.env.*
+*.pem
+VRP_SIGNING_KEY*
+
+# Local scratch / agent artifacts
+tests/_scratch_*
+agent-transcripts/

serverless_data_mesh-0.2.0/CHANGELOG.md

@@ -0,0 +1,81 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+## [0.2.0] - 2026-06-15
+
+### Added
+
+- **Auto VRP reprocessing** (`orchestration/reprocess.py`): detect drops, repair sink, re-proof, commit or escalate
+- **Consumer SLA enforcement** (`governance/consumer_sla.py`): freshness, completeness, required columns; LF grant hook
+- **Live CloudWatch trust dashboard** (Terraform `monitoring/dashboard.tf` + `publish_vrp_metric`)
+- **Grafana template** (`infrastructure/grafana/mesh-trust-dashboard.json`)
+- **Lake Formation governance module** (`terraform/modules/governance/`)
+- CLI: `serverless-data-mesh canary`, `reprocess-demo`, `dashboard --cloudwatch`
+- Scaffold: `consumer_sla.yaml`, `step_function.asl.json`, richer handler imports
+- Docs: `docs/mesh-trust-dashboard.md`
+- Tests: reprocess, consumer SLA, canary, auto-repair demo
+
+### Changed
+
+- `IceGuardDurableCoordinator` supports optional `sink_reader` + `enable_auto_repair`
+- Trust dashboard supports proofs scan, CloudWatch live mode, and demo fallback
+
+## [0.1.3] - 2026-06-15
+
+### Added
+
+- **Pure-Python fallback verifier** when veridata-recon wheels unavailable (Windows/Mac demo works)
+- `serverless-data-mesh init` domain scaffold CLI
+- `serverless-data-mesh dashboard` mesh trust dashboard HTML
+- `benchmarks/run_cost_estimate.py` with published cost estimates (100K/1M/10M)
+- `ConsumerSLAContract` type and `run_canary_comparison` for progressive delivery
+- Tests: fallback verifier, scaffold, dashboard
+
+### Changed
+
+- Local demo, gate demo, multi-domain, and benchmark use `create_proof_generator()` auto-backend
+- `benchmarks/results/2026-06-baseline.json` populated (pricing model estimated)
+
+## [0.1.2] - 2026-06-15
+
+### Added
+
+- **Local demo**: `serverless-data-mesh demo` runs PVDM in <60s without AWS (`LocalPVDMRuntime`)
+- `examples/tutorials/verification_gate_demo.py`: corrupt data blocked before consumers
+- `examples/multi-domain-orders-payments/`: deferred mesh leader commit + `test_atomicity.py`
+- `benchmarks/`: cost comparison methodology, workloads (100k/1M/10M), results placeholder
+- `serverless_data_mesh.lineage`: OpenLineage `emit_openlineage_event` on commit
+- Tests: `test_local_runtime.py`, `test_openlineage.py`
+
+## [0.1.1] - 2026-06-15
+
+### Added
+
+- **Vaquar Pattern** (`docs/vaquar-pattern.md`): proof-gated serverless lakehouse publication (PVDM)
+- Blog-style [why-serverless-data-mesh.md](docs/why-serverless-data-mesh.md) with connectivity diagrams
+- New diagram assets: `why-sdm-hero-connectivity.png`, `why-sdm-four-phase-connectivity.png`, `why-sdm-trust-gap.png`
+
+### Fixed
+
+- Hatch `VERSION` parsing: `pattern` for plain semver file (CI build green)
+- `__version__` reads from package metadata when installed; dev fallback uses `parents[2]`
+- Multi-account Terraform: Steward creates only checkpoint/proof buckets; Publisher creates only lakehouse
+- Storage module: optional per-bucket creation via nullable bucket names
+
+### Changed
+
+- Pattern catalog: Vaquar Pattern as flagship pattern #0 in [data-mesh-patterns.md](docs/data-mesh-patterns.md)
+
+## [0.1.0] - 2026-06-14
+
+### Added
+
+- Initial release: IceGuard + Durable Execution + veridata-recon + Glue REST
+- `IceGuardDurableCoordinator` for cross-domain transaction boundaries
+- `GlueRestCatalogAdapter` for SigV4 Iceberg metadata commits
+- `VRPProofGenerator` and `validate_then_commit` hooks
+- Domain writer Lambda example with SAM template
+- `VERSION` file, pre-commit, Dependabot, SECURITY.md, consumer safety benchmark
+- `SparkRulesConnector` (`[rules]` extra), PyPI publish workflow
+- CI workflow, unit tests, and documentation

serverless_data_mesh-0.2.0/LICENSE

@@ -0,0 +1,17 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   Copyright 2026 Serverless Data Mesh Contributors
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

serverless_data_mesh-0.2.0/PKG-INFO

@@ -0,0 +1,143 @@
+Metadata-Version: 2.4
+Name: serverless-data-mesh
+Version: 0.2.0
+Summary: Federated data mesh framework for proof-gated, exactly-once Iceberg data product publication on AWS Lambda.
+Project-URL: Homepage, https://github.com/vaquarkhan/aws-serverless-datamesh-framework
+Project-URL: Documentation, https://github.com/vaquarkhan/aws-serverless-datamesh-framework#readme
+Project-URL: Repository, https://github.com/vaquarkhan/aws-serverless-datamesh-framework
+Project-URL: Changelog, https://github.com/vaquarkhan/aws-serverless-datamesh-framework/blob/main/CHANGELOG.md
+Project-URL: PyPI, https://pypi.org/project/serverless-data-mesh/
+Author: Serverless Data Mesh Contributors
+License-Expression: Apache-2.0
+License-File: LICENSE
+Keywords: aws,data-as-a-product,data-mesh,domain-oriented,durable-execution,federated-data-mesh,federated-governance,iceberg,iceguard,lakehouse,lambda,pyiceberg,rule-engine,serverless,sparkrules,vaquar-pattern,veridata-recon
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Database
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Classifier: Topic :: System :: Distributed Computing
+Classifier: Typing :: Typed
+Requires-Python: >=3.12
+Requires-Dist: aws-durable-execution-sdk-python>=1.0.0
+Requires-Dist: boto3>=1.34
+Requires-Dist: iceguard>=1.0.0
+Requires-Dist: pyiceberg[glue,rest-sigv4]>=0.7.0
+Requires-Dist: veridata-recon>=0.1.0
+Provides-Extra: all
+Requires-Dist: pyspark>=3.5.0; extra == 'all'
+Requires-Dist: sparkrules>=1.2.0; extra == 'all'
+Provides-Extra: dev
+Requires-Dist: build>=1.2; extra == 'dev'
+Requires-Dist: mypy>=1.8; extra == 'dev'
+Requires-Dist: pre-commit>=4; extra == 'dev'
+Requires-Dist: pytest-cov>=5; extra == 'dev'
+Requires-Dist: pytest>=8; extra == 'dev'
+Requires-Dist: ruff>=0.4; extra == 'dev'
+Requires-Dist: twine>=5; extra == 'dev'
+Provides-Extra: publish
+Requires-Dist: build>=1.2; extra == 'publish'
+Requires-Dist: twine>=5; extra == 'publish'
+Provides-Extra: rules
+Requires-Dist: sparkrules>=1.2.0; extra == 'rules'
+Provides-Extra: spark
+Requires-Dist: pyspark>=3.5.0; extra == 'spark'
+Requires-Dist: sparkrules>=1.2.0; extra == 'spark'
+Description-Content-Type: text/markdown
+
+# Serverless Data Mesh
+
+**Federated data mesh lakehouse publication on AWS Lambda - with cryptographic proof, not just green job logs.**
+
+An open Python framework for **domain-oriented**, **federated data mesh** teams on AWS. Implements **data as a product**, **self-serve write infrastructure**, and **federated computational governance** at the Iceberg lakehouse layer.
+
+- **Producer** domains publish governed **data products**
+- **Steward** notaries enforce mesh contracts with VRP proofs
+- **Publisher** zones expose consumer-ready **Iceberg data products**
+
+Introduces the **[Vaquar Pattern](https://github.com/vaquarkhan/aws-serverless-datamesh-framework/blob/main/docs/vaquar-pattern.md)** (PVDM): Physical → Verify → Durable → Metadata. Invariant: `commit_metadata ⟹ VRP = PASS`.
+
+## Install
+
+```bash
+pip install serverless-data-mesh
+serverless-data-mesh demo    # <60s local PVDM demo, no AWS
+pip install "serverless-data-mesh[rules]"   # + SparkRules on Lambda
+pip install "serverless-data-mesh[spark]"   # + PySpark + SparkRules
+pip install "serverless-data-mesh[all]"     # rules + spark
+pip install "serverless-data-mesh[dev]"     # pytest, ruff, pre-commit
+```
+
+**Works on **Windows, Mac, and Linux**. Uses veridata-recon cryptographic VRP when wheels are available; **pure-Python fallback** otherwise (same PASS/FAIL gate).**
+
+## What it solves
+
+| Problem | Answer |
+|---------|--------|
+| Silent data loss on backfill | VRP `FAIL` blocks Iceberg snapshot |
+| "Job succeeded" is not proof | Cryptographic multiset proof per chunk |
+| Lambda 15-minute limit | Durable Execution + Step Functions (90+ min) |
+| Retry duplicates data | IceGuard rollback + `workload_id` checkpoints |
+| Central ETL bottleneck | Per-domain Lambda writer + transaction boundary |
+| Federated blast radius | Producer · Steward · Publisher accounts |
+
+## Building blocks
+
+| Package | Role |
+|---------|------|
+| [iceguard](https://pypi.org/project/iceguard/) | Physical SafeWriter, timeout rollback, S3 resume |
+| [veridata-recon](https://pypi.org/project/veridata-recon/) | VRP proof generation and validation |
+| [aws-durable-execution-sdk-python](https://pypi.org/project/aws-durable-execution-sdk-python/) | Cross-invocation step replay |
+| [pyiceberg](https://pypi.org/project/pyiceberg/) | Glue Iceberg REST metadata via `GlueCatalogConnector` |
+| [sparkrules](https://pypi.org/project/sparkrules/) | Optional DRL business rules (`[rules]` extra) |
+
+## Quick example
+
+```python
+from serverless_data_mesh import (
+    IceGuardDurableCoordinator,
+    DomainTransactionBoundary,
+    VRPProofGenerator,
+)
+
+boundary = DomainTransactionBoundary(
+    domain_id="orders-domain",
+    source_namespace="raw_orders",
+    target_table="orders_curated",
+    partition_spec={"dt": "2026-06-14"},
+)
+
+coordinator = IceGuardDurableCoordinator(
+    durable_context=durable_ctx,
+    lambda_context=lambda_ctx,
+    proof_generator=VRPProofGenerator(),
+    catalog_adapter=glue_adapter,
+)
+outcome = coordinator.run_workload(workload)
+```
+
+## Optional extras
+
+| Extra | Adds |
+|-------|------|
+| `rules` | SparkRules DRL on Lambda |
+| `spark` | PySpark + SparkRules |
+| `all` | `rules` + `spark` |
+| `dev` | pytest, ruff, mypy, pre-commit |
+| `publish` | build, twine |
+
+## Documentation
+
+- [GitHub README](https://github.com/vaquarkhan/aws-serverless-datamesh-framework#readme)
+- [Vaquar Pattern](https://github.com/vaquarkhan/aws-serverless-datamesh-framework/blob/main/docs/vaquar-pattern.md)
+- [Getting started](https://github.com/vaquarkhan/aws-serverless-datamesh-framework/blob/main/docs/getting-started.md)
+- [PyPI guide](https://github.com/vaquarkhan/aws-serverless-datamesh-framework/blob/main/docs/pypi.md)
+- [Full documentation](https://github.com/vaquarkhan/aws-serverless-datamesh-framework/tree/main/docs)
+
+## License
+
+Apache-2.0

serverless_data_mesh-0.2.0/PYPI.md

@@ -0,0 +1,92 @@
+# Serverless Data Mesh
+
+**Federated data mesh lakehouse publication on AWS Lambda - with cryptographic proof, not just green job logs.**
+
+An open Python framework for **domain-oriented**, **federated data mesh** teams on AWS. Implements **data as a product**, **self-serve write infrastructure**, and **federated computational governance** at the Iceberg lakehouse layer.
+
+- **Producer** domains publish governed **data products**
+- **Steward** notaries enforce mesh contracts with VRP proofs
+- **Publisher** zones expose consumer-ready **Iceberg data products**
+
+Introduces the **[Vaquar Pattern](https://github.com/vaquarkhan/aws-serverless-datamesh-framework/blob/main/docs/vaquar-pattern.md)** (PVDM): Physical → Verify → Durable → Metadata. Invariant: `commit_metadata ⟹ VRP = PASS`.
+
+## Install
+
+```bash
+pip install serverless-data-mesh
+serverless-data-mesh demo    # <60s local PVDM demo, no AWS
+pip install "serverless-data-mesh[rules]"   # + SparkRules on Lambda
+pip install "serverless-data-mesh[spark]"   # + PySpark + SparkRules
+pip install "serverless-data-mesh[all]"     # rules + spark
+pip install "serverless-data-mesh[dev]"     # pytest, ruff, pre-commit
+```
+
+**Works on **Windows, Mac, and Linux**. Uses veridata-recon cryptographic VRP when wheels are available; **pure-Python fallback** otherwise (same PASS/FAIL gate).**
+
+## What it solves
+
+| Problem | Answer |
+|---------|--------|
+| Silent data loss on backfill | VRP `FAIL` blocks Iceberg snapshot |
+| "Job succeeded" is not proof | Cryptographic multiset proof per chunk |
+| Lambda 15-minute limit | Durable Execution + Step Functions (90+ min) |
+| Retry duplicates data | IceGuard rollback + `workload_id` checkpoints |
+| Central ETL bottleneck | Per-domain Lambda writer + transaction boundary |
+| Federated blast radius | Producer · Steward · Publisher accounts |
+
+## Building blocks
+
+| Package | Role |
+|---------|------|
+| [iceguard](https://pypi.org/project/iceguard/) | Physical SafeWriter, timeout rollback, S3 resume |
+| [veridata-recon](https://pypi.org/project/veridata-recon/) | VRP proof generation and validation |
+| [aws-durable-execution-sdk-python](https://pypi.org/project/aws-durable-execution-sdk-python/) | Cross-invocation step replay |
+| [pyiceberg](https://pypi.org/project/pyiceberg/) | Glue Iceberg REST metadata via `GlueCatalogConnector` |
+| [sparkrules](https://pypi.org/project/sparkrules/) | Optional DRL business rules (`[rules]` extra) |
+
+## Quick example
+
+```python
+from serverless_data_mesh import (
+    IceGuardDurableCoordinator,
+    DomainTransactionBoundary,
+    VRPProofGenerator,
+)
+
+boundary = DomainTransactionBoundary(
+    domain_id="orders-domain",
+    source_namespace="raw_orders",
+    target_table="orders_curated",
+    partition_spec={"dt": "2026-06-14"},
+)
+
+coordinator = IceGuardDurableCoordinator(
+    durable_context=durable_ctx,
+    lambda_context=lambda_ctx,
+    proof_generator=VRPProofGenerator(),
+    catalog_adapter=glue_adapter,
+)
+outcome = coordinator.run_workload(workload)
+```
+
+## Optional extras
+
+| Extra | Adds |
+|-------|------|
+| `rules` | SparkRules DRL on Lambda |
+| `spark` | PySpark + SparkRules |
+| `all` | `rules` + `spark` |
+| `dev` | pytest, ruff, mypy, pre-commit |
+| `publish` | build, twine |
+
+## Documentation
+
+- [GitHub README](https://github.com/vaquarkhan/aws-serverless-datamesh-framework#readme)
+- [Vaquar Pattern](https://github.com/vaquarkhan/aws-serverless-datamesh-framework/blob/main/docs/vaquar-pattern.md)
+- [Getting started](https://github.com/vaquarkhan/aws-serverless-datamesh-framework/blob/main/docs/getting-started.md)
+- [PyPI guide](https://github.com/vaquarkhan/aws-serverless-datamesh-framework/blob/main/docs/pypi.md)
+- [Full documentation](https://github.com/vaquarkhan/aws-serverless-datamesh-framework/tree/main/docs)
+
+## License
+
+Apache-2.0