serverless-data-mesh 0.2.0__tar.gz → 1.2.0__tar.gz

{serverless_data_mesh-0.2.0 → serverless_data_mesh-1.2.0}/.gitignore

@@ -59,3 +59,13 @@ VRP_SIGNING_KEY*
 # Local scratch / agent artifacts
 tests/_scratch_*
 agent-transcripts/
+
+# Metadata-driven compile output
+examples/retail-mesh/generated/
+examples/medallion-e2e/generated/
+domains/
+infrastructure/terraform/fixtures/generated-mesh/
+
+# Keep VS Code YAML schema mapping
+!.vscode/
+!.vscode/settings.json

serverless_data_mesh-1.2.0/CHANGELOG.md

@@ -0,0 +1,146 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+## [Unreleased]
+
+## [1.2.0] - 2026-07-21
+
+### Added
+
+- **PVDM-A Decision Attestation** + CLI `attest` + coordinator/local wiring
+- **Optional KMS VRP envelopes** (`VRP_KMS_KEY_ID`)
+- **Pre-physical rules gate** (`SDM_RULES_GATE` / SparkRules)
+- **Mesh control center UI** with dashboards, `/walkthrough` GIF demo, one-click demos
+- **README GIF walkthrough** + durable-compute example + Docker image (`ghcr.io`)
+- **100% coverage gate** for attestation / kms_sign / rules.gate / ui.data (`scripts/coverage_gate.py`)
+
+### Fixed
+
+- **Lambda VERSION path**: `__init__.py` tries `serverless_data_mesh/VERSION`, repo root, and metadata; wheel bundles `VERSION` in package
+- **IAM**: `cloudwatch:PutMetricData` for `ServerlessDataMesh*` namespace on domain writer role
+- **Terraform storage**: lifecycle `for_each` uses static bucket keys (not computed `.id` map keys)
+- **Metrics**: `SDM_METRICS_ENABLED` / `SDM_DISABLE_METRICS` opt-out; PutMetricData failures are non-fatal
+- **Lambda packaging**: `SDM_PIPELINE_SRC` overlays compiled `handler.py` at zip root; `handler.lambda_handler` in manifest and Terraform
+- **Observability:** structured `pvdm_outcome` JSON logs; LocalPVDMRuntime S3 proofs when `VRP_PROOF_BUCKET` set
+- **Monitoring:** ops dashboard widgets (invocations, errors, duration, DLQ); DLQ alarm; JSON log metric filters
+- **Scripts:** `aws_observability_smoke.sh`, `aws_dlq_smoke_test.sh`; [observability-production.md](docs/observability-production.md)
+
+## [1.1.0] - 2026-06-14
+
+Enterprise completeness: reference readers, per-layer Lambda fleet, deploy CLI, Backstage, schemas, UI, benchmarks, multi-cloud stubs.
+
+### Added
+
+- **Reference readers**: S3 bronze landing + upstream Parquet (`serverless_data_mesh.readers`)
+- **Per-layer Lambda fleet** Terraform module (`lambda-fleet`) + `layer_lambda.manifest.json` from compile
+- **`serverless-data-mesh deploy`** — apply + terraform + optional Step Functions start
+- **`serverless-data-mesh catalog export`** — Backstage `catalog-info.yaml` entities
+- **`serverless-data-mesh ui`** — local mesh control panel (HTTP)
+- **JSON Schema** for `sdm/v1` contracts + VS Code YAML autocomplete
+- **Portable cloud** adapters (AWS production, GCP/Azure stubs)
+- **Docs**: `docs/first-mesh-on-aws.md` (5-minute guide)
+- **CI**: medallion Terraform validate, local benchmark, smoke-aws + benchmark workflows
+- **Benchmarks**: `run_local_benchmark.py`, `run_aws_benchmark.py`
+
+## [1.0.0] - 2026-06-14
+
+Production release: metadata-driven medallion mesh, zero-friction CLI, and Terraform deploy path.
+
+### Added
+
+- **Metadata-driven pipeline compiler** (`serverless-data-mesh compile`)
+- `sdm/v1` `DataProductPipeline` and `MedallionMesh` YAML contract models
+- Generated artifacts: handler, readers stub, pipeline_config, Step Functions, EventBridge, tests
+- `serverless_data_mesh.compile.runtime.run_metadata_pipeline`
+- Example contract: `examples/contracts/payments.mesh.pipeline.yaml`
+- **Real-world retail ETL walkthrough**: `examples/retail-mesh/` (5 pipelines, PySpark on Lambda)
+- **Medallion E2E mesh**: `examples/medallion-e2e/` — one YAML → bronze/silver/gold for all domains
+- **Complete metadata guide**: `docs/metadata-driven-pipeline.md` — full schema, deploy, CI/CD
+- **Zero-friction CLI**: `new`, `apply`, `validate`, `doctor` — YAML to full mesh in 2 commands
+- **Terraform medallion-mesh module** and `environments/medallion/` stack
+- Prod stack: mesh trust dashboard domains, optional Lake Formation governance module
+- Dependency: `pyyaml>=6`
+
+### Changed
+
+- `serverless-data-mesh init` now uses the compile pipeline (outputs `mesh.pipeline.yaml`)
+- PyPI classifier: **Production/Stable**
+
+## [0.2.0] - 2026-06-15
+
+### Added
+
+- **Auto VRP reprocessing** (`orchestration/reprocess.py`): detect drops, repair sink, re-proof, commit or escalate
+- **Consumer SLA enforcement** (`governance/consumer_sla.py`): freshness, completeness, required columns; LF grant hook
+- **Live CloudWatch trust dashboard** (Terraform `monitoring/dashboard.tf` + `publish_vrp_metric`)
+- **Grafana template** (`infrastructure/grafana/mesh-trust-dashboard.json`)
+- **Lake Formation governance module** (`terraform/modules/governance/`)
+- CLI: `serverless-data-mesh canary`, `reprocess-demo`, `dashboard --cloudwatch`
+- Scaffold: `consumer_sla.yaml`, `step_function.asl.json`, richer handler imports
+- Docs: `docs/mesh-trust-dashboard.md`
+- Tests: reprocess, consumer SLA, canary, auto-repair demo
+
+### Changed
+
+- `IceGuardDurableCoordinator` supports optional `sink_reader` + `enable_auto_repair`
+- Trust dashboard supports proofs scan, CloudWatch live mode, and demo fallback
+
+## [0.1.3] - 2026-06-15
+
+### Added
+
+- **Pure-Python fallback verifier** when veridata-recon wheels unavailable (Windows/Mac demo works)
+- `serverless-data-mesh init` domain scaffold CLI
+- `serverless-data-mesh dashboard` mesh trust dashboard HTML
+- `benchmarks/run_cost_estimate.py` with published cost estimates (100K/1M/10M)
+- `ConsumerSLAContract` type and `run_canary_comparison` for progressive delivery
+- Tests: fallback verifier, scaffold, dashboard
+
+### Changed
+
+- Local demo, gate demo, multi-domain, and benchmark use `create_proof_generator()` auto-backend
+- `benchmarks/results/2026-06-baseline.json` populated (pricing model estimated)
+
+## [0.1.2] - 2026-06-15
+
+### Added
+
+- **Local demo**: `serverless-data-mesh demo` runs PVDM in <60s without AWS (`LocalPVDMRuntime`)
+- `examples/tutorials/verification_gate_demo.py`: corrupt data blocked before consumers
+- `examples/multi-domain-orders-payments/`: deferred mesh leader commit + `test_atomicity.py`
+- `benchmarks/`: cost comparison methodology, workloads (100k/1M/10M), results placeholder
+- `serverless_data_mesh.lineage`: OpenLineage `emit_openlineage_event` on commit
+- Tests: `test_local_runtime.py`, `test_openlineage.py`
+
+## [0.1.1] - 2026-06-15
+
+### Added
+
+- **Vaquar Pattern** (`docs/vaquar-pattern.md`): proof-gated serverless lakehouse publication (PVDM)
+- Blog-style [why-serverless-data-mesh.md](docs/why-serverless-data-mesh.md) with connectivity diagrams
+- New diagram assets: `why-sdm-hero-connectivity.png`, `why-sdm-four-phase-connectivity.png`, `why-sdm-trust-gap.png`
+
+### Fixed
+
+- Hatch `VERSION` parsing: `pattern` for plain semver file (CI build green)
+- `__version__` reads from package metadata when installed; dev fallback uses `parents[2]`
+- Multi-account Terraform: Steward creates only checkpoint/proof buckets; Publisher creates only lakehouse
+- Storage module: optional per-bucket creation via nullable bucket names
+
+### Changed
+
+- Pattern catalog: Vaquar Pattern as flagship pattern #0 in [data-mesh-patterns.md](docs/data-mesh-patterns.md)
+
+## [0.1.0] - 2026-06-14
+
+### Added
+
+- Initial release: IceGuard + Durable Execution + veridata-recon + Glue REST
+- `IceGuardDurableCoordinator` for cross-domain transaction boundaries
+- `GlueRestCatalogAdapter` for SigV4 Iceberg metadata commits
+- `VRPProofGenerator` and `validate_then_commit` hooks
+- Domain writer Lambda example with SAM template
+- `VERSION` file, pre-commit, Dependabot, SECURITY.md, consumer safety benchmark
+- `SparkRulesConnector` (`[rules]` extra), PyPI publish workflow
+- CI workflow, unit tests, and documentation

{serverless_data_mesh-0.2.0 → serverless_data_mesh-1.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: serverless-data-mesh
-Version: 0.2.0
+Version: 1.2.0
 Summary: Federated data mesh framework for proof-gated, exactly-once Iceberg data product publication on AWS Lambda.
 Project-URL: Homepage, https://github.com/vaquarkhan/aws-serverless-datamesh-framework
 Project-URL: Documentation, https://github.com/vaquarkhan/aws-serverless-datamesh-framework#readme
@@ -10,8 +10,8 @@ Project-URL: PyPI, https://pypi.org/project/serverless-data-mesh/
 Author: Serverless Data Mesh Contributors
 License-Expression: Apache-2.0
 License-File: LICENSE
-Keywords: aws,data-as-a-product,data-mesh,domain-oriented,durable-execution,federated-data-mesh,federated-governance,iceberg,iceguard,lakehouse,lambda,pyiceberg,rule-engine,serverless,sparkrules,vaquar-pattern,veridata-recon
-Classifier: Development Status :: 3 - Alpha
+Keywords: aws,bronze-silver-gold,data-as-a-product,data-mesh,domain-oriented,durable-execution,etl,federated-data-mesh,federated-governance,iceberg,iceguard,lakehouse,lambda,medallion,metadata-driven,pyiceberg,rule-engine,serverless,sparkrules,vaquar-pattern,veridata-recon
+Classifier: Development Status :: 5 - Production/Stable
 Classifier: Intended Audience :: Developers
 Classifier: License :: OSI Approved :: Apache Software License
 Classifier: Operating System :: OS Independent
@@ -27,6 +27,7 @@ Requires-Dist: aws-durable-execution-sdk-python>=1.0.0
 Requires-Dist: boto3>=1.34
 Requires-Dist: iceguard>=1.0.0
 Requires-Dist: pyiceberg[glue,rest-sigv4]>=0.7.0
+Requires-Dist: pyyaml>=6
 Requires-Dist: veridata-recon>=0.1.0
 Provides-Extra: all
 Requires-Dist: pyspark>=3.5.0; extra == 'all'
@@ -66,6 +67,8 @@ Introduces the **[Vaquar Pattern](https://github.com/vaquarkhan/aws-serverless-d
 ```bash
 pip install serverless-data-mesh
 serverless-data-mesh demo    # <60s local PVDM demo, no AWS
+serverless-data-mesh new --template medallion --output my-mesh
+serverless-data-mesh apply --contract my-mesh/mesh.yaml --output my-mesh/generated
 pip install "serverless-data-mesh[rules]"   # + SparkRules on Lambda
 pip install "serverless-data-mesh[spark]"   # + PySpark + SparkRules
 pip install "serverless-data-mesh[all]"     # rules + spark

{serverless_data_mesh-0.2.0 → serverless_data_mesh-1.2.0}/PYPI.md

@@ -15,6 +15,8 @@ Introduces the **[Vaquar Pattern](https://github.com/vaquarkhan/aws-serverless-d
 ```bash
 pip install serverless-data-mesh
 serverless-data-mesh demo    # <60s local PVDM demo, no AWS
+serverless-data-mesh new --template medallion --output my-mesh
+serverless-data-mesh apply --contract my-mesh/mesh.yaml --output my-mesh/generated
 pip install "serverless-data-mesh[rules]"   # + SparkRules on Lambda
 pip install "serverless-data-mesh[spark]"   # + PySpark + SparkRules
 pip install "serverless-data-mesh[all]"     # rules + spark