composable-data-stack 0.8.0__tar.gz → 0.9.1__tar.gz

{composable_data_stack-0.8.0 → composable_data_stack-0.9.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: composable-data-stack
-Version: 0.8.0
+Version: 0.9.1
 Summary: CLI for validating and composing composable data stack profiles
 Author: Ronald Hensbergen
 License-Expression: Apache-2.0
@@ -22,64 +22,169 @@ Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: PyYAML>=6.0
 Requires-Dist: jsonschema>=4.22.0
-Requires-Dist: jinja2>=3.0
 Requires-Dist: packaging>=23.0
 Requires-Dist: rich>=15.0
 Provides-Extra: completion
 Requires-Dist: argcomplete<4,>=3.0.0; extra == "completion"
+Provides-Extra: test
+Requires-Dist: jinja2>=3.0; extra == "test"
 Provides-Extra: dev
 Requires-Dist: coverage>=7.6; extra == "dev"
 Requires-Dist: bandit>=1.7; extra == "dev"
 Requires-Dist: pip-audit>=2.7; extra == "dev"
-Requires-Dist: ruff==0.16.6; extra == "dev"
+Requires-Dist: ruff==0.16.8; extra == "dev"
 Dynamic: license-file
 
 # 🚀 Composable Data Stack (CDS)
 
-> **Terraform for data platforms.**
-> Build, validate, secure, and evolve data stacks using modular components and explicit contracts.
+> **The modern data stack has a composition problem. CDS is an open framework
+> for building data platforms that can evolve without being rebuilt.**
 
-![Composable Data Stack logo](assets/branding/logo.svg)
+![Composable Data Stack logo](https://raw.githubusercontent.com/RonaldHensbergen/composable-data-stack/main/assets/branding/logo.svg)
 
----
+Project:
+
+[![CI](https://github.com/RonaldHensbergen/composable-data-stack/actions/workflows/ci.yml/badge.svg)](https://github.com/RonaldHensbergen/composable-data-stack/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/composable-data-stack.svg)](https://pypi.org/project/composable-data-stack/)
+[![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)
+
+SonarCloud — Quality Gate & Ratings:
 
-## 🧠 What Is CDS (In 1 Minute)
+[![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=alert_status)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)
+[![Reliability Rating](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=reliability_rating)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)
+[![Security Rating](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=security_rating)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)
+[![Maintainability Rating](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=sqale_rating)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)
 
-Composable Data Stack (CDS) is a framework for defining and assembling data platforms from reusable modules such as orchestrators, warehouses, BI tools, and secrets providers.
+SonarCloud — Issues:
 
-## 🤝 Get Involved
+[![Bugs](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=bugs)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)
+[![Vulnerabilities](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=vulnerabilities)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)
+[![Code Smells](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=code_smells)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)
 
-- **Star and follow** on GitHub: [RonaldHensbergen/composable-data-stack](https://github.com/RonaldHensbergen/composable-data-stack)
-- **Contribute**: open a discussion, file an issue, or send a PR to help shape CDS
-- **Proof it**: if you run it in a real workflow, share your feedback — good or bad
+SonarCloud — Metrics:
 
-> **Note:** Development helper tools are located in the `tools/` directory (git-ignored). See `tools/pr-cli/README.md` for PR creation scripts.
+[![Coverage](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=coverage)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)
+[![Duplicated Lines (%)](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=duplicated_lines_density)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)
+[![Technical Debt](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=sqale_index)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)
 
 ---
 
-## ⚡ Why CDS
+You start with a straightforward goal: orchestrate some pipelines, store the
+data, and make it useful.
 
-Instead of hardcoding integrations or relying on fragile pipelines, CDS introduces:
+Then the tool decisions begin.
 
-- 🔧 **Modules**: reusable components (Dagster, Postgres, Superset)
-- 🔗 **Contracts**: explicit interfaces between components
-- 🧩 **Profiles**: fully composed, runnable stacks
+Airflow, Dagster, or Prefect? Iceberg, Delta, or Hudi? Superset or Metabase?
+Every choice opens another comparison, another integration, and another
+architecture diagram. Every diagram looks different. Every tool has its own
+configuration model, assumptions, and lifecycle. Before long, the platform is
+less a designed system than a collection of decisions held together by custom
+code and institutional memory.
 
-Think of it as Infrastructure as Code, but for data platforms.
+The hardest question is no longer **"Which tool is best?"** It is **"How do
+these tools fit together, and can we change our minds later?"**
 
-Modern data platforms force a trade-off:
+## Why Data Platforms Have a Composition Problem
 
-|Approach|Problem|
-|---|---|
-|Monolithic stack|Rigid, hard to evolve|
-|Custom pipelines|Flexible but fragile and inconsistent|
+The modern data ecosystem is excellent at producing capable tools and terrible
+at making them work as interchangeable parts.
+
+Teams are expected to assemble orchestration, storage, transformation,
+cataloging, observability, security, and BI into one dependable platform. But
+the boundaries between those tools are usually implicit. Connection details
+leak into environment variables. Service names become dependencies. Deployment
+logic assumes a specific vendor. Compatibility lives in a staff engineer's
+head—or is discovered during an incident.
+
+That fragmentation creates a familiar cycle:
+
+- evaluate an overwhelming number of tools
+- debate architecture instead of delivering data products
+- hand-build integrations that only one team understands
+- duplicate stacks across development, CI, and production
+- postpone upgrades because nobody knows what they will break
+- accept lock-in because replacing one component feels like replacing the
+  platform
+
+This is not primarily a tooling problem. **It is a composition problem.** We
+have many good components, but no shared way to declare what they provide,
+what they need, and whether a particular assembly is valid.
+
+## The Cost of Choosing Wrong
+
+A technology choice that looks local rarely stays local.
+
+Choose an orchestrator and its concepts spread into deployment, secrets,
+logging, and team workflows. Choose a table format and it constrains engines,
+catalogs, and migration paths. Choose a BI layer and dashboards become coupled
+to authentication, database behavior, and operational assumptions.
+
+The bill arrives later:
+
+- migrations become multi-quarter programs
+- upgrades become risky, manual compatibility exercises
+- platform teams maintain glue code instead of platform capabilities
+- product teams wait while engineers renegotiate architecture
+- vendors gain leverage because the cost of leaving keeps rising
+
+There is no universally correct stack, and pretending otherwise is expensive.
+The right choice today may be the wrong choice at a different scale, under a
+new regulatory constraint, or after the team learns more. A healthy platform
+must make change routine—not heroic.
+
+## What If Data Platforms Were Composable?
+
+What if a data platform described capabilities and interfaces before naming
+products?
+
+What if an orchestrator could request a database contract without hardcoding a
+specific database? What if a profile could declare a supported combination,
+validate it before deployment, and produce the runtime configuration? What if
+replacing a component meant satisfying the same contract instead of rebuilding
+every integration around it?
+
+That is the goal of **Composable Data Stack (CDS)**.
+
+CDS is an open-source framework for defining, validating, securing, and
+assembling data platforms from reusable components. Think of it as
+**Infrastructure as Code for the shape of a data platform**: the components,
+the interfaces between them, and the rules that make the whole stack coherent.
+
+CDS introduces three building blocks:
 
-CDS gives you the best of both:
+- 🔧 **Modules** package reusable capabilities such as orchestration, storage,
+  BI, and secret management.
+- 🔗 **Contracts** define explicit, testable interfaces between those
+  capabilities.
+- 🧩 **Profiles** compose modules into supported, reproducible stacks.
 
-- composability without chaos
-- flexibility with guarantees
-- modularity with structure
-- no vendor lock-in by design
+Instead of burying platform architecture in scripts and tribal knowledge, CDS
+makes it declarative. It answers:
+
+- **How do these tools fit together?** Through explicit contracts and declared
+  bindings.
+- **What happens if I replace a component later?** CDS validates the new
+  composition against the same requirements before runtime.
+- **How do I avoid vendor and technology lock-in?** Consumers depend on
+  capabilities, not hardcoded product identities.
+
+The result is composability without chaos, flexibility with guarantees, and a
+platform that can evolve without starting over.
+
+## ⚡ Why CDS
+
+Most teams are forced into one of two bad options:
+
+|Approach|What You Gain|What It Costs|
+|---|---|---|
+|Monolithic stack|A consistent starting point|Rigid boundaries and an expensive exit|
+|Custom integrations|Freedom to choose every component|Fragile glue, inconsistent stacks, and hidden coupling|
+|CDS|Explicit composition and replaceable components|A contract and profile model the community maintains together|
+
+CDS does not claim that every tool is interchangeable or that integration
+complexity disappears. It makes compatibility explicit, validates known
+constraints early, and gives the ecosystem a shared place to improve how tools
+compose.
 
 ---
 
@@ -87,10 +192,12 @@ CDS gives you the best of both:
 
 Use CDS if you:
 
+- expect your architecture to evolve as requirements change
 - want to swap tools (Airflow ↔ Dagster, Superset ↔ Metabase)
 - need reproducible environments across dev, CI, and prod
 - are building a platform for multiple teams
 - want contract-driven integration instead of implicit coupling
+- want to contribute to an open interoperability model for data tooling
 
 CDS may be overkill if:
 
@@ -99,7 +206,7 @@ CDS may be overkill if:
 
 ---
 
-## 🏗️ Example
+## 🏗️ See Composition in Practice
 
 The `local-dagster-postgres-superset` profile defines:
 
@@ -107,7 +214,7 @@ The `local-dagster-postgres-superset` profile defines:
 - Postgres -> storage
 - Superset -> BI
 
-### What CDS Does
+CDS then:
 
 1. Validates module definitions
 2. Resolves contract bindings
@@ -116,7 +223,8 @@ The `local-dagster-postgres-superset` profile defines:
 
 `cds plan` resolves the full dependency graph before runtime configuration is generated, ensuring all module interactions are valid and predictable.
 
-You can replace components without changing system behavior:
+The composition model creates a path to replace components while preserving
+the contracts the rest of the platform depends on:
 
 ```text
 Dagster -> Airflow
@@ -134,23 +242,7 @@ CDS wires modules through **contracts**, not direct dependencies. This section h
 
 Below, `local-dagster-postgres-superset` wires Dagster to Postgres to Superset through contracts:
 
-```mermaid
-flowchart TD
-    Dagster[Dagster]
-    Postgres[(Postgres)]
-    Superset[Superset]
-
-    Dagster -->|transformation-runner| Postgres
-    Postgres -->|warehouse-query| Superset
-
-    classDef tool stroke:#818cf8,fill:#eef2ff
-    classDef database stroke:#2dd4bf,fill:#f0fdfa
-    classDef viz stroke:#a78bfa,fill:#f5f3ff
-
-    class Dagster tool
-    class Postgres database
-    class Superset viz
-```
+![Dagster, Postgres, and Superset connected through contracts](https://raw.githubusercontent.com/RonaldHensbergen/composable-data-stack/main/docs/diagrams/architecture/architecture_overview.svg)
 
 ### Internal Flow
 
@@ -172,51 +264,7 @@ consumption in Superset), see
 [docs/profile-testing/test-plan.md](docs/profile-testing/test-plan.md) and
 [docs/profile-testing/failure-path-and-ci.md](docs/profile-testing/failure-path-and-ci.md).
 
-```mermaid
-flowchart TD
-    subgraph compile["Compile-time (cds)"]
-        direction TB
-        Profile[/profile.yaml/]
-        Validate[Validate]
-        Security["Security checks<br/>(cds test only)"]
-        Plan[Plan]
-        Render[Render]
-        Compose[/docker-compose.yaml/]
-        Stop1((stops here))
-
-        Profile --> Validate
-        Validate -->|structural + contract checks| Security
-        Security -->|rule-based checks| Plan
-        Plan -->|resolve + substitute| Render
-        Render --> Compose
-
-        Validate -.->|E020, E041, E042, E081| Stop1
-    end
-
-    subgraph runtime["Runtime (docker compose, cds up only)"]
-        direction TB
-        Build["docker compose build<br/>(skip with --no-build)"]
-        Up["docker compose up"]
-        Env[(".env file")]
-        Containers["running containers,<br/>real secret values injected"]
-
-        Build --> Up
-        Env -.->|resolves CDS_VAR| Up
-        Up --> Containers
-    end
-
-    Compose --> Build
-
-    classDef stage stroke:#818cf8,fill:#eef2ff
-    classDef artifact stroke:#2dd4bf,fill:#f0fdfa
-    classDef stop stroke:#f87171,fill:#fef2f2,stroke-dasharray: 3 3
-    classDef runtimeNode stroke:#a78bfa,fill:#f5f3ff
-
-    class Validate,Security,Plan,Render stage
-    class Profile,Compose artifact
-    class Stop1 stop
-    class Build,Up,Env,Containers runtimeNode
-```
+![CDS compile-time and runtime flow](https://raw.githubusercontent.com/RonaldHensbergen/composable-data-stack/main/docs/diagrams/processes/internal_flow.svg)
 
 This mirrors the [`cds` command table](#️-cli) below: `validate`, `plan`, and `render` are each callable on their own; `security` only runs as part of `cds test`, not `cds up`. Module and contract definitions follow the [Contract-First](#contract-first) design principle, so most of what "Validate" and "Plan" check comes directly from `module.yaml` and `profile.yaml`.
 
@@ -246,15 +294,18 @@ cds security local-dagster-postgres-superset
 
 ## 📦 What You Get
 
-When you run CDS:
+When a profile compiles, CDS produces:
 
 - validated module graph
 - resolved contract bindings
 - dependency-aware execution plan
 - generated Docker Compose configuration
+- generated Helm chart for Kubernetes
 - reproducible stack definition
 
-This allows you to go from a declarative profile to a runnable local data stack.
+This turns a declarative platform design into a reproducible, runnable local
+data stack—and turns architectural assumptions into things the project can
+inspect, validate, and improve.
 
 ---
 
@@ -284,6 +335,10 @@ The `cds up` command and generated local profiles additionally require:
 - Write access to the checkout for `.env`, `docker-compose.yml`, and workdir
   data
 
+For Kubernetes, install Helm and kubectl and provide an explicit kube context.
+The local k3s workflow uses k3d and is documented in
+[docs/kubernetes.md](docs/kubernetes.md).
+
 Docker Desktop on Windows must use the WSL 2 backend. See the
 [support policy](docs/support-policy.md) for supported operating systems and
 runtime versions. For platform-specific setup from an empty machine, follow
@@ -448,6 +503,12 @@ cds render local-dagster-postgres-superset
 
 By default, this writes `docker-compose.yml` to the project root.
 
+Render the same plan as a Helm chart with:
+
+```bash
+cds render local-dagster-postgres-superset --target helm
+```
+
 Use a custom location when needed:
 
 ```bash
@@ -714,12 +775,13 @@ runs, with a dedicated diagnostic code:
 |cds get \<profile\> [--remote \<owner/repo\>] [--ref \<ref\>] [--local \<dir\>] [--into \<dir\>]|Fetch a profile plus its dependent module/runtime assets from GitHub into a local CDS layout|
 |cds list profiles\|modules\|images [--remote \<owner/repo\>] [--ref \<ref\>] [--local \<dir\>]|List available profiles, module sources, or module images and check for newer versions; add `--remote`/`--local` to inspect another repository before fetching from it|
 |cds init [profile]|Generate a project `.env` template from profile secret definitions|
-|cds validate [profile]|Validate modules and contracts|
+|cds validate [profile]|Validate modules and contracts; use `--target helm` for Kubernetes checks|
 |cds preflight [profile]|Check runtime tools, required environment values, and host ports without starting services|
 |cds plan [profile]|Resolve dependencies and generate an execution plan|
-|cds render [profile]|Generate Docker Compose configuration from a resolved plan|
-|cds up [profile]|Validate, plan, render, build, and start services with docker compose; logs output to a file and shows a live `cds state` view until the stack settles (use `--no-build` to skip build, `--detach` to skip the live view, `--log-file`/`--timeout`/`--no-color` to override defaults)|
-|cds state [profile]|Show running service status grouped by health (use `--no-color` to disable colored labels)|
+|cds render [profile]|Generate Docker Compose or a Helm chart from a resolved plan|
+|cds up [profile]|Validate, plan, render, and start the Compose or Helm target; use `--target helm` for Kubernetes|
+|cds down [profile]|Stop Compose or uninstall a Helm release; Helm PVCs are retained by default|
+|cds state [profile]|Show Compose services or Kubernetes workloads grouped by health|
 |cds test [profile]|One-shot smoke validation: validate, security, plan, and render|
 |cds security [profile]|Run rule-based security validation on a profile|
 |cds diff [profile] --from \<env\> --to \<env\>|Show effective configuration differences between two environment overlays, secrets never included|
@@ -937,11 +999,11 @@ MVP ready:
 - security checks
 - profile composition
 - Docker Compose rendering
+- Kubernetes Helm rendering and local k3s lifecycle
+- runtime startup, health state, and shutdown
 
 Next:
 
-- runtime orchestration
-- Kubernetes support
 - advanced secret providers
 - stack bootstrap and health checks
 
@@ -952,7 +1014,20 @@ See [docs/support-policy.md](docs/support-policy.md) for OS support policy and p
 
 ## 🤝 Contributing
 
-Contributions are welcome.
+CDS will only become a useful interoperability layer if the people building and
+operating data platforms shape it together.
+
+- **Run a profile** and report what works, what breaks, and what is missing.
+- **Document compatibility** between providers and consumers with evidence.
+- **Add or improve modules** without coupling them to a specific stack.
+- **Strengthen the model** through contracts, validation, security rules, and
+  representative profiles.
+- **Star and follow** the
+  [repository](https://github.com/RonaldHensbergen/composable-data-stack) if
+  you want to track the project.
+
+Real-world feedback is especially valuable. CDS should reflect how platforms
+actually fail and evolve, not how architecture diagrams pretend they do.
 
 Please read these first:
 
@@ -964,18 +1039,16 @@ Please read these first:
 - [CHANGELOG.md](CHANGELOG.md)
 - [RELEASE.md](RELEASE.md)
 
-Good first contributions:
-
-- adding new modules
-- improving profile examples
-- extending contract definitions
-- adding validation or security rules
+Development helper tools are located in the gitignored `tools/` directory. See
+`tools/pr-cli/README.md` for PR creation scripts.
 
 ---
 
 ## 📖 Documentation
 
 - [Quickstart](README.md#-quickstart) — get running in 5 minutes
+- [How CDS Works](docs/how-it-works.md) — C4 + sequence diagrams of the validate → plan → render engine
+- [Kubernetes Target](docs/kubernetes.md): Helm rendering and isolated local k3s workflow
 - [From Docker Compose to CDS Profile](docs/from-docker-to-cds-profile.md) — complete transformation guide
 - [Architecture](docs/architecture.md) — design and core concepts
 - [Modules](docs/modules.md) — how to structure reusable components