composable-data-stack 0.7.0__tar.gz → 0.9.0__tar.gz

{composable_data_stack-0.7.0 → composable_data_stack-0.9.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: composable-data-stack
-Version: 0.7.0
+Version: 0.9.0
 Summary: CLI for validating and composing composable data stack profiles
 Author: Ronald Hensbergen
 License-Expression: Apache-2.0
@@ -22,16 +22,17 @@ Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: PyYAML>=6.0
 Requires-Dist: jsonschema>=4.22.0
-Requires-Dist: jinja2>=3.0
 Requires-Dist: packaging>=23.0
 Requires-Dist: rich>=15.0
 Provides-Extra: completion
 Requires-Dist: argcomplete<4,>=3.0.0; extra == "completion"
+Provides-Extra: test
+Requires-Dist: jinja2>=3.0; extra == "test"
 Provides-Extra: dev
 Requires-Dist: coverage>=7.6; extra == "dev"
 Requires-Dist: bandit>=1.7; extra == "dev"
 Requires-Dist: pip-audit>=2.7; extra == "dev"
-Requires-Dist: ruff==0.16.5; extra == "dev"
+Requires-Dist: ruff==0.16.7; extra == "dev"
 Dynamic: license-file
 
 # 🚀 Composable Data Stack (CDS)
@@ -41,6 +42,31 @@ Dynamic: license-file
 
 ![Composable Data Stack logo](assets/branding/logo.svg)
 
+Project:
+
+[![CI](https://github.com/RonaldHensbergen/composable-data-stack/actions/workflows/ci.yml/badge.svg)](https://github.com/RonaldHensbergen/composable-data-stack/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/composable-data-stack.svg)](https://pypi.org/project/composable-data-stack/)
+[![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)
+
+SonarCloud — Quality Gate & Ratings:
+
+[![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=alert_status)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)
+[![Reliability Rating](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=reliability_rating)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)
+[![Security Rating](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=security_rating)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)
+[![Maintainability Rating](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=sqale_rating)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)
+
+SonarCloud — Issues:
+
+[![Bugs](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=bugs)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)
+[![Vulnerabilities](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=vulnerabilities)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)
+[![Code Smells](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=code_smells)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)
+
+SonarCloud — Metrics:
+
+[![Coverage](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=coverage)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)
+[![Duplicated Lines (%)](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=duplicated_lines_density)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)
+[![Technical Debt](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=sqale_index)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)
+
 ---
 
 ## 🧠 What Is CDS (In 1 Minute)
@@ -135,7 +161,6 @@ CDS wires modules through **contracts**, not direct dependencies. This section h
 Below, `local-dagster-postgres-superset` wires Dagster to Postgres to Superset through contracts:
 
 ```mermaid
----
 flowchart TD
     Dagster[Dagster]
     Postgres[(Postgres)]
@@ -174,7 +199,6 @@ consumption in Superset), see
 [docs/profile-testing/failure-path-and-ci.md](docs/profile-testing/failure-path-and-ci.md).
 
 ```mermaid
----
 flowchart TD
     subgraph compile["Compile-time (cds)"]
         direction TB
@@ -254,6 +278,7 @@ When you run CDS:
 - resolved contract bindings
 - dependency-aware execution plan
 - generated Docker Compose configuration
+- generated Helm chart for Kubernetes
 - reproducible stack definition
 
 This allows you to go from a declarative profile to a runnable local data stack.
@@ -286,6 +311,10 @@ The `cds up` command and generated local profiles additionally require:
 - Write access to the checkout for `.env`, `docker-compose.yml`, and workdir
   data
 
+For Kubernetes, install Helm and kubectl and provide an explicit kube context.
+The local k3s workflow uses k3d and is documented in
+[docs/kubernetes.md](docs/kubernetes.md).
+
 Docker Desktop on Windows must use the WSL 2 backend. See the
 [support policy](docs/support-policy.md) for supported operating systems and
 runtime versions. For platform-specific setup from an empty machine, follow
@@ -450,6 +479,12 @@ cds render local-dagster-postgres-superset
 
 By default, this writes `docker-compose.yml` to the project root.
 
+Render the same plan as a Helm chart with:
+
+```bash
+cds render local-dagster-postgres-superset --target helm
+```
+
 Use a custom location when needed:
 
 ```bash
@@ -639,18 +674,90 @@ prod`; commands without `--environment` then use that overlay. An explicit
 
 ---
 
+### Profile Composition (`extends`)
+
+A profile can also factor out shared configuration into one or more parent
+profiles instead of duplicating it, using a top-level `extends` field:
+
+```yaml
+# profiles/analytics-prod/profile.yaml
+apiVersion: cds/v1alpha1
+kind: Profile
+metadata:
+  name: analytics-prod
+  environment: production
+extends:
+  - analytics-base   # a bare name resolves to profiles/analytics-base/profile.yaml
+spec:
+  modules:
+    - id: postgres
+      config:
+        storage:
+          size: 20Gi
+```
+
+`extends` accepts a non-empty list of parent references, each either a bare
+profile name (resolved under the profiles root) or a path relative to the
+child profile's directory (e.g. `../shared/profile.yaml`). Parents are
+resolved and merged left-to-right — later parents win over earlier ones —
+and then the child profile's own document is merged on top of all parents.
+Composition uses the exact same deep-merge/module-merge-by-id engine as
+environment overlays: mappings merge recursively, `spec.modules` entries
+merge by stable `id`, and any other array is replaced wholesale rather than
+concatenated. Parent profiles may themselves use `extends` (chains are
+resolved transitively).
+
+`extends` is not a CLI flag — it's read directly from `profile.yaml`, so
+every profile-consuming command resolves it automatically, with no new
+syntax to learn:
+
+```bash
+cds validate analytics-prod
+cds plan analytics-prod
+cds up analytics-prod
+```
+
+A profile can extend more than one parent, which is merged in the order
+listed (later entries win over earlier ones):
+
+```yaml
+extends:
+  - networking-base
+  - observability-base
+```
+
+`extends` and `--environment` compose together: parents are merged first,
+then the child, then the selected environment overlay is applied on top of
+that fully-composed result — so a shared base profile and environment
+promotion can both be used without duplicating configuration in either
+dimension.
+
+A malformed `extends` chain fails validation/planning before anything else
+runs, with a dedicated diagnostic code:
+
+|Code|Meaning|
+|---|---|
+|E110|`extends` is missing, not a list, empty, or contains a non-string/empty entry|
+|E111|A parent reference resolves outside the profiles root|
+|E112|A referenced parent profile does not exist|
+|E113|A cycle was detected in the `extends` chain|
+
+---
+
 ## ⚙️ CLI
 
 |Command|Description|
 |---|---|
 |cds get \<profile\> [--remote \<owner/repo\>] [--ref \<ref\>] [--local \<dir\>] [--into \<dir\>]|Fetch a profile plus its dependent module/runtime assets from GitHub into a local CDS layout|
+|cds list profiles\|modules\|images [--remote \<owner/repo\>] [--ref \<ref\>] [--local \<dir\>]|List available profiles, module sources, or module images and check for newer versions; add `--remote`/`--local` to inspect another repository before fetching from it|
 |cds init [profile]|Generate a project `.env` template from profile secret definitions|
-|cds validate [profile]|Validate modules and contracts|
+|cds validate [profile]|Validate modules and contracts; use `--target helm` for Kubernetes checks|
 |cds preflight [profile]|Check runtime tools, required environment values, and host ports without starting services|
 |cds plan [profile]|Resolve dependencies and generate an execution plan|
-|cds render [profile]|Generate Docker Compose configuration from a resolved plan|
-|cds up [profile]|Validate, plan, render, build, and start services with docker compose; logs output to a file and shows a live `cds state` view until the stack settles (use `--no-build` to skip build, `--detach` to skip the live view, `--log-file`/`--timeout`/`--no-color` to override defaults)|
-|cds state [profile]|Show running service status grouped by health (use `--no-color` to disable colored labels)|
+|cds render [profile]|Generate Docker Compose or a Helm chart from a resolved plan|
+|cds up [profile]|Validate, plan, render, and start the Compose or Helm target; use `--target helm` for Kubernetes|
+|cds down [profile]|Stop Compose or uninstall a Helm release; Helm PVCs are retained by default|
+|cds state [profile]|Show Compose services or Kubernetes workloads grouped by health|
 |cds test [profile]|One-shot smoke validation: validate, security, plan, and render|
 |cds security [profile]|Run rule-based security validation on a profile|
 |cds diff [profile] --from \<env\> --to \<env\>|Show effective configuration differences between two environment overlays, secrets never included|
@@ -676,6 +783,12 @@ destination root, `--dry-run` to inspect the copy plan first, and `--force` to
 replace conflicting local files. Successful fetches record tracking metadata
 in `.cds/get-manifest.json` for future update workflows.
 
+`cds list profiles`, `cds list modules`, and `cds list images` accept the same
+`--remote`/`--ref`/`--local` source-repository selection as `cds get`, so you
+can discover what's available in another repository (a fork, or an existing
+local checkout) before running `cds get` against it. Without these flags,
+`cds list` inspects the local project as before.
+
 ### Project defaults
 
 `cds config` manages the gitignored `.cds/config.json` file (or the path in
@@ -755,6 +868,8 @@ Common errors from `cds validate`, `cds plan`, and `cds render`, and how to fix
 | `[E041] ... Contract ref "x.y" points to unknown module "x"` | A `consumes` binding's `contractRef` refers to a module ID that isn't defined in the profile. | Check `spec.modules` for the correct module `id`, and confirm the contract ref follows `<module-id>.<contract-name>`. |
 | `[E041] ... but it does not provide "<contract-name>"` | The referenced module exists, but its `spec.provides` list doesn't expose that contract name. | Check the producing module's `module.yaml` for the contracts it actually provides, and fix the consumer's `contractRef` to match. |
 | `[E042] ... Contract kind mismatch` | The consumer expects one contract kind (e.g. `sql-database`) but the producer exposes a different kind. | Point the binding at a module that provides the expected contract kind, or update the consumer's expected kind if the mismatch is intentional. |
+| `[E103] ... config.image.tag is required ... when config.image.source is "registry"` | A module instance sets `config.image.source: registry` without also setting `config.image.tag`. | Set `config.image.tag` to a version, or switch back to `config.image.source: build`. |
+| `[W097] ... config.image.tag is "latest" with config.image.source "registry"` | `config.image.tag: latest` under `source: registry` still validates, but drifts silently between deploys instead of pinning a reproducible version. | Pin an explicit tag from `publish-images.yml`'s output (or `tests/fixtures/signed-images.json`) instead of `latest`. |
 
 All diagnostics print with their error code and YAML path (e.g. `spec.modules[1].config`), so search the profile file for that path to find the exact line to fix.
 
@@ -860,11 +975,11 @@ MVP ready:
 - security checks
 - profile composition
 - Docker Compose rendering
+- Kubernetes Helm rendering and local k3s lifecycle
+- runtime startup, health state, and shutdown
 
 Next:
 
-- runtime orchestration
-- Kubernetes support
 - advanced secret providers
 - stack bootstrap and health checks
 
@@ -899,6 +1014,8 @@ Good first contributions:
 ## 📖 Documentation
 
 - [Quickstart](README.md#-quickstart) — get running in 5 minutes
+- [How CDS Works](docs/how-it-works.md) — C4 + sequence diagrams of the validate → plan → render engine
+- [Kubernetes Target](docs/kubernetes.md): Helm rendering and isolated local k3s workflow
 - [From Docker Compose to CDS Profile](docs/from-docker-to-cds-profile.md) — complete transformation guide
 - [Architecture](docs/architecture.md) — design and core concepts
 - [Modules](docs/modules.md) — how to structure reusable components

{composable_data_stack-0.7.0 → composable_data_stack-0.9.0}/README.md

@@ -5,6 +5,31 @@
 
 ![Composable Data Stack logo](assets/branding/logo.svg)
 
+Project:
+
+[![CI](https://github.com/RonaldHensbergen/composable-data-stack/actions/workflows/ci.yml/badge.svg)](https://github.com/RonaldHensbergen/composable-data-stack/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/composable-data-stack.svg)](https://pypi.org/project/composable-data-stack/)
+[![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)
+
+SonarCloud — Quality Gate & Ratings:
+
+[![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=alert_status)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)
+[![Reliability Rating](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=reliability_rating)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)
+[![Security Rating](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=security_rating)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)
+[![Maintainability Rating](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=sqale_rating)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)
+
+SonarCloud — Issues:
+
+[![Bugs](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=bugs)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)
+[![Vulnerabilities](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=vulnerabilities)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)
+[![Code Smells](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=code_smells)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)
+
+SonarCloud — Metrics:
+
+[![Coverage](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=coverage)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)
+[![Duplicated Lines (%)](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=duplicated_lines_density)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)
+[![Technical Debt](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=sqale_index)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)
+
 ---
 
 ## 🧠 What Is CDS (In 1 Minute)
@@ -99,7 +124,6 @@ CDS wires modules through **contracts**, not direct dependencies. This section h
 Below, `local-dagster-postgres-superset` wires Dagster to Postgres to Superset through contracts:
 
 ```mermaid
----
 flowchart TD
     Dagster[Dagster]
     Postgres[(Postgres)]
@@ -138,7 +162,6 @@ consumption in Superset), see
 [docs/profile-testing/failure-path-and-ci.md](docs/profile-testing/failure-path-and-ci.md).
 
 ```mermaid
----
 flowchart TD
     subgraph compile["Compile-time (cds)"]
         direction TB
@@ -218,6 +241,7 @@ When you run CDS:
 - resolved contract bindings
 - dependency-aware execution plan
 - generated Docker Compose configuration
+- generated Helm chart for Kubernetes
 - reproducible stack definition
 
 This allows you to go from a declarative profile to a runnable local data stack.
@@ -250,6 +274,10 @@ The `cds up` command and generated local profiles additionally require:
 - Write access to the checkout for `.env`, `docker-compose.yml`, and workdir
   data
 
+For Kubernetes, install Helm and kubectl and provide an explicit kube context.
+The local k3s workflow uses k3d and is documented in
+[docs/kubernetes.md](docs/kubernetes.md).
+
 Docker Desktop on Windows must use the WSL 2 backend. See the
 [support policy](docs/support-policy.md) for supported operating systems and
 runtime versions. For platform-specific setup from an empty machine, follow
@@ -414,6 +442,12 @@ cds render local-dagster-postgres-superset
 
 By default, this writes `docker-compose.yml` to the project root.
 
+Render the same plan as a Helm chart with:
+
+```bash
+cds render local-dagster-postgres-superset --target helm
+```
+
 Use a custom location when needed:
 
 ```bash
@@ -603,18 +637,90 @@ prod`; commands without `--environment` then use that overlay. An explicit
 
 ---
 
+### Profile Composition (`extends`)
+
+A profile can also factor out shared configuration into one or more parent
+profiles instead of duplicating it, using a top-level `extends` field:
+
+```yaml
+# profiles/analytics-prod/profile.yaml
+apiVersion: cds/v1alpha1
+kind: Profile
+metadata:
+  name: analytics-prod
+  environment: production
+extends:
+  - analytics-base   # a bare name resolves to profiles/analytics-base/profile.yaml
+spec:
+  modules:
+    - id: postgres
+      config:
+        storage:
+          size: 20Gi
+```
+
+`extends` accepts a non-empty list of parent references, each either a bare
+profile name (resolved under the profiles root) or a path relative to the
+child profile's directory (e.g. `../shared/profile.yaml`). Parents are
+resolved and merged left-to-right — later parents win over earlier ones —
+and then the child profile's own document is merged on top of all parents.
+Composition uses the exact same deep-merge/module-merge-by-id engine as
+environment overlays: mappings merge recursively, `spec.modules` entries
+merge by stable `id`, and any other array is replaced wholesale rather than
+concatenated. Parent profiles may themselves use `extends` (chains are
+resolved transitively).
+
+`extends` is not a CLI flag — it's read directly from `profile.yaml`, so
+every profile-consuming command resolves it automatically, with no new
+syntax to learn:
+
+```bash
+cds validate analytics-prod
+cds plan analytics-prod
+cds up analytics-prod
+```
+
+A profile can extend more than one parent, which is merged in the order
+listed (later entries win over earlier ones):
+
+```yaml
+extends:
+  - networking-base
+  - observability-base
+```
+
+`extends` and `--environment` compose together: parents are merged first,
+then the child, then the selected environment overlay is applied on top of
+that fully-composed result — so a shared base profile and environment
+promotion can both be used without duplicating configuration in either
+dimension.
+
+A malformed `extends` chain fails validation/planning before anything else
+runs, with a dedicated diagnostic code:
+
+|Code|Meaning|
+|---|---|
+|E110|`extends` is missing, not a list, empty, or contains a non-string/empty entry|
+|E111|A parent reference resolves outside the profiles root|
+|E112|A referenced parent profile does not exist|
+|E113|A cycle was detected in the `extends` chain|
+
+---
+
 ## ⚙️ CLI
 
 |Command|Description|
 |---|---|
 |cds get \<profile\> [--remote \<owner/repo\>] [--ref \<ref\>] [--local \<dir\>] [--into \<dir\>]|Fetch a profile plus its dependent module/runtime assets from GitHub into a local CDS layout|
+|cds list profiles\|modules\|images [--remote \<owner/repo\>] [--ref \<ref\>] [--local \<dir\>]|List available profiles, module sources, or module images and check for newer versions; add `--remote`/`--local` to inspect another repository before fetching from it|
 |cds init [profile]|Generate a project `.env` template from profile secret definitions|
-|cds validate [profile]|Validate modules and contracts|
+|cds validate [profile]|Validate modules and contracts; use `--target helm` for Kubernetes checks|
 |cds preflight [profile]|Check runtime tools, required environment values, and host ports without starting services|
 |cds plan [profile]|Resolve dependencies and generate an execution plan|
-|cds render [profile]|Generate Docker Compose configuration from a resolved plan|
-|cds up [profile]|Validate, plan, render, build, and start services with docker compose; logs output to a file and shows a live `cds state` view until the stack settles (use `--no-build` to skip build, `--detach` to skip the live view, `--log-file`/`--timeout`/`--no-color` to override defaults)|
-|cds state [profile]|Show running service status grouped by health (use `--no-color` to disable colored labels)|
+|cds render [profile]|Generate Docker Compose or a Helm chart from a resolved plan|
+|cds up [profile]|Validate, plan, render, and start the Compose or Helm target; use `--target helm` for Kubernetes|
+|cds down [profile]|Stop Compose or uninstall a Helm release; Helm PVCs are retained by default|
+|cds state [profile]|Show Compose services or Kubernetes workloads grouped by health|
 |cds test [profile]|One-shot smoke validation: validate, security, plan, and render|
 |cds security [profile]|Run rule-based security validation on a profile|
 |cds diff [profile] --from \<env\> --to \<env\>|Show effective configuration differences between two environment overlays, secrets never included|
@@ -640,6 +746,12 @@ destination root, `--dry-run` to inspect the copy plan first, and `--force` to
 replace conflicting local files. Successful fetches record tracking metadata
 in `.cds/get-manifest.json` for future update workflows.
 
+`cds list profiles`, `cds list modules`, and `cds list images` accept the same
+`--remote`/`--ref`/`--local` source-repository selection as `cds get`, so you
+can discover what's available in another repository (a fork, or an existing
+local checkout) before running `cds get` against it. Without these flags,
+`cds list` inspects the local project as before.
+
 ### Project defaults
 
 `cds config` manages the gitignored `.cds/config.json` file (or the path in
@@ -719,6 +831,8 @@ Common errors from `cds validate`, `cds plan`, and `cds render`, and how to fix
 | `[E041] ... Contract ref "x.y" points to unknown module "x"` | A `consumes` binding's `contractRef` refers to a module ID that isn't defined in the profile. | Check `spec.modules` for the correct module `id`, and confirm the contract ref follows `<module-id>.<contract-name>`. |
 | `[E041] ... but it does not provide "<contract-name>"` | The referenced module exists, but its `spec.provides` list doesn't expose that contract name. | Check the producing module's `module.yaml` for the contracts it actually provides, and fix the consumer's `contractRef` to match. |
 | `[E042] ... Contract kind mismatch` | The consumer expects one contract kind (e.g. `sql-database`) but the producer exposes a different kind. | Point the binding at a module that provides the expected contract kind, or update the consumer's expected kind if the mismatch is intentional. |
+| `[E103] ... config.image.tag is required ... when config.image.source is "registry"` | A module instance sets `config.image.source: registry` without also setting `config.image.tag`. | Set `config.image.tag` to a version, or switch back to `config.image.source: build`. |
+| `[W097] ... config.image.tag is "latest" with config.image.source "registry"` | `config.image.tag: latest` under `source: registry` still validates, but drifts silently between deploys instead of pinning a reproducible version. | Pin an explicit tag from `publish-images.yml`'s output (or `tests/fixtures/signed-images.json`) instead of `latest`. |
 
 All diagnostics print with their error code and YAML path (e.g. `spec.modules[1].config`), so search the profile file for that path to find the exact line to fix.
 
@@ -824,11 +938,11 @@ MVP ready:
 - security checks
 - profile composition
 - Docker Compose rendering
+- Kubernetes Helm rendering and local k3s lifecycle
+- runtime startup, health state, and shutdown
 
 Next:
 
-- runtime orchestration
-- Kubernetes support
 - advanced secret providers
 - stack bootstrap and health checks
 
@@ -863,6 +977,8 @@ Good first contributions:
 ## 📖 Documentation
 
 - [Quickstart](README.md#-quickstart) — get running in 5 minutes
+- [How CDS Works](docs/how-it-works.md) — C4 + sequence diagrams of the validate → plan → render engine
+- [Kubernetes Target](docs/kubernetes.md): Helm rendering and isolated local k3s workflow
 - [From Docker Compose to CDS Profile](docs/from-docker-to-cds-profile.md) — complete transformation guide
 - [Architecture](docs/architecture.md) — design and core concepts
 - [Modules](docs/modules.md) — how to structure reusable components

{composable_data_stack-0.7.0 → composable_data_stack-0.9.0}/cli/getter.py

@@ -18,6 +18,7 @@ from urllib.error import HTTPError, URLError
 from urllib.request import Request, urlopen
 
 from .loader import load_yaml_file, resolve_module_dir
+from .overlay import _derive_profiles_root, _resolve_extends_ref, resolve_extends
 from .planner import MaxNestingDepthExceeded, apply_defaults, substitute_string
 
 # The upstream repository `cds get` downloads from when no `--remote` is
@@ -71,7 +72,7 @@ def fetch_profile(
     destination_root: Path | None = None,
     force: bool = False,
     dry_run: bool = False,
-) -> tuple[list[CopyAction], Path]:
+) -> tuple[list[CopyAction], Path, list[str]]:
     target_root = (destination_root or Path.cwd()).expanduser().resolve()
 
     with _prepare_source_repository(remote, ref, local) as source_repo:
@@ -79,10 +80,15 @@ def fetch_profile(
         asset_roots = _collect_asset_roots(source_repo, profile_path)
 
         actions = _build_copy_plan(source_repo, asset_roots, target_root)
+        # Conflict detection must happen while `source_repo` (which may be a
+        # temporary directory holding a downloaded tarball) is still alive,
+        # since `_find_conflicts()` reads each action's source file to
+        # compare content. Compute it up front so both the dry-run and
+        # real-run paths can report/enforce conflicts consistently (#452).
+        conflicts = _find_conflicts(actions)
         if dry_run:
-            return actions, target_root / _TRACKING_FILE
+            return actions, target_root / _TRACKING_FILE, conflicts
 
-        conflicts = _find_conflicts(actions)
         if conflicts and not force:
             rendered = ", ".join(conflicts[:5])
             extra = "" if len(conflicts) <= 5 else f" (+{len(conflicts) - 5} more)"
@@ -104,18 +110,37 @@ def fetch_profile(
             actions=actions,
             asset_roots=asset_roots,
         )
-    return actions, target_root / _TRACKING_FILE
+    return actions, target_root / _TRACKING_FILE, conflicts
 
 
-def format_get_plan(actions: list[CopyAction], *, destination_root: Path) -> str:
+def format_get_plan(
+    actions: list[CopyAction],
+    *,
+    destination_root: Path,
+    conflicts: list[str] | None = None,
+) -> str:
+    conflicts = conflicts or []
+    conflict_set = set(conflicts)
+    clean_actions = [a for a in actions if a.repo_relative_path not in conflict_set]
+
     if not actions:
         return f"No file changes required under {destination_root}."
 
     lines = [f"Planned {len(actions)} file(s) under {destination_root}:"]
-    for action in actions[:20]:
-        lines.append(f"  - {action.repo_relative_path}")
-    if len(actions) > 20:
-        lines.append(f"  - ... {len(actions) - 20} more")
+    if clean_actions:
+        lines.append(f"  Would copy ({len(clean_actions)}):")
+        for action in clean_actions[:20]:
+            lines.append(f"    - {action.repo_relative_path}")
+        if len(clean_actions) > 20:
+            lines.append(f"    - ... {len(clean_actions) - 20} more")
+
+    if conflicts:
+        lines.append(f"  Would conflict without --force ({len(conflicts)}):")
+        for path in conflicts[:20]:
+            lines.append(f"    - {path}")
+        if len(conflicts) > 20:
+            lines.append(f"    - ... {len(conflicts) - 20} more")
+
     return "\n".join(lines)
 
 
@@ -233,15 +258,67 @@ def _resolve_source_profile_path(source_repo: Path, profile: str) -> Path:
     )
 
 
+def _collect_extends_profile_dirs(
+    source_repo: Path, profile_path: Path, _visited: frozenset[Path] = frozenset()
+) -> set[Path]:
+    """
+    Returns every parent profile directory reachable via `extends`, so
+    `cds get` copies parent profile.yaml files (and, via
+    _collect_asset_roots's module walk, their modules) too instead of
+    silently omitting anything only declared in a parent profile. Reuses
+    cli.overlay's own extends-ref resolution rather than re-implementing it.
+
+    `_visited` tracks resolved profile paths already seen along the current
+    extends chain so a cyclic `extends` graph raises a clean GetError
+    instead of recursing until Python's recursion limit is hit; this
+    mirrors cli.overlay._compose_extends's own cycle-detection stack.
+    """
+    resolved_profile_path = profile_path.resolve()
+    if resolved_profile_path in _visited:
+        raise GetError(f"Cycle detected in extends chain at {profile_path}")
+    _visited = _visited | {resolved_profile_path}
+
+    doc, _diagnostics = load_yaml_file(profile_path)
+    if doc is None:
+        raise GetError(f"Could not load source profile {profile_path}")
+
+    extends = doc.get("extends")
+    if not extends:
+        return set()
+
+    profile_dir = profile_path.parent.resolve()
+    profiles_root = _derive_profiles_root(profile_dir)
+    if profiles_root is None:
+        raise GetError(f'Could not derive a profiles root for extends in {profile_path}')
+
+    dirs: set[Path] = set()
+    for ref in extends:
+        parent_path = _resolve_extends_ref(ref, profile_dir, profiles_root)
+        if not parent_path.is_file():
+            raise GetError(f'Could not resolve extends parent "{ref}" for {profile_path}')
+        _require_within_repo(parent_path, source_repo, f'extends parent "{ref}"')
+        dirs.add(parent_path.parent)
+        dirs.update(_collect_extends_profile_dirs(source_repo, parent_path, _visited))
+
+    return dirs
+
+
 def _collect_asset_roots(source_repo: Path, profile_path: Path) -> list[Path]:
     profile_dir = profile_path.parent
     asset_roots: set[Path] = {
         profile_dir if profile_path.name == "profile.yaml" else profile_path
     }
-    profile_doc, profile_diags = load_yaml_file(profile_path)
-    if profile_diags or profile_doc is None:
+
+    # Resolve through `extends` so modules/config contributed only by a
+    # parent profile (not redeclared in the child) are still collected;
+    # cli.overlay is the single source of truth for extends semantics (see
+    # cli.planner/cli.validator, which route through it the same way).
+    profile_doc, _provenance, diagnostics = resolve_extends(str(profile_path))
+    if profile_doc is None or any(d.level == "error" for d in diagnostics):
         raise GetError(f"Could not load source profile {profile_path}")
 
+    asset_roots.update(_collect_extends_profile_dirs(source_repo, profile_path))
+
     spec = profile_doc.get("spec")
     modules = spec.get("modules", []) if isinstance(spec, dict) else []
     if not isinstance(modules, list):
@@ -595,7 +672,19 @@ def _add_copy_action(
     destination_root: Path,
     actions_by_destination: dict[Path, CopyAction],
 ) -> None:
-    repo_relative = source_file.resolve().relative_to(source_repo.resolve())
+    resolved_source = source_file.resolve()
+    try:
+        repo_relative = resolved_source.relative_to(source_repo.resolve())
+    except ValueError as exc:
+        # A symlink (or directory junction) planted under an asset root can
+        # resolve to a real path outside the source repository. Without this
+        # guard, Path.relative_to() raises a bare ValueError that escapes as
+        # an unhandled traceback instead of the stable GetError callers rely
+        # on (#454).
+        raise GetError(
+            f"Asset {source_file} resolves outside the source repository "
+            f"({source_repo}): {resolved_source}"
+        ) from exc
     # Deliberately do NOT call .resolve() on the combined destination path:
     # destination_root is already an absolute, resolved path (see
     # fetch_profile()), and resolving the full path here would follow a
@@ -607,12 +696,12 @@ def _add_copy_action(
     existing = actions_by_destination.get(destination)
     if existing is None:
         actions_by_destination[destination] = CopyAction(
-            source=source_file.resolve(),
+            source=resolved_source,
             destination=destination,
             repo_relative_path=repo_relative.as_posix(),
         )
         return
-    if existing.source.resolve() != source_file.resolve():
+    if existing.source.resolve() != resolved_source:
         raise GetError(
             f"Multiple source files would map to the same destination: {destination}"
         )
@@ -649,10 +738,17 @@ def _write_actions(actions: list[CopyAction]) -> None:
     (e.g. a symlink planted at profiles/foo/profile.yaml pointing outside
     the destination tree)."""
     for action in actions:
-        action.destination.parent.mkdir(parents=True, exist_ok=True)
-        if action.destination.is_symlink():
-            action.destination.unlink()
-        shutil.copy2(action.source, action.destination)
+        try:
+            action.destination.parent.mkdir(parents=True, exist_ok=True)
+            if action.destination.is_symlink():
+                action.destination.unlink()
+            shutil.copy2(action.source, action.destination)
+        except OSError as exc:
+            # Surface filesystem failures (permission denied, read-only
+            # destination, disk full, ...) as a stable GetError instead of a
+            # raw OSError, so callers get a predictable error path and
+            # message regardless of the underlying platform/errno (#454).
+            raise GetError(f"Could not write {action.destination}: {exc}") from exc
 
 
 def _write_tracking_manifest(