composable-data-stack 0.7.0__tar.gz → 0.8.0__tar.gz

{composable_data_stack-0.7.0 → composable_data_stack-0.8.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: composable-data-stack
-Version: 0.7.0
+Version: 0.8.0
 Summary: CLI for validating and composing composable data stack profiles
 Author: Ronald Hensbergen
 License-Expression: Apache-2.0
@@ -31,7 +31,7 @@ Provides-Extra: dev
 Requires-Dist: coverage>=7.6; extra == "dev"
 Requires-Dist: bandit>=1.7; extra == "dev"
 Requires-Dist: pip-audit>=2.7; extra == "dev"
-Requires-Dist: ruff==0.16.5; extra == "dev"
+Requires-Dist: ruff==0.16.6; extra == "dev"
 Dynamic: license-file
 
 # 🚀 Composable Data Stack (CDS)
@@ -135,7 +135,6 @@ CDS wires modules through **contracts**, not direct dependencies. This section h
 Below, `local-dagster-postgres-superset` wires Dagster to Postgres to Superset through contracts:
 
 ```mermaid
----
 flowchart TD
     Dagster[Dagster]
     Postgres[(Postgres)]
@@ -174,7 +173,6 @@ consumption in Superset), see
 [docs/profile-testing/failure-path-and-ci.md](docs/profile-testing/failure-path-and-ci.md).
 
 ```mermaid
----
 flowchart TD
     subgraph compile["Compile-time (cds)"]
         direction TB
@@ -639,11 +637,82 @@ prod`; commands without `--environment` then use that overlay. An explicit
 
 ---
 
+### Profile Composition (`extends`)
+
+A profile can also factor out shared configuration into one or more parent
+profiles instead of duplicating it, using a top-level `extends` field:
+
+```yaml
+# profiles/analytics-prod/profile.yaml
+apiVersion: cds/v1alpha1
+kind: Profile
+metadata:
+  name: analytics-prod
+  environment: production
+extends:
+  - analytics-base   # a bare name resolves to profiles/analytics-base/profile.yaml
+spec:
+  modules:
+    - id: postgres
+      config:
+        storage:
+          size: 20Gi
+```
+
+`extends` accepts a non-empty list of parent references, each either a bare
+profile name (resolved under the profiles root) or a path relative to the
+child profile's directory (e.g. `../shared/profile.yaml`). Parents are
+resolved and merged left-to-right — later parents win over earlier ones —
+and then the child profile's own document is merged on top of all parents.
+Composition uses the exact same deep-merge/module-merge-by-id engine as
+environment overlays: mappings merge recursively, `spec.modules` entries
+merge by stable `id`, and any other array is replaced wholesale rather than
+concatenated. Parent profiles may themselves use `extends` (chains are
+resolved transitively).
+
+`extends` is not a CLI flag — it's read directly from `profile.yaml`, so
+every profile-consuming command resolves it automatically, with no new
+syntax to learn:
+
+```bash
+cds validate analytics-prod
+cds plan analytics-prod
+cds up analytics-prod
+```
+
+A profile can extend more than one parent, which is merged in the order
+listed (later entries win over earlier ones):
+
+```yaml
+extends:
+  - networking-base
+  - observability-base
+```
+
+`extends` and `--environment` compose together: parents are merged first,
+then the child, then the selected environment overlay is applied on top of
+that fully-composed result — so a shared base profile and environment
+promotion can both be used without duplicating configuration in either
+dimension.
+
+A malformed `extends` chain fails validation/planning before anything else
+runs, with a dedicated diagnostic code:
+
+|Code|Meaning|
+|---|---|
+|E110|`extends` is missing, not a list, empty, or contains a non-string/empty entry|
+|E111|A parent reference resolves outside the profiles root|
+|E112|A referenced parent profile does not exist|
+|E113|A cycle was detected in the `extends` chain|
+
+---
+
 ## ⚙️ CLI
 
 |Command|Description|
 |---|---|
 |cds get \<profile\> [--remote \<owner/repo\>] [--ref \<ref\>] [--local \<dir\>] [--into \<dir\>]|Fetch a profile plus its dependent module/runtime assets from GitHub into a local CDS layout|
+|cds list profiles\|modules\|images [--remote \<owner/repo\>] [--ref \<ref\>] [--local \<dir\>]|List available profiles, module sources, or module images and check for newer versions; add `--remote`/`--local` to inspect another repository before fetching from it|
 |cds init [profile]|Generate a project `.env` template from profile secret definitions|
 |cds validate [profile]|Validate modules and contracts|
 |cds preflight [profile]|Check runtime tools, required environment values, and host ports without starting services|
@@ -676,6 +745,12 @@ destination root, `--dry-run` to inspect the copy plan first, and `--force` to
 replace conflicting local files. Successful fetches record tracking metadata
 in `.cds/get-manifest.json` for future update workflows.
 
+`cds list profiles`, `cds list modules`, and `cds list images` accept the same
+`--remote`/`--ref`/`--local` source-repository selection as `cds get`, so you
+can discover what's available in another repository (a fork, or an existing
+local checkout) before running `cds get` against it. Without these flags,
+`cds list` inspects the local project as before.
+
 ### Project defaults
 
 `cds config` manages the gitignored `.cds/config.json` file (or the path in
@@ -755,6 +830,8 @@ Common errors from `cds validate`, `cds plan`, and `cds render`, and how to fix
 | `[E041] ... Contract ref "x.y" points to unknown module "x"` | A `consumes` binding's `contractRef` refers to a module ID that isn't defined in the profile. | Check `spec.modules` for the correct module `id`, and confirm the contract ref follows `<module-id>.<contract-name>`. |
 | `[E041] ... but it does not provide "<contract-name>"` | The referenced module exists, but its `spec.provides` list doesn't expose that contract name. | Check the producing module's `module.yaml` for the contracts it actually provides, and fix the consumer's `contractRef` to match. |
 | `[E042] ... Contract kind mismatch` | The consumer expects one contract kind (e.g. `sql-database`) but the producer exposes a different kind. | Point the binding at a module that provides the expected contract kind, or update the consumer's expected kind if the mismatch is intentional. |
+| `[E103] ... config.image.tag is required ... when config.image.source is "registry"` | A module instance sets `config.image.source: registry` without also setting `config.image.tag`. | Set `config.image.tag` to a version, or switch back to `config.image.source: build`. |
+| `[W097] ... config.image.tag is "latest" with config.image.source "registry"` | `config.image.tag: latest` under `source: registry` still validates, but drifts silently between deploys instead of pinning a reproducible version. | Pin an explicit tag from `publish-images.yml`'s output (or `tests/fixtures/signed-images.json`) instead of `latest`. |
 
 All diagnostics print with their error code and YAML path (e.g. `spec.modules[1].config`), so search the profile file for that path to find the exact line to fix.
 

{composable_data_stack-0.7.0 → composable_data_stack-0.8.0}/README.md

@@ -99,7 +99,6 @@ CDS wires modules through **contracts**, not direct dependencies. This section h
 Below, `local-dagster-postgres-superset` wires Dagster to Postgres to Superset through contracts:
 
 ```mermaid
----
 flowchart TD
     Dagster[Dagster]
     Postgres[(Postgres)]
@@ -138,7 +137,6 @@ consumption in Superset), see
 [docs/profile-testing/failure-path-and-ci.md](docs/profile-testing/failure-path-and-ci.md).
 
 ```mermaid
----
 flowchart TD
     subgraph compile["Compile-time (cds)"]
         direction TB
@@ -603,11 +601,82 @@ prod`; commands without `--environment` then use that overlay. An explicit
 
 ---
 
+### Profile Composition (`extends`)
+
+A profile can also factor out shared configuration into one or more parent
+profiles instead of duplicating it, using a top-level `extends` field:
+
+```yaml
+# profiles/analytics-prod/profile.yaml
+apiVersion: cds/v1alpha1
+kind: Profile
+metadata:
+  name: analytics-prod
+  environment: production
+extends:
+  - analytics-base   # a bare name resolves to profiles/analytics-base/profile.yaml
+spec:
+  modules:
+    - id: postgres
+      config:
+        storage:
+          size: 20Gi
+```
+
+`extends` accepts a non-empty list of parent references, each either a bare
+profile name (resolved under the profiles root) or a path relative to the
+child profile's directory (e.g. `../shared/profile.yaml`). Parents are
+resolved and merged left-to-right — later parents win over earlier ones —
+and then the child profile's own document is merged on top of all parents.
+Composition uses the exact same deep-merge/module-merge-by-id engine as
+environment overlays: mappings merge recursively, `spec.modules` entries
+merge by stable `id`, and any other array is replaced wholesale rather than
+concatenated. Parent profiles may themselves use `extends` (chains are
+resolved transitively).
+
+`extends` is not a CLI flag — it's read directly from `profile.yaml`, so
+every profile-consuming command resolves it automatically, with no new
+syntax to learn:
+
+```bash
+cds validate analytics-prod
+cds plan analytics-prod
+cds up analytics-prod
+```
+
+A profile can extend more than one parent, which is merged in the order
+listed (later entries win over earlier ones):
+
+```yaml
+extends:
+  - networking-base
+  - observability-base
+```
+
+`extends` and `--environment` compose together: parents are merged first,
+then the child, then the selected environment overlay is applied on top of
+that fully-composed result — so a shared base profile and environment
+promotion can both be used without duplicating configuration in either
+dimension.
+
+A malformed `extends` chain fails validation/planning before anything else
+runs, with a dedicated diagnostic code:
+
+|Code|Meaning|
+|---|---|
+|E110|`extends` is missing, not a list, empty, or contains a non-string/empty entry|
+|E111|A parent reference resolves outside the profiles root|
+|E112|A referenced parent profile does not exist|
+|E113|A cycle was detected in the `extends` chain|
+
+---
+
 ## ⚙️ CLI
 
 |Command|Description|
 |---|---|
 |cds get \<profile\> [--remote \<owner/repo\>] [--ref \<ref\>] [--local \<dir\>] [--into \<dir\>]|Fetch a profile plus its dependent module/runtime assets from GitHub into a local CDS layout|
+|cds list profiles\|modules\|images [--remote \<owner/repo\>] [--ref \<ref\>] [--local \<dir\>]|List available profiles, module sources, or module images and check for newer versions; add `--remote`/`--local` to inspect another repository before fetching from it|
 |cds init [profile]|Generate a project `.env` template from profile secret definitions|
 |cds validate [profile]|Validate modules and contracts|
 |cds preflight [profile]|Check runtime tools, required environment values, and host ports without starting services|
@@ -640,6 +709,12 @@ destination root, `--dry-run` to inspect the copy plan first, and `--force` to
 replace conflicting local files. Successful fetches record tracking metadata
 in `.cds/get-manifest.json` for future update workflows.
 
+`cds list profiles`, `cds list modules`, and `cds list images` accept the same
+`--remote`/`--ref`/`--local` source-repository selection as `cds get`, so you
+can discover what's available in another repository (a fork, or an existing
+local checkout) before running `cds get` against it. Without these flags,
+`cds list` inspects the local project as before.
+
 ### Project defaults
 
 `cds config` manages the gitignored `.cds/config.json` file (or the path in
@@ -719,6 +794,8 @@ Common errors from `cds validate`, `cds plan`, and `cds render`, and how to fix
 | `[E041] ... Contract ref "x.y" points to unknown module "x"` | A `consumes` binding's `contractRef` refers to a module ID that isn't defined in the profile. | Check `spec.modules` for the correct module `id`, and confirm the contract ref follows `<module-id>.<contract-name>`. |
 | `[E041] ... but it does not provide "<contract-name>"` | The referenced module exists, but its `spec.provides` list doesn't expose that contract name. | Check the producing module's `module.yaml` for the contracts it actually provides, and fix the consumer's `contractRef` to match. |
 | `[E042] ... Contract kind mismatch` | The consumer expects one contract kind (e.g. `sql-database`) but the producer exposes a different kind. | Point the binding at a module that provides the expected contract kind, or update the consumer's expected kind if the mismatch is intentional. |
+| `[E103] ... config.image.tag is required ... when config.image.source is "registry"` | A module instance sets `config.image.source: registry` without also setting `config.image.tag`. | Set `config.image.tag` to a version, or switch back to `config.image.source: build`. |
+| `[W097] ... config.image.tag is "latest" with config.image.source "registry"` | `config.image.tag: latest` under `source: registry` still validates, but drifts silently between deploys instead of pinning a reproducible version. | Pin an explicit tag from `publish-images.yml`'s output (or `tests/fixtures/signed-images.json`) instead of `latest`. |
 
 All diagnostics print with their error code and YAML path (e.g. `spec.modules[1].config`), so search the profile file for that path to find the exact line to fix.
 

{composable_data_stack-0.7.0 → composable_data_stack-0.8.0}/cli/getter.py

@@ -18,6 +18,7 @@ from urllib.error import HTTPError, URLError
 from urllib.request import Request, urlopen
 
 from .loader import load_yaml_file, resolve_module_dir
+from .overlay import _derive_profiles_root, _resolve_extends_ref, resolve_extends
 from .planner import MaxNestingDepthExceeded, apply_defaults, substitute_string
 
 # The upstream repository `cds get` downloads from when no `--remote` is
@@ -71,7 +72,7 @@ def fetch_profile(
     destination_root: Path | None = None,
     force: bool = False,
     dry_run: bool = False,
-) -> tuple[list[CopyAction], Path]:
+) -> tuple[list[CopyAction], Path, list[str]]:
     target_root = (destination_root or Path.cwd()).expanduser().resolve()
 
     with _prepare_source_repository(remote, ref, local) as source_repo:
@@ -79,10 +80,15 @@ def fetch_profile(
         asset_roots = _collect_asset_roots(source_repo, profile_path)
 
         actions = _build_copy_plan(source_repo, asset_roots, target_root)
+        # Conflict detection must happen while `source_repo` (which may be a
+        # temporary directory holding a downloaded tarball) is still alive,
+        # since `_find_conflicts()` reads each action's source file to
+        # compare content. Compute it up front so both the dry-run and
+        # real-run paths can report/enforce conflicts consistently (#452).
+        conflicts = _find_conflicts(actions)
         if dry_run:
-            return actions, target_root / _TRACKING_FILE
+            return actions, target_root / _TRACKING_FILE, conflicts
 
-        conflicts = _find_conflicts(actions)
         if conflicts and not force:
             rendered = ", ".join(conflicts[:5])
             extra = "" if len(conflicts) <= 5 else f" (+{len(conflicts) - 5} more)"
@@ -104,18 +110,37 @@ def fetch_profile(
             actions=actions,
             asset_roots=asset_roots,
         )
-    return actions, target_root / _TRACKING_FILE
+    return actions, target_root / _TRACKING_FILE, conflicts
 
 
-def format_get_plan(actions: list[CopyAction], *, destination_root: Path) -> str:
+def format_get_plan(
+    actions: list[CopyAction],
+    *,
+    destination_root: Path,
+    conflicts: list[str] | None = None,
+) -> str:
+    conflicts = conflicts or []
+    conflict_set = set(conflicts)
+    clean_actions = [a for a in actions if a.repo_relative_path not in conflict_set]
+
     if not actions:
         return f"No file changes required under {destination_root}."
 
     lines = [f"Planned {len(actions)} file(s) under {destination_root}:"]
-    for action in actions[:20]:
-        lines.append(f"  - {action.repo_relative_path}")
-    if len(actions) > 20:
-        lines.append(f"  - ... {len(actions) - 20} more")
+    if clean_actions:
+        lines.append(f"  Would copy ({len(clean_actions)}):")
+        for action in clean_actions[:20]:
+            lines.append(f"    - {action.repo_relative_path}")
+        if len(clean_actions) > 20:
+            lines.append(f"    - ... {len(clean_actions) - 20} more")
+
+    if conflicts:
+        lines.append(f"  Would conflict without --force ({len(conflicts)}):")
+        for path in conflicts[:20]:
+            lines.append(f"    - {path}")
+        if len(conflicts) > 20:
+            lines.append(f"    - ... {len(conflicts) - 20} more")
+
     return "\n".join(lines)
 
 
@@ -233,15 +258,67 @@ def _resolve_source_profile_path(source_repo: Path, profile: str) -> Path:
     )
 
 
+def _collect_extends_profile_dirs(
+    source_repo: Path, profile_path: Path, _visited: frozenset[Path] = frozenset()
+) -> set[Path]:
+    """
+    Returns every parent profile directory reachable via `extends`, so
+    `cds get` copies parent profile.yaml files (and, via
+    _collect_asset_roots's module walk, their modules) too instead of
+    silently omitting anything only declared in a parent profile. Reuses
+    cli.overlay's own extends-ref resolution rather than re-implementing it.
+
+    `_visited` tracks resolved profile paths already seen along the current
+    extends chain so a cyclic `extends` graph raises a clean GetError
+    instead of recursing until Python's recursion limit is hit; this
+    mirrors cli.overlay._compose_extends's own cycle-detection stack.
+    """
+    resolved_profile_path = profile_path.resolve()
+    if resolved_profile_path in _visited:
+        raise GetError(f"Cycle detected in extends chain at {profile_path}")
+    _visited = _visited | {resolved_profile_path}
+
+    doc, _diagnostics = load_yaml_file(profile_path)
+    if doc is None:
+        raise GetError(f"Could not load source profile {profile_path}")
+
+    extends = doc.get("extends")
+    if not extends:
+        return set()
+
+    profile_dir = profile_path.parent.resolve()
+    profiles_root = _derive_profiles_root(profile_dir)
+    if profiles_root is None:
+        raise GetError(f'Could not derive a profiles root for extends in {profile_path}')
+
+    dirs: set[Path] = set()
+    for ref in extends:
+        parent_path = _resolve_extends_ref(ref, profile_dir, profiles_root)
+        if not parent_path.is_file():
+            raise GetError(f'Could not resolve extends parent "{ref}" for {profile_path}')
+        _require_within_repo(parent_path, source_repo, f'extends parent "{ref}"')
+        dirs.add(parent_path.parent)
+        dirs.update(_collect_extends_profile_dirs(source_repo, parent_path, _visited))
+
+    return dirs
+
+
 def _collect_asset_roots(source_repo: Path, profile_path: Path) -> list[Path]:
     profile_dir = profile_path.parent
     asset_roots: set[Path] = {
         profile_dir if profile_path.name == "profile.yaml" else profile_path
     }
-    profile_doc, profile_diags = load_yaml_file(profile_path)
-    if profile_diags or profile_doc is None:
+
+    # Resolve through `extends` so modules/config contributed only by a
+    # parent profile (not redeclared in the child) are still collected;
+    # cli.overlay is the single source of truth for extends semantics (see
+    # cli.planner/cli.validator, which route through it the same way).
+    profile_doc, _provenance, diagnostics = resolve_extends(str(profile_path))
+    if profile_doc is None or any(d.level == "error" for d in diagnostics):
         raise GetError(f"Could not load source profile {profile_path}")
 
+    asset_roots.update(_collect_extends_profile_dirs(source_repo, profile_path))
+
     spec = profile_doc.get("spec")
     modules = spec.get("modules", []) if isinstance(spec, dict) else []
     if not isinstance(modules, list):
@@ -595,7 +672,19 @@ def _add_copy_action(
     destination_root: Path,
     actions_by_destination: dict[Path, CopyAction],
 ) -> None:
-    repo_relative = source_file.resolve().relative_to(source_repo.resolve())
+    resolved_source = source_file.resolve()
+    try:
+        repo_relative = resolved_source.relative_to(source_repo.resolve())
+    except ValueError as exc:
+        # A symlink (or directory junction) planted under an asset root can
+        # resolve to a real path outside the source repository. Without this
+        # guard, Path.relative_to() raises a bare ValueError that escapes as
+        # an unhandled traceback instead of the stable GetError callers rely
+        # on (#454).
+        raise GetError(
+            f"Asset {source_file} resolves outside the source repository "
+            f"({source_repo}): {resolved_source}"
+        ) from exc
     # Deliberately do NOT call .resolve() on the combined destination path:
     # destination_root is already an absolute, resolved path (see
     # fetch_profile()), and resolving the full path here would follow a
@@ -607,12 +696,12 @@ def _add_copy_action(
     existing = actions_by_destination.get(destination)
     if existing is None:
         actions_by_destination[destination] = CopyAction(
-            source=source_file.resolve(),
+            source=resolved_source,
             destination=destination,
             repo_relative_path=repo_relative.as_posix(),
         )
         return
-    if existing.source.resolve() != source_file.resolve():
+    if existing.source.resolve() != resolved_source:
         raise GetError(
             f"Multiple source files would map to the same destination: {destination}"
         )
@@ -649,10 +738,17 @@ def _write_actions(actions: list[CopyAction]) -> None:
     (e.g. a symlink planted at profiles/foo/profile.yaml pointing outside
     the destination tree)."""
     for action in actions:
-        action.destination.parent.mkdir(parents=True, exist_ok=True)
-        if action.destination.is_symlink():
-            action.destination.unlink()
-        shutil.copy2(action.source, action.destination)
+        try:
+            action.destination.parent.mkdir(parents=True, exist_ok=True)
+            if action.destination.is_symlink():
+                action.destination.unlink()
+            shutil.copy2(action.source, action.destination)
+        except OSError as exc:
+            # Surface filesystem failures (permission denied, read-only
+            # destination, disk full, ...) as a stable GetError instead of a
+            # raw OSError, so callers get a predictable error path and
+            # message regardless of the underlying platform/errno (#454).
+            raise GetError(f"Could not write {action.destination}: {exc}") from exc
 
 
 def _write_tracking_manifest(