composable-data-stack 0.10.0__tar.gz → 0.12.0__tar.gz

{composable_data_stack-0.10.0/composable_data_stack.egg-info → composable_data_stack-0.12.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: composable-data-stack
-Version: 0.10.0
+Version: 0.12.0
 Summary: CLI for validating and composing composable data stack profiles
 Author: Ronald Hensbergen
 License-Expression: Apache-2.0
@@ -32,7 +32,7 @@ Provides-Extra: dev
 Requires-Dist: coverage>=7.6; extra == "dev"
 Requires-Dist: bandit>=1.7; extra == "dev"
 Requires-Dist: pip-audit>=2.7; extra == "dev"
-Requires-Dist: ruff==0.16.9; extra == "dev"
+Requires-Dist: ruff==0.17.0; extra == "dev"
 Dynamic: license-file
 
 # 🚀 Composable Data Stack (CDS)
@@ -770,89 +770,11 @@ runs, with a dedicated diagnostic code:
 
 ## ⚙️ CLI
 
-|Command|Description|
-|---|---|
-|cds get \<profile\> [--remote \<owner/repo\>] [--ref \<ref\>] [--local \<dir\>] [--into \<dir\>]|Fetch a profile plus its dependent module/runtime assets from GitHub into a local CDS layout|
-|cds list profiles\|modules\|images [--remote \<owner/repo\>] [--ref \<ref\>] [--local \<dir\>]|List available profiles, module sources, or module images and check for newer versions; add `--remote`/`--local` to inspect another repository before fetching from it|
-|cds init [profile]|Generate a project `.env` template from profile secret definitions|
-|cds validate [profile]|Validate modules and contracts; use `--target helm` for Kubernetes checks|
-|cds preflight [profile]|Check runtime tools, required environment values, and host ports without starting services|
-|cds plan [profile]|Resolve dependencies and generate an execution plan|
-|cds render [profile]|Generate Docker Compose or a Helm chart from a resolved plan|
-|cds up [profile]|Validate, plan, render, and start the Compose or Helm target; use `--target helm` for Kubernetes|
-|cds down [profile]|Stop Compose or uninstall a Helm release; Helm PVCs are retained by default|
-|cds state [profile]|Show Compose services or Kubernetes workloads grouped by health|
-|cds test [profile]|One-shot smoke validation: validate, security, plan, and render|
-|cds security [profile]|Run rule-based security validation on a profile|
-|cds diff [profile] --from \<env\> --to \<env\>|Show effective configuration differences between two environment overlays, secrets never included|
-|cds use [profile] [--clear]|Save (show/clear) a default profile so it doesn't have to be passed to other commands|
-|cds config get\|set\|unset\|list|Manage persisted project defaults in `.cds/config.json`|
-|cds completion \<bash\|zsh\|powershell\>|Print shell setup instructions for tab-completion|
-
-`init`, `validate`, `preflight`, `plan`, `render`, `up`, `test`, and `security`
-all accept `--environment <name>` (or `-e <name>`) to merge
-`environments/<name>.yaml` over the base profile before resolving; see
-[Environment Overlays](#environment-overlays).
-
-`cds get` copies the selected `profiles/<name>/` tree, every referenced module
-directory, and any local build-context assets referenced by those modules'
-Dockerfiles. By design it downloads from GitHub rather than a local checkout:
-by default it fetches this project's upstream repository at the `main` branch,
-downloading a tarball via the GitHub API (no `git` binary required). Use
-`--remote <owner/repo>` (or a `github.com/...` URL) to fetch a fork, and
-`--ref <branch|tag|sha>` to select a specific revision. Pass `--local <dir>`
-to use an existing local directory instead of downloading (mutually exclusive
-with `--remote`/`--ref`) for offline/dev workflows. Use `--into` to choose a
-destination root, `--dry-run` to inspect the copy plan first, and `--force` to
-replace conflicting local files. Successful fetches record tracking metadata
-in `.cds/get-manifest.json` for future update workflows.
-
-`cds list profiles`, `cds list modules`, and `cds list images` accept the same
-`--remote`/`--ref`/`--local` source-repository selection as `cds get`, so you
-can discover what's available in another repository (a fork, or an existing
-local checkout) before running `cds get` against it. Without these flags,
-`cds list` inspects the local project as before.
-
-### Project defaults
-
-`cds config` manages the gitignored `.cds/config.json` file (or the path in
-`CDS_CONFIG_PATH`). Supported settings are `profile`, `environment`, and
-`security.strict`:
-
-```bash
-cds config set profile my-profile
-cds config set environment prod
-cds config set security.strict true
-cds config list
-```
-
-`profile` is stored as its resolved path, and `environment` is validated
-against that profile's `environments/` directory. `security.strict true`
-applies the existing production security rules even if the profile declares a
-local environment. `cds use` remains a shortcut for setting, showing, or
-clearing `profile`.
-
-CLI flags take precedence over these defaults. In particular,
-`--environment` overrides `config environment`; `CDS_PROFILE_PATH` continues
-to override the saved profile.
-
-`[profile]` accepts:
-
-| Form | Example |
-| ---- | ------- |
-| Profile name | `local-dagster-postgres-superset` |
-| Path to a `profile.yaml` file | `profiles/local-dagster-postgres-superset/profile.yaml` |
-| Path to a profiles root directory | `profiles/` |
-
-When `[profile]` is omitted, resolution falls back in order to: `CDS_PROFILE_PATH` if set (accepts the same three forms), then the default profile saved via `cds config set profile` (or `cds use <profile>`), then the single profile under `profiles/` if there is exactly one. An explicitly-set env var takes precedence over the persisted project default, matching common CLI convention (env vars are per-invocation and reflect the current session more reliably than a saved, gitignored default that's easy to forget about).
-
-To view the full list of options for any command, use the `--help` flag:
-
-```bash
-cds --help
-cds validate --help
-cds plan --help
-```
+See [docs/cli-reference.md](docs/cli-reference.md) for the full command
+table, flag reference, and `[profile]` resolution rules. The commands used in
+the [Quickstart](#-quickstart) above are `cds init`, `cds validate`,
+`cds preflight`, `cds security`, `cds plan`, `cds render`, and `cds up`; run
+`cds --help` or `cds <command> --help` for inline usage at any time.
 
 ---
 
@@ -1034,10 +956,9 @@ Please read these first:
 
 - [CONTRIBUTING.md](CONTRIBUTING.md)
 - [docs/maintainer-merge-policy.md](docs/maintainer-merge-policy.md)
-- [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md)
-- [SECURITY.md](SECURITY.md)
-- [SUPPORT.md](SUPPORT.md)
-- [docs/security-support-policy.md](docs/security-support-policy.md)
+- [SECURITY.md](SECURITY.md) — how to report a vulnerability
+- [LEGAL.md](LEGAL.md) — license, code of conduct, security, support, and
+  regulatory-compliance documentation index
 - [CHANGELOG.md](CHANGELOG.md)
 - [RELEASE.md](RELEASE.md)
 
@@ -1049,6 +970,7 @@ Development helper tools are located in the gitignored `tools/` directory. See
 ## 📖 Documentation
 
 - [Quickstart](README.md#-quickstart) — get running in 5 minutes
+- [CLI Reference](docs/cli-reference.md) — full command/flag table and `[profile]` resolution rules
 - [How CDS Works](docs/how-it-works.md) — C4 + sequence diagrams of the validate → plan → render engine
 - [Kubernetes Target](docs/kubernetes.md): Helm rendering and isolated local k3s workflow
 - [From Docker Compose to CDS Profile](docs/from-docker-to-cds-profile.md) — complete transformation guide
@@ -1060,4 +982,5 @@ Development helper tools are located in the gitignored `tools/` directory. See
 
 ## 📜 License
 
-See `LICENSE`.
+See `LICENSE`. For security, support, and regulatory-compliance
+documentation, see [LEGAL.md](LEGAL.md).

{composable_data_stack-0.10.0 → composable_data_stack-0.12.0}/README.md

@@ -733,89 +733,11 @@ runs, with a dedicated diagnostic code:
 
 ## ⚙️ CLI
 
-|Command|Description|
-|---|---|
-|cds get \<profile\> [--remote \<owner/repo\>] [--ref \<ref\>] [--local \<dir\>] [--into \<dir\>]|Fetch a profile plus its dependent module/runtime assets from GitHub into a local CDS layout|
-|cds list profiles\|modules\|images [--remote \<owner/repo\>] [--ref \<ref\>] [--local \<dir\>]|List available profiles, module sources, or module images and check for newer versions; add `--remote`/`--local` to inspect another repository before fetching from it|
-|cds init [profile]|Generate a project `.env` template from profile secret definitions|
-|cds validate [profile]|Validate modules and contracts; use `--target helm` for Kubernetes checks|
-|cds preflight [profile]|Check runtime tools, required environment values, and host ports without starting services|
-|cds plan [profile]|Resolve dependencies and generate an execution plan|
-|cds render [profile]|Generate Docker Compose or a Helm chart from a resolved plan|
-|cds up [profile]|Validate, plan, render, and start the Compose or Helm target; use `--target helm` for Kubernetes|
-|cds down [profile]|Stop Compose or uninstall a Helm release; Helm PVCs are retained by default|
-|cds state [profile]|Show Compose services or Kubernetes workloads grouped by health|
-|cds test [profile]|One-shot smoke validation: validate, security, plan, and render|
-|cds security [profile]|Run rule-based security validation on a profile|
-|cds diff [profile] --from \<env\> --to \<env\>|Show effective configuration differences between two environment overlays, secrets never included|
-|cds use [profile] [--clear]|Save (show/clear) a default profile so it doesn't have to be passed to other commands|
-|cds config get\|set\|unset\|list|Manage persisted project defaults in `.cds/config.json`|
-|cds completion \<bash\|zsh\|powershell\>|Print shell setup instructions for tab-completion|
-
-`init`, `validate`, `preflight`, `plan`, `render`, `up`, `test`, and `security`
-all accept `--environment <name>` (or `-e <name>`) to merge
-`environments/<name>.yaml` over the base profile before resolving; see
-[Environment Overlays](#environment-overlays).
-
-`cds get` copies the selected `profiles/<name>/` tree, every referenced module
-directory, and any local build-context assets referenced by those modules'
-Dockerfiles. By design it downloads from GitHub rather than a local checkout:
-by default it fetches this project's upstream repository at the `main` branch,
-downloading a tarball via the GitHub API (no `git` binary required). Use
-`--remote <owner/repo>` (or a `github.com/...` URL) to fetch a fork, and
-`--ref <branch|tag|sha>` to select a specific revision. Pass `--local <dir>`
-to use an existing local directory instead of downloading (mutually exclusive
-with `--remote`/`--ref`) for offline/dev workflows. Use `--into` to choose a
-destination root, `--dry-run` to inspect the copy plan first, and `--force` to
-replace conflicting local files. Successful fetches record tracking metadata
-in `.cds/get-manifest.json` for future update workflows.
-
-`cds list profiles`, `cds list modules`, and `cds list images` accept the same
-`--remote`/`--ref`/`--local` source-repository selection as `cds get`, so you
-can discover what's available in another repository (a fork, or an existing
-local checkout) before running `cds get` against it. Without these flags,
-`cds list` inspects the local project as before.
-
-### Project defaults
-
-`cds config` manages the gitignored `.cds/config.json` file (or the path in
-`CDS_CONFIG_PATH`). Supported settings are `profile`, `environment`, and
-`security.strict`:
-
-```bash
-cds config set profile my-profile
-cds config set environment prod
-cds config set security.strict true
-cds config list
-```
-
-`profile` is stored as its resolved path, and `environment` is validated
-against that profile's `environments/` directory. `security.strict true`
-applies the existing production security rules even if the profile declares a
-local environment. `cds use` remains a shortcut for setting, showing, or
-clearing `profile`.
-
-CLI flags take precedence over these defaults. In particular,
-`--environment` overrides `config environment`; `CDS_PROFILE_PATH` continues
-to override the saved profile.
-
-`[profile]` accepts:
-
-| Form | Example |
-| ---- | ------- |
-| Profile name | `local-dagster-postgres-superset` |
-| Path to a `profile.yaml` file | `profiles/local-dagster-postgres-superset/profile.yaml` |
-| Path to a profiles root directory | `profiles/` |
-
-When `[profile]` is omitted, resolution falls back in order to: `CDS_PROFILE_PATH` if set (accepts the same three forms), then the default profile saved via `cds config set profile` (or `cds use <profile>`), then the single profile under `profiles/` if there is exactly one. An explicitly-set env var takes precedence over the persisted project default, matching common CLI convention (env vars are per-invocation and reflect the current session more reliably than a saved, gitignored default that's easy to forget about).
-
-To view the full list of options for any command, use the `--help` flag:
-
-```bash
-cds --help
-cds validate --help
-cds plan --help
-```
+See [docs/cli-reference.md](docs/cli-reference.md) for the full command
+table, flag reference, and `[profile]` resolution rules. The commands used in
+the [Quickstart](#-quickstart) above are `cds init`, `cds validate`,
+`cds preflight`, `cds security`, `cds plan`, `cds render`, and `cds up`; run
+`cds --help` or `cds <command> --help` for inline usage at any time.
 
 ---
 
@@ -997,10 +919,9 @@ Please read these first:
 
 - [CONTRIBUTING.md](CONTRIBUTING.md)
 - [docs/maintainer-merge-policy.md](docs/maintainer-merge-policy.md)
-- [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md)
-- [SECURITY.md](SECURITY.md)
-- [SUPPORT.md](SUPPORT.md)
-- [docs/security-support-policy.md](docs/security-support-policy.md)
+- [SECURITY.md](SECURITY.md) — how to report a vulnerability
+- [LEGAL.md](LEGAL.md) — license, code of conduct, security, support, and
+  regulatory-compliance documentation index
 - [CHANGELOG.md](CHANGELOG.md)
 - [RELEASE.md](RELEASE.md)
 
@@ -1012,6 +933,7 @@ Development helper tools are located in the gitignored `tools/` directory. See
 ## 📖 Documentation
 
 - [Quickstart](README.md#-quickstart) — get running in 5 minutes
+- [CLI Reference](docs/cli-reference.md) — full command/flag table and `[profile]` resolution rules
 - [How CDS Works](docs/how-it-works.md) — C4 + sequence diagrams of the validate → plan → render engine
 - [Kubernetes Target](docs/kubernetes.md): Helm rendering and isolated local k3s workflow
 - [From Docker Compose to CDS Profile](docs/from-docker-to-cds-profile.md) — complete transformation guide
@@ -1023,4 +945,5 @@ Development helper tools are located in the gitignored `tools/` directory. See
 
 ## 📜 License
 
-See `LICENSE`.
+See `LICENSE`. For security, support, and regulatory-compliance
+documentation, see [LEGAL.md](LEGAL.md).

composable_data_stack-0.12.0/cli/audit_log.py

@@ -0,0 +1,155 @@
+"""
+Local, append-only audit trail of rendered/applied stacks (#737).
+
+This is distinct from `cli/getter.py`'s `.cds/get-manifest.json`, which only
+tracks file provenance for `cds get`. This module instead records *what was
+run* for a profile over time: timestamp, profile identity, resolved module
+versions/image references, the command invoked, and its outcome. Users who
+are themselves subject to incident-reporting obligations (e.g.
+NIS2/Cyberbeveiligingswet-style regimes) can use this to reconstruct what
+was running and when, without relying on ad hoc memory or external
+infrastructure logs.
+
+The log is strictly local evidence for the user's own purposes: nothing
+here is sent to CDS or any third party. Only secret *aliases* are ever
+recorded (e.g. "CDS_DB_PASSWORD"), the same `${CDS_*}` placeholder
+convention used by the planner/renderer (see `cli/secrets.py`) -- never
+secret values.
+"""
+from __future__ import annotations
+
+import json
+import os
+import sys
+from datetime import UTC, datetime
+from pathlib import Path
+from typing import Any
+
+_LOG_FILE = Path(".cds") / "audit-log.jsonl"
+
+# Set to disable the audit log without touching project config, e.g. for a
+# single invocation or a CI job that should not write local evidence files.
+_ENV_DISABLE = "CDS_AUDIT_LOG_DISABLE"
+_TRUTHY = {"1", "true", "yes", "on"}
+
+
+def audit_log_path(project_root: Path | None = None) -> Path:
+    """Return the path audit entries are appended to for `project_root`.
+
+    No `cli.main` call site currently passes an explicit `project_root`, so
+    this defaults to `Path.cwd()` rather than the `resolve_project_root
+    (profile_path)` convention used elsewhere (e.g. for rendered Compose
+    output). That means audit entries land in whatever directory the user
+    happened to invoke `cds` from, not necessarily the profile's own
+    project root (e.g. when `CDS_PROFILE_PATH` points at a separate
+    profiles checkout). Tracked by #822.
+    """
+    root = (project_root or Path.cwd()).expanduser().resolve()
+    return root / _LOG_FILE
+
+
+def is_enabled(*, config_disabled: bool = False) -> bool:
+    """Whether audit logging is active.
+
+    Disabled if the project config has `audit.enabled` set to `false`
+    (`config_disabled=True`), or if `CDS_AUDIT_LOG_DISABLE` is truthy.
+    """
+    if config_disabled:
+        return False
+    return os.environ.get(_ENV_DISABLE, "").strip().lower() not in _TRUTHY
+
+
+def _module_summary(plan: dict[str, Any] | None) -> list[dict[str, Any]]:
+    """Resolved module id/source/version and image reference fields.
+
+    Only reference-shaped image fields (repository/tag/digest/variant) are
+    included -- never the full module config, which could carry
+    `secrets.<alias>` references or other user-provided values.
+    """
+    if not isinstance(plan, dict):
+        return []
+
+    summary: list[dict[str, Any]] = []
+    for module in plan.get("modules", []):
+        if not isinstance(module, dict):
+            continue
+        entry: dict[str, Any] = {
+            "id": module.get("id"),
+            "source": module.get("source"),
+            "version": module.get("version"),
+        }
+        config = module.get("config")
+        image_config = config.get("image") if isinstance(config, dict) else None
+        if isinstance(image_config, dict):
+            image_ref = {
+                key: image_config[key]
+                for key in ("repository", "tag", "digest", "variant")
+                if key in image_config
+            }
+            if image_ref:
+                entry["image"] = image_ref
+        summary.append(entry)
+    return summary
+
+
+def _secret_aliases(plan: dict[str, Any] | None) -> list[str]:
+    """Secret alias names referenced by the plan, never their values.
+
+    `plan["secrets"]` already maps alias/env names to `CDS_*` env var
+    *names* (see `cli/secrets.py:load_profile_secrets`) -- no secret value
+    ever passes through the planner, so this is safe to record verbatim.
+    """
+    if not isinstance(plan, dict):
+        return []
+    secrets = plan.get("secrets")
+    if not isinstance(secrets, dict):
+        return []
+    return sorted(secrets.keys())
+
+
+def record_entry(
+    *,
+    command: str,
+    profile: str,
+    outcome: str,
+    environment: str | None = None,
+    plan: dict[str, Any] | None = None,
+    details: dict[str, Any] | None = None,
+    project_root: Path | None = None,
+    config_disabled: bool = False,
+) -> Path | None:
+    """Append one JSON Lines entry describing a render/apply attempt.
+
+    Returns the log path written to, or `None` if audit logging is
+    disabled. Never raises: a failure to write the audit log must not
+    block the primary command it is observing.
+    """
+    if not is_enabled(config_disabled=config_disabled):
+        return None
+
+    log_path = audit_log_path(project_root)
+    entry: dict[str, Any] = {
+        "timestamp": datetime.now(UTC).isoformat(),
+        "command": command,
+        "profile": profile,
+        "environment": environment,
+        "outcome": outcome,
+        "modules": _module_summary(plan),
+        "secretAliases": _secret_aliases(plan),
+    }
+    if details:
+        entry["details"] = details
+
+    try:
+        log_path.parent.mkdir(parents=True, exist_ok=True)
+        if log_path.is_symlink():
+            # Never follow/overwrite-through a symlink planted at the log
+            # path (same guard as the get-manifest writer in getter.py).
+            log_path.unlink()
+        with open(log_path, "a", encoding="utf-8") as f:
+            f.write(json.dumps(entry, sort_keys=True) + "\n")
+    except OSError as exc:
+        print(f"WARNING Could not write audit log entry to {log_path}: {exc}", file=sys.stderr)
+        return None
+
+    return log_path