datasecops-cli 0.7.0b1__tar.gz → 0.7.0b2__tar.gz

{datasecops_cli-0.7.0b1 → datasecops_cli-0.7.0b2}/.gitignore

@@ -219,4 +219,5 @@ __marimo__/
 
 
 legacy_framework/
-plans/
+plans/
+.cortex/

{datasecops_cli-0.7.0b1 → datasecops_cli-0.7.0b2}/CHANGELOG.md

@@ -4,6 +4,22 @@ All notable changes to the DataSecOps CLI are documented in this file.
 
 ## [Unreleased]
 
+### Fixed
+
+- **Default app database renamed to `MONITORIAL_ORBIT`** — the `datasecops setup` suggested app-database name (and matching docs examples) was `DATA_ENGINEERS_DATASECOPS_FRAMEWORK`; it now matches the native app's actual installed name.
+- **`dbt_project.yml` generator matches the source-based project spec** — added `docs-paths`, fixed `analysis-paths` (was `analyses`), added top-level `seeds:`/`tests:` blocks, `+static_analysis`/`+meta`/`+tags` at the model root, per-object-type `+static_analysis` (stored procedures/UDFs off, tasks baseline), the commented-out post-hook/on-run-start/on-run-end guidance, and the data-governance `vars` block.
+- **`profiles.yml` deploy targets now run as the framework's CI identity** — every target except `local-dev` previously used the local developer's Snowflake user with each target's own role/warehouse. They now use the CI Username/Role/Warehouse from the native app's Deployment settings (new `ProjectSettings.ci_username`/`ci_role`/`ci_warehouse`, falling back to the previous behavior when unset), matching what the CI pipeline actually deploys with.
+- **`datasecops setup` could silently default to GitHub for a non-GitHub platform** — `PROJECT`/`SOURCE_CONTROL`/`WORK_ITEMS` framework config and profile/`profiles.yml` resolution were one single `try/except`; any failure after connecting (including just the `SOURCE_CONTROL` fetch itself) discarded already-fetched config and silently fell back to `SourceControl()`'s default platform (`GitHub`) with no indication why, so the MCP setup step could prompt for the wrong PAT. Each framework config fetch is now independent, so one failing can't blank out another; a missing/failed `SOURCE_CONTROL` config, or one missing `source_control_platform` specifically, now warns explicitly which platform it's defaulting to.
+- **`local-dev` could be overwritten by a same-named framework target, or by the CI identity on a plain `dbt run`** — `generate_profiles_yml` no longer lets a framework target literally named `local-dev` clobber the dedicated external-browser output, and the profile's default target is now always `local-dev` (previously it followed whatever the native app marked as its own default, which could be a deploy-only target running as the CI identity). `_run_interactive` and bootstrap's `dbt deps` step now always run as `local-dev` instead of trusting the framework's default target.
+- **Re-running `datasecops setup` for an already-known profile never refreshed its deployment targets** — a CI identity change in the native app had no effect until `profiles.yml` was deleted. Setup now always refreshes the CI/deployment targets (while preserving an existing `local-dev` target if already customized), and skips writing `profiles.yml` entirely if the `PROJECT` framework config couldn't be loaded (previously it wrote incorrect defaults).
+- **Hardened `dbt_project.yml` generation against unsafe characters and a YAML boolean-coercion bug** — project/profile name, dbt version and warehouse name are now validated against a safe identifier pattern before being substituted into the raw YAML template (closing a potential injection via a newline or quote in native-app-configured values), and `static_analysis: off` is now quoted so YAML doesn't resolve it to the boolean `false`.
+
+### Docs
+
+- **Documented setting up `~/.dbt/profiles.yml`** — `datasecops bootstrap` writes a generated `profiles.yml` inside the dbt project folder as a reference only; dbt itself reads from `~/.dbt` (per the native app's `profile_dir` setting), which getting-started.md now explains, including the minimal `local-dev` block a developer needs there.
+
+## [0.7.0b1] - 2026-10-26
+
 ### Added
 
 - **Shared DCM model (`datasecops_model`)** — one versioned JSON document (`config/dcm_model.json`, schema `datasecops.dcm/v1`) describing every DCM project (type, dir, `depends_on`, targets with deploy roles), the security model (role groups with environment fan-out, SCIM/UGE prefixes, diagram layout) and the warehouses (the `warehouse()` macro arguments, grants, application grants, budgets). The same package is vendored into the DataSecOps native app (`scripts/vendor_model.py`), so the app and the CLI validate, render and draw identically.

{datasecops_cli-0.7.0b1 → datasecops_cli-0.7.0b2}/DEVELOPMENT.md

@@ -38,7 +38,7 @@ chmod +x setup.sh
 The script will prompt you for:
 
 - **Connection name** — the name of your Snowflake connection from `~/.snowflake/connections.toml`
-- **Native app database name** — the database where the DataSecOps Framework app is installed (e.g. `DATA_ENGINEERS_DATASECOPS_FRAMEWORK`)
+- **Native app database name** — the database where the DataSecOps Framework app is installed (e.g. `MONITORIAL_ORBIT`)
 
 ### 2. Activate the virtual environment
 
@@ -120,7 +120,7 @@ Created by the setup script in your project root. Contains:
 
 ```yaml
 connection_name: "my_connection"    # Snowflake connection name
-app_database: "DATA_ENGINEERS_DATASECOPS_FRAMEWORK"  # Native app database
+app_database: "MONITORIAL_ORBIT"  # Native app database
 profile_name: ""                    # Auto-detected from dbt_project.yml
 ```
 

{datasecops_cli-0.7.0b1 → datasecops_cli-0.7.0b2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: datasecops-cli
-Version: 0.7.0b1
+Version: 0.7.0b2
 Summary: DataSecOps Framework CLI for Snowflake Native App
 License-Expression: MIT
 License-File: LICENSE
@@ -120,7 +120,7 @@ Your pipeline needs two things:
 
    ```yaml
    connection_name: "ci"
-   app_database: "DATA_ENGINEERS_DATASECOPS_FRAMEWORK"
+   app_database: "MONITORIAL_ORBIT"
    ```
 
 2. **A Snowflake connection** in `~/.snowflake/connections.toml` for the CI service account:
@@ -264,7 +264,7 @@ The CLI reads from a `.datasecops.yml` file in your project root (created by the
 
 ```yaml
 connection_name: "my_connection"
-app_database: "DATA_ENGINEERS_DATASECOPS_FRAMEWORK"
+app_database: "MONITORIAL_ORBIT"
 ```
 
 Project profiles, linting rules, pipeline templates, and deployment targets are all managed centrally in the native app and pulled down by the CLI.

{datasecops_cli-0.7.0b1 → datasecops_cli-0.7.0b2}/README.md

@@ -100,7 +100,7 @@ Your pipeline needs two things:
 
    ```yaml
    connection_name: "ci"
-   app_database: "DATA_ENGINEERS_DATASECOPS_FRAMEWORK"
+   app_database: "MONITORIAL_ORBIT"
    ```
 
 2. **A Snowflake connection** in `~/.snowflake/connections.toml` for the CI service account:
@@ -244,7 +244,7 @@ The CLI reads from a `.datasecops.yml` file in your project root (created by the
 
 ```yaml
 connection_name: "my_connection"
-app_database: "DATA_ENGINEERS_DATASECOPS_FRAMEWORK"
+app_database: "MONITORIAL_ORBIT"
 ```
 
 Project profiles, linting rules, pipeline templates, and deployment targets are all managed centrally in the native app and pulled down by the CLI.

{datasecops_cli-0.7.0b1 → datasecops_cli-0.7.0b2}/docs/getting-started.md

@@ -59,7 +59,7 @@ This creates `.datasecops.yml` in your project root:
 
 ```yaml
 connection_name: "my_connection"
-app_database: "DATA_ENGINEERS_DATASECOPS_FRAMEWORK"
+app_database: "MONITORIAL_ORBIT"
 ```
 
 If you prefer to create it manually, the two required fields are:
@@ -91,6 +91,43 @@ You should see the main menu, confirming the CLI can reach the native app.
 
 ---
 
+## Step 3b: Set up your local dbt profile (`~/.dbt/profiles.yml`)
+
+Running `datasecops bootstrap` (or **Configuration → new project** from the main menu) scaffolds a
+dbt project from the native app's framework config, including a generated `dbt_project.yml` and
+`profiles.yml`. The generated `profiles.yml` is written *inside* the dbt project folder (e.g.
+`./dbt/profiles.yml`) as a reference — it's gitignored and never committed.
+
+dbt itself does **not** read that file. It reads whatever `profiles.yml` lives in the directory set
+by `--profiles-dir`, which the CLI passes as the native app's `PROJECT.profile_dir` setting
+(defaults to `~/.dbt`). So you need to copy the generated `local-dev` target into your own
+`~/.dbt/profiles.yml` (create the file/folder if they don't exist yet):
+
+```yaml
+<profile_name>:          # matches the `profile:` field in dbt_project.yml
+  target: local-dev
+  outputs:
+    local-dev:
+      type: snowflake
+      account: "<your_account>"
+      authenticator: externalbrowser   # SSO login, no password/PAT needed
+      role: DEVELOPERS
+      warehouse: DEV_WH
+      database: "<profile's target database>"
+      schema: PUBLIC
+      threads: 4
+```
+
+You only need the `local-dev` target locally. The other targets in the generated `profiles.yml`
+(e.g. `dev`, `test`, `prod`) run as the framework's CI/deployment identity (the CI Username, Role
+and Warehouse configured in the native app's Deployment settings) — they're what the CI pipeline
+deploys with, not something a developer runs against directly.
+
+If you work on multiple framework projects, each profile (one per project, named after its
+`profile:` field) gets its own block in the same `~/.dbt/profiles.yml` file.
+
+---
+
 ## Step 4: Install MCP servers
 
 ### 4a. DataSecOps Framework MCP Server (included)

{datasecops_cli-0.7.0b1 → datasecops_cli-0.7.0b2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "datasecops-cli"
-version = "0.7.0b1"
+version = "0.7.0b2"
 description = "DataSecOps Framework CLI for Snowflake Native App"
 readme = "README.md"
 requires-python = ">=3.10"

datasecops_cli-0.7.0b2/src/datasecops_cli/__init__.py

@@ -0,0 +1 @@
+__version__ = "0.7.0b2"

{datasecops_cli-0.7.0b1 → datasecops_cli-0.7.0b2}/src/datasecops_cli/main.py

@@ -475,7 +475,7 @@ def _run_setup(project_dir: Path):
     # --- App database ---
     app_database = get_input_string_or_default(
         "Enter native app database name",
-        "DATA_ENGINEERS_DATASECOPS_FRAMEWORK"
+        "MONITORIAL_ORBIT"
     )
 
     # --- CoCo agent connection ---
@@ -496,104 +496,146 @@ def _run_setup(project_dir: Path):
     source_control = SourceControl()
     work_items = WorkItems()
     matched_profile = None
+    account_identifier = ""
+    project_settings_loaded = False
 
+    connected = False
     try:
         temp_sf.connect()
         success_line("Connected")
+        connected = True
+    except Exception as e:
+        warning_line(f"Could not connect to Snowflake: {e}")
+        warning_line("You can set profile_name manually in .datasecops.yml")
+        warning_line("You may need to create profiles.yml manually")
 
-        # Load framework PROJECT settings (for profile_dir, targets)
-        raw_project = temp_sf.get_framework_config("PROJECT")
-        if raw_project:
-            project_settings = ProjectSettings(**{k: v for k, v in raw_project.items() if k in ProjectSettings.model_fields})
+    profile_name = ""
+    if connected:
+        # Each framework config code is fetched independently: a failure (or a missing
+        # config row) must not discard config that was already fetched successfully, and
+        # must not silently fall back to a default (e.g. GitHub) without saying so.
+        try:
+            raw_project = temp_sf.get_framework_config("PROJECT")
+            if raw_project:
+                project_settings = ProjectSettings(**{k: v for k, v in raw_project.items() if k in ProjectSettings.model_fields})
+                project_settings_loaded = True
+            else:
+                warning_line("No PROJECT framework config found in native app - using defaults")
+        except Exception as e:
+            warning_line(f"Could not load PROJECT framework config: {e}")
 
-        # Load SOURCE_CONTROL settings (for source_control_platform)
-        raw_sc = temp_sf.get_framework_config("SOURCE_CONTROL")
-        if raw_sc:
-            source_control = SourceControl(**{k: v for k, v in raw_sc.items() if k in SourceControl.model_fields})
+        try:
+            raw_sc = temp_sf.get_framework_config("SOURCE_CONTROL")
+            if raw_sc:
+                if "source_control_platform" not in raw_sc:
+                    warning_line(f"SOURCE_CONTROL framework config has no source_control_platform - "
+                                 f"defaulting to {source_control.source_control_platform}")
+                source_control = SourceControl(**{k: v for k, v in raw_sc.items() if k in SourceControl.model_fields})
+            else:
+                warning_line(f"No SOURCE_CONTROL framework config found in native app - "
+                             f"defaulting source control platform to {source_control.source_control_platform}")
+        except Exception as e:
+            warning_line(f"Could not load SOURCE_CONTROL framework config "
+                         f"(defaulting source control platform to {source_control.source_control_platform}): {e}")
 
-        # Load WORK_ITEMS settings (for ticket system and ticket_number_required)
-        raw_wi = temp_sf.get_framework_config("WORK_ITEMS")
-        if raw_wi:
-            work_items = WorkItems(**{k: v for k, v in raw_wi.items() if k in WorkItems.model_fields})
+        try:
+            raw_wi = temp_sf.get_framework_config("WORK_ITEMS")
+            if raw_wi:
+                work_items = WorkItems(**{k: v for k, v in raw_wi.items() if k in WorkItems.model_fields})
+        except Exception as e:
+            warning_line(f"Could not load WORK_ITEMS framework config: {e}")
 
         # Get account identifier for MCP URL construction
-        account_identifier = ""
         try:
             org_rows = temp_sf.execute_query("SELECT CURRENT_ORGANIZATION_NAME() || '-' || CURRENT_ACCOUNT_NAME() AS acct")
             account_identifier = org_rows[0]["ACCT"].lower() if org_rows else ""
         except Exception as e:
             warning_line(f"Could not determine account identifier (AI Agent MCP will be skipped): {e}")
 
-        # --- Resolve profile name ---
-        from datasecops_cli.utilities.yaml_utils import get_profile_name
-        profile_name = get_profile_name(project_dir)
-        if profile_name:
-            success_line(f"Profile found in dbt_project.yml: {profile_name}")
-        else:
-            info_line("No dbt_project.yml found. Fetching project profiles...")
-            profiles_data = temp_sf.get_project_profiles()
-            if profiles_data:
-                profile_names = [p.get("PROFILE_NAME", p.get("profile_name", "")) for p in profiles_data]
-                profile_names = [n for n in profile_names if n]
-                if len(profile_names) == 1:
-                    profile_name = profile_names[0]
-                    success_line(f"Auto-selected profile: {profile_name}")
-                elif len(profile_names) > 1:
-                    info_line(f"Found {len(profile_names)} project profiles:")
-                    profile_name = select_from_list(profile_names, "profile", add_back=False)
+        try:
+            # --- Resolve profile name ---
+            from datasecops_cli.utilities.yaml_utils import get_profile_name
+            profile_name = get_profile_name(project_dir)
+            if profile_name:
+                success_line(f"Profile found in dbt_project.yml: {profile_name}")
+            else:
+                info_line("No dbt_project.yml found. Fetching project profiles...")
+                profiles_data = temp_sf.get_project_profiles()
+                if profiles_data:
+                    profile_names = [p.get("PROFILE_NAME", p.get("profile_name", "")) for p in profiles_data]
+                    profile_names = [n for n in profile_names if n]
+                    if len(profile_names) == 1:
+                        profile_name = profile_names[0]
+                        success_line(f"Auto-selected profile: {profile_name}")
+                    elif len(profile_names) > 1:
+                        info_line(f"Found {len(profile_names)} project profiles:")
+                        profile_name = select_from_list(profile_names, "profile", add_back=False)
+                    else:
+                        warning_line("No profiles found in native app")
                 else:
                     warning_line("No profiles found in native app")
-            else:
-                warning_line("No profiles found in native app")
-
-        # Find the full profile object for profiles.yml generation
-        if profile_name:
-            profiles_data = temp_sf.get_project_profiles()
-            if profiles_data:
-                for p in profiles_data:
-                    pn = p.get("PROFILE_NAME", p.get("profile_name", ""))
-                    if pn == profile_name:
-                        matched_profile = ProjectProfile(**{k.lower(): v for k, v in p.items()})
-                        break
-
-        # --- Ensure profiles.yml has the profile entry ---
-        if profile_name and matched_profile:
-            import yaml
-            from datasecops_cli.utilities.file_utils import ensure_dir, write_file
-            from datasecops_cli.services.dbt_project_generator import generate_profiles_yml
 
-            profiles_dir = Path(project_settings.profile_dir).expanduser()
-            profiles_path = profiles_dir / "profiles.yml"
+            # Find the full profile object for profiles.yml generation
+            if profile_name:
+                profiles_data = temp_sf.get_project_profiles()
+                if profiles_data:
+                    for p in profiles_data:
+                        pn = p.get("PROFILE_NAME", p.get("profile_name", ""))
+                        if pn == profile_name:
+                            matched_profile = ProjectProfile(**{k.lower(): v for k, v in p.items()})
+                            break
+
+            # --- Ensure profiles.yml has the profile entry ---
+            if profile_name and matched_profile and not project_settings_loaded:
+                warning_line(f"Skipping profiles.yml for '{profile_name}': PROJECT framework config "
+                             "was not loaded, so the correct CI identity/targets aren't known")
+            elif profile_name and matched_profile:
+                import yaml
+                from datasecops_cli.utilities.file_utils import ensure_dir, write_file
+                from datasecops_cli.services.dbt_project_generator import generate_profiles_yml
+
+                profiles_dir = Path(project_settings.profile_dir).expanduser()
+                profiles_path = profiles_dir / "profiles.yml"
+
+                # Read existing profiles.yml or start fresh
+                existing_profiles = {}
+                if profiles_path.exists():
+                    with open(profiles_path, "r", encoding="utf-8") as f:
+                        existing_profiles = yaml.safe_load(f) or {}
 
-            # Read existing profiles.yml or start fresh
-            existing_profiles = {}
-            if profiles_path.exists():
-                with open(profiles_path, "r", encoding="utf-8") as f:
-                    existing_profiles = yaml.safe_load(f) or {}
-
-            if profile_name in existing_profiles:
-                success_line(f"Profile '{profile_name}' already exists in {profiles_path}")
-            else:
-                info_line(f"Adding profile '{profile_name}' to {profiles_path}...")
                 account_info = temp_sf.get_account_info()
                 account = account_info.get("ACCOUNT", account_info.get("account", ""))
                 user = account_info.get("USER_NAME", account_info.get("user_name", ""))
                 new_profiles_yaml = generate_profiles_yml(matched_profile, project_settings, account, user)
                 new_profiles = yaml.safe_load(new_profiles_yaml) or {}
-                existing_profiles.update(new_profiles)
-                ensure_dir(profiles_dir)
-                write_file(profiles_path, yaml.dump(existing_profiles, default_flow_style=False, sort_keys=False))
-                success_line(f"Profile '{profile_name}' written to {profiles_path}")
+                new_outputs = new_profiles.get(profile_name, {}).get("outputs", {})
+
+                if profile_name in existing_profiles:
+                    # Refresh the CI/deployment targets every run (so a CI identity change in
+                    # the native app takes effect), but keep an existing local-dev target as
+                    # the developer may have customized it.
+                    existing_outputs = existing_profiles[profile_name].setdefault("outputs", {})
+                    for name, output in new_outputs.items():
+                        if name == "local-dev" and name in existing_outputs:
+                            continue
+                        existing_outputs[name] = output
+                    existing_profiles[profile_name].setdefault("target", new_profiles[profile_name]["target"])
+                    write_file(profiles_path, yaml.dump(existing_profiles, default_flow_style=False, sort_keys=False))
+                    success_line(f"Updated deployment targets for profile '{profile_name}' in {profiles_path}")
+                else:
+                    info_line(f"Adding profile '{profile_name}' to {profiles_path}...")
+                    existing_profiles.update(new_profiles)
+                    ensure_dir(profiles_dir)
+                    write_file(profiles_path, yaml.dump(existing_profiles, default_flow_style=False, sort_keys=False))
+                    success_line(f"Profile '{profile_name}' written to {profiles_path}")
+        except Exception as e:
+            warning_line(f"Could not resolve the project profile / write profiles.yml: {e}")
+            warning_line("You can set profile_name manually in .datasecops.yml")
+            warning_line("You may need to create profiles.yml manually")
+            from datasecops_cli.utilities.yaml_utils import get_profile_name
+            profile_name = get_profile_name(project_dir) or ""
 
-    except Exception as e:
-        warning_line(f"Could not connect to Snowflake: {e}")
-        warning_line("You can set profile_name manually in .datasecops.yml")
-        warning_line("You may need to create profiles.yml manually")
-        # Fall back to reading profile from dbt_project.yml if available
-        from datasecops_cli.utilities.yaml_utils import get_profile_name
-        profile_name = get_profile_name(project_dir) or ""
-    finally:
-        temp_sf.close()
+    temp_sf.close()
 
     # --- Write config ---
     config_data = {
@@ -860,7 +902,9 @@ def _run_interactive(config: Config):
         dbt_runner = DbtRunner(
             project_dir=config.dbt_project_dir,
             profiles_dir=config.get_dbt_profiles_dir(),
-            target=config.project_settings.get_default_target().target_name if config.get_default_target() else "dev",
+            # Always the developer's own identity, never the framework's CI/deployment
+            # default target (which may be a deploy-only target running as the CI identity).
+            target="local-dev",
             engine=config.datasecops.dbt_engine or "dbt_core"
         )
 

{datasecops_cli-0.7.0b1 → datasecops_cli-0.7.0b2}/src/datasecops_cli/models/project_config.py

@@ -40,6 +40,9 @@ class ProjectSettings(BaseModel):
     execution_mode: str = "dbt_cli"
     dbt_engine: str = "dbt_core"
     ai_agents_enabled: bool = False
+    ci_username: str = ""
+    ci_role: str = ""
+    ci_warehouse: str = ""
     targets: list[DbtTarget] = Field(default_factory=lambda: [
         DbtTarget(target_name="dev", branch_name="dev", target_role="DEVELOPERS", target_warehouse="DEV_WH", is_default=True),
         DbtTarget(target_name="test", branch_name="test", target_role="DATAOPS_ADMIN", target_warehouse="DATAOPS_WH"),

{datasecops_cli-0.7.0b1 → datasecops_cli-0.7.0b2}/src/datasecops_cli/services/bootstrap_service.py

@@ -112,7 +112,9 @@ class BootstrapService:
             dbt_runner = DbtRunner(
                 project_dir=dbt_project_dir,
                 profiles_dir=Path(self.project_settings.profile_dir).expanduser(),
-                target=self.project_settings.get_default_target().target_name if self.project_settings.get_default_target() else "dev",
+                # Always the developer's own identity, never the framework's CI/deployment
+                # default target.
+                target="local-dev",
                 engine=self.datasecops_config.dbt_engine,
             )
             result = dbt_runner.deps()