datasecops-cli 0.7.0b2__tar.gz → 0.7.0b5__tar.gz

{datasecops_cli-0.7.0b2 → datasecops_cli-0.7.0b5}/CHANGELOG.md

@@ -4,6 +4,21 @@ All notable changes to the DataSecOps CLI are documented in this file.
 
 ## [Unreleased]
 
+## [0.7.0b5] - 2026-10-26
+
+### Added
+
+- **`datasecops pipeline env`** — resolves the dbt target for a pipeline run from the native app's dbt Targets (`--branch` matches a target's Branch, then a target named like the branch; `--target` picks one by name) and sets `TARGET`, `SF_ROLE`, `SF_WAREHOUSE` and `TARGET_SCHEMA` (`$GITHUB_ENV` on GitHub Actions, `##vso` on Azure DevOps, `KEY=value` otherwise). Generated pipelines use it instead of hard-coding the CI role and warehouse.
+
+### Fixed
+
+- **`profiles.yml` deployment targets were being written to a developer's own `~/.dbt/profiles.yml`** — CI/deployment targets (`dev`/`test`/`prod`) belong to the native app's CI pipeline, not a developer's machine, so they're no longer written there at all. Only the `local-dev` target is ever added or refreshed; any other targets already present in the file (added manually, or by a framework target sharing the profile's name) are left untouched. Removed the now-unused CI-identity profile generation (`ProjectSettings.ci_username`, the old `generate_profiles_yml`).
+- **`local-dev` used a hardcoded `authenticator: externalbrowser` and a shared database instead of the developer's own identity** — `local-dev` now reuses whatever Snowflake CLI connection is already configured for the project in `~/.snowflake/connections.toml` (account, user, and whatever auth method is already in use — key pair, SSO, password, etc.), read directly via `snowflake.connector.config_manager` without establishing a connection. `role`/`warehouse`/`schema` still always come from the project's dbt targets config. The developer is prompted once for their name (persisted to `.datasecops.yml`), used to build a personal sandpit database (`<target_database>SANDPIT_<name>`) so multiple developers never collide.
+- **`profiles.yml` is no longer written inside the dbt project folder** — dbt never reads a `profiles.yml` there (it reads `--profiles-dir`, i.e. `~/.dbt` per the native app's `profile_dir` setting), so the project-local copy was always a dead reference file. `datasecops setup`/`bootstrap`/`new project` now write/merge the `local-dev` target directly into `~/.dbt/profiles.yml` via a shared `write_profiles_yml()` helper, creating the folder/file if needed.
+- **`app_database` is validated before it is used in SQL** — it is interpolated into the native app `CALL` statements (CLI and MCP server), so a value that isn't a single plain or double-quoted identifier is now rejected instead of being able to change the statement.
+
+## [0.7.0b2] - 2026-10-26
+
 ### Fixed
 
 - **Default app database renamed to `MONITORIAL_ORBIT`** — the `datasecops setup` suggested app-database name (and matching docs examples) was `DATA_ENGINEERS_DATASECOPS_FRAMEWORK`; it now matches the native app's actual installed name.

{datasecops_cli-0.7.0b2 → datasecops_cli-0.7.0b5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: datasecops-cli
-Version: 0.7.0b2
+Version: 0.7.0b5
 Summary: DataSecOps Framework CLI for Snowflake Native App
 License-Expression: MIT
 License-File: LICENSE
@@ -146,6 +146,33 @@ Your pipeline needs two things:
 
 The exit code is `0` on success, `1` if any download fails.
 
+### Environment settings for the branch
+
+`datasecops pipeline env` reads the dbt Targets from the native app (**Settings › Data › dbt
+Targets**) and sets `TARGET`, `SF_ROLE`, `SF_WAREHOUSE` and `TARGET_SCHEMA` for the target mapped to
+the branch, so pipelines don't hard-code them:
+
+```bash
+datasecops pipeline env --branch "$GITHUB_REF_NAME"         # GitHub Actions: target whose Branch is this branch
+datasecops pipeline env --branch "$BUILD_SOURCEBRANCH"      # Azure DevOps (the full ref, not BUILD_SOURCEBRANCHNAME)
+datasecops pipeline env --target unit-test                  # a named target
+```
+
+Pass the branch through the runner's environment variable, not a `${{ ... }}` / `$(...)` expression:
+those are substituted into the script before the shell runs, so a branch name containing `$(...)`
+would execute.
+
+- `--branch` matches a target's **Branch**, then a target named like the branch; `refs/heads/` is
+  ignored and a pull request's merge ref resolves to its source branch (`$GITHUB_HEAD_REF` on GitHub,
+  `$SYSTEM_PULLREQUEST_SOURCEBRANCH` on Azure DevOps). On Azure DevOps pass `$BUILD_SOURCEBRANCH`:
+  `$BUILD_SOURCEBRANCHNAME` is only the last part of the ref (`release/prod` -> `prod`).
+  Exact names win; a case-insensitive match is used only when it is unambiguous.
+- GitHub Actions: the values are appended to `$GITHUB_ENV`. Azure DevOps: they are set with
+  `##vso[task.setvariable]`. Elsewhere (or `--format shell`) they are printed as shell-quoted `KEY=value` lines.
+- A target with no role, warehouse or schema leaves that variable as it is.
+- The command connects with `.datasecops.yml`'s connection, so the CI user needs the `orbit_developer`
+  application role. It exits `1` when no target matches.
+
 ## DCM projects
 
 ```bash

{datasecops_cli-0.7.0b2 → datasecops_cli-0.7.0b5}/README.md

@@ -126,6 +126,33 @@ Your pipeline needs two things:
 
 The exit code is `0` on success, `1` if any download fails.
 
+### Environment settings for the branch
+
+`datasecops pipeline env` reads the dbt Targets from the native app (**Settings › Data › dbt
+Targets**) and sets `TARGET`, `SF_ROLE`, `SF_WAREHOUSE` and `TARGET_SCHEMA` for the target mapped to
+the branch, so pipelines don't hard-code them:
+
+```bash
+datasecops pipeline env --branch "$GITHUB_REF_NAME"         # GitHub Actions: target whose Branch is this branch
+datasecops pipeline env --branch "$BUILD_SOURCEBRANCH"      # Azure DevOps (the full ref, not BUILD_SOURCEBRANCHNAME)
+datasecops pipeline env --target unit-test                  # a named target
+```
+
+Pass the branch through the runner's environment variable, not a `${{ ... }}` / `$(...)` expression:
+those are substituted into the script before the shell runs, so a branch name containing `$(...)`
+would execute.
+
+- `--branch` matches a target's **Branch**, then a target named like the branch; `refs/heads/` is
+  ignored and a pull request's merge ref resolves to its source branch (`$GITHUB_HEAD_REF` on GitHub,
+  `$SYSTEM_PULLREQUEST_SOURCEBRANCH` on Azure DevOps). On Azure DevOps pass `$BUILD_SOURCEBRANCH`:
+  `$BUILD_SOURCEBRANCHNAME` is only the last part of the ref (`release/prod` -> `prod`).
+  Exact names win; a case-insensitive match is used only when it is unambiguous.
+- GitHub Actions: the values are appended to `$GITHUB_ENV`. Azure DevOps: they are set with
+  `##vso[task.setvariable]`. Elsewhere (or `--format shell`) they are printed as shell-quoted `KEY=value` lines.
+- A target with no role, warehouse or schema leaves that variable as it is.
+- The command connects with `.datasecops.yml`'s connection, so the CI user needs the `orbit_developer`
+  application role. It exits `1` when no target matches.
+
 ## DCM projects
 
 ```bash

{datasecops_cli-0.7.0b2 → datasecops_cli-0.7.0b5}/docs/getting-started.md

@@ -93,15 +93,24 @@ You should see the main menu, confirming the CLI can reach the native app.
 
 ## Step 3b: Set up your local dbt profile (`~/.dbt/profiles.yml`)
 
-Running `datasecops bootstrap` (or **Configuration → new project** from the main menu) scaffolds a
-dbt project from the native app's framework config, including a generated `dbt_project.yml` and
-`profiles.yml`. The generated `profiles.yml` is written *inside* the dbt project folder (e.g.
-`./dbt/profiles.yml`) as a reference — it's gitignored and never committed.
-
-dbt itself does **not** read that file. It reads whatever `profiles.yml` lives in the directory set
-by `--profiles-dir`, which the CLI passes as the native app's `PROJECT.profile_dir` setting
-(defaults to `~/.dbt`). So you need to copy the generated `local-dev` target into your own
-`~/.dbt/profiles.yml` (create the file/folder if they don't exist yet):
+Running `datasecops setup` or `datasecops bootstrap` (or **Configuration → new project** from the
+main menu) writes a `local-dev` target for your profile directly to `~/.dbt/profiles.yml` (per the
+native app's `PROJECT.profile_dir` setting, used as dbt's `--profiles-dir`) — not inside the dbt
+project folder, since dbt never reads a `profiles.yml` there. The `.dbt` folder and `profiles.yml`
+file are created if they don't exist yet; if a profile entry already exists, only its `local-dev`
+target is refreshed — any other targets already in the file (e.g. CI/deployment targets you or a
+pipeline added) are left untouched. **Only `local-dev` is ever written here** — deployment targets
+(`dev`/`test`/`prod`) belong to the native app's CI pipeline, never to a developer's machine.
+
+The `local-dev` target reuses whatever Snowflake CLI connection you already have configured in
+`~/.snowflake/connections.toml` for the `connection_name` in your `.datasecops.yml` — account, user,
+and whatever auth method you already use (external browser, key pair, password, etc.) are copied
+across as-is, so there's nothing extra to configure. You'll be asked for your name once (used to
+build your personal sandpit database, e.g. `jon` → `SDS_SANDPIT_JON`), and it's saved to
+`.datasecops.yml` so you won't be asked again. `role`/`warehouse`/`schema` always come from the
+project's dbt targets config, never from connections.toml.
+
+The resulting `~/.dbt/profiles.yml` looks like:
 
 ```yaml
 <profile_name>:          # matches the `profile:` field in dbt_project.yml
@@ -109,20 +118,17 @@ by `--profiles-dir`, which the CLI passes as the native app's `PROJECT.profile_d
   outputs:
     local-dev:
       type: snowflake
-      account: "<your_account>"
-      authenticator: externalbrowser   # SSO login, no password/PAT needed
+      account: "<from your connections.toml connection>"
+      user: "<from your connections.toml connection>"
+      authenticator: "<from your connections.toml connection, if set>"
+      private_key_path: "<from your connections.toml connection, if set>"
       role: DEVELOPERS
       warehouse: DEV_WH
-      database: "<profile's target database>"
+      database: "<target_database>SANDPIT_<your name>"
       schema: PUBLIC
       threads: 4
 ```
 
-You only need the `local-dev` target locally. The other targets in the generated `profiles.yml`
-(e.g. `dev`, `test`, `prod`) run as the framework's CI/deployment identity (the CI Username, Role
-and Warehouse configured in the native app's Deployment settings) — they're what the CI pipeline
-deploys with, not something a developer runs against directly.
-
 If you work on multiple framework projects, each profile (one per project, named after its
 `profile:` field) gets its own block in the same `~/.dbt/profiles.yml` file.
 

{datasecops_cli-0.7.0b2 → datasecops_cli-0.7.0b5}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "datasecops-cli"
-version = "0.7.0b2"
+version = "0.7.0b5"
 description = "DataSecOps Framework CLI for Snowflake Native App"
 readme = "README.md"
 requires-python = ">=3.10"

datasecops_cli-0.7.0b5/src/datasecops_cli/__init__.py

@@ -0,0 +1 @@
+__version__ = "0.7.0b5"

{datasecops_cli-0.7.0b2 → datasecops_cli-0.7.0b5}/src/datasecops_cli/config.py

@@ -23,8 +23,9 @@ class Config:
         self.dbt_project_dir: Path = Path.cwd()
         self.profile_name: str = ""
     
-    def load(self, project_dir: Path = None) -> bool:
-        """Load configuration from .datasecops.yml and dbt_project.yml."""
+    def load(self, project_dir: Path = None, connection_name: Optional[str] = None) -> bool:
+        """Load configuration from .datasecops.yml and dbt_project.yml. `connection_name`
+        (a --connection option) overrides the file's and may stand in for a missing one."""
         self.project_dir = project_dir or Path.cwd()
         
         # Load .datasecops.yml
@@ -38,6 +39,7 @@ class Config:
             app_database=raw.get("app_database", ""),
             profile_name=raw.get("profile_name", ""),
             cortex_connection=raw.get("cortex_connection", ""),
+            sandpit_name=raw.get("sandpit_name", ""),
             dbt_engine=raw.get("dbt_engine", "dbt_core"),
             dcm_enabled=raw.get("dcm_enabled", False),
             dcm_security_dir=raw.get("dcm_security_dir", "security"),
@@ -53,6 +55,8 @@ class Config:
             warning_line(f"Invalid dbt_engine '{self.datasecops.dbt_engine}' in .datasecops.yml — falling back to dbt_core")
             self.datasecops.dbt_engine = "dbt_core"
         
+        if connection_name:
+            self.datasecops.connection_name = connection_name
         if not self.datasecops.connection_name or not self.datasecops.app_database:
             error_line(".datasecops.yml is missing connection_name or app_database.")
             return False

{datasecops_cli-0.7.0b2 → datasecops_cli-0.7.0b5}/src/datasecops_cli/main.py

@@ -17,7 +17,8 @@ from datasecops_cli.services.download_service import DownloadService
 from datasecops_cli.services.skill_service import SkillService
 from datasecops_cli.services.bootstrap_service import BootstrapService
 from datasecops_cli.services.dcm_service import DcmService
-from datasecops_cli.services import dcm_registry, model_commands
+from datasecops_cli.services import dcm_registry, model_commands, pipeline_env
+from datasecops_cli.models.project_config import DbtTarget
 from datasecops_cli.menus.development import DevelopmentMenu
 from datasecops_cli.menus.git_operations import GitOperationsMenu
 from datasecops_cli.menus.downloads import DownloadsMenu
@@ -239,6 +240,55 @@ def dcm_pipelines_cmd(platform, names, force, projects, connection):
         sf_service.close()
 
 
+# ── pipeline commands ──────────────────────────────────────────────────────
+
+
+@cli.group()
+def pipeline():
+    """Helpers for CI/CD pipelines."""
+
+
+@pipeline.command("env")
+@click.option("--branch", "-b", default=None,
+              help="Branch being built: picks the dbt target mapped to it (a GitHub pull request's "
+                   "<n>/merge ref resolves to its source branch)")
+@click.option("--target", "-t", default=None, help="dbt target name (instead of --branch)")
+@click.option("--connection", "-c", default=None, help="Snowflake connection name (overrides .datasecops.yml)")
+@click.option("--format", "fmt", type=click.Choice(["auto", *pipeline_env.PLATFORMS]), default="auto",
+              show_default=True, help="auto: GitHub when $GITHUB_ENV is set, Azure DevOps when $TF_BUILD is set")
+def pipeline_env_cmd(branch: Optional[str], target: Optional[str], connection: Optional[str], fmt: str):
+    """Set TARGET, SF_ROLE, SF_WAREHOUSE and TARGET_SCHEMA from the app's dbt Targets.
+
+    GitHub Actions: appended to $GITHUB_ENV. Azure DevOps: set with ##vso logging commands.
+    Otherwise printed as KEY=value. Empty role / warehouse / schema values are left unset.
+    """
+    if bool(branch) == bool(target):
+        raise click.UsageError("give exactly one of --branch or --target")
+    config = Config()
+    if not config.load(connection_name=connection):
+        sys.exit(1)
+    sf_service = SnowflakeService(config.datasecops)
+    try:
+        raw = sf_service.get_framework_config("PROJECT") or {}
+    except Exception as e:  # noqa: BLE001 - connector errors
+        error_line(f"Failed to read the dbt targets from {config.datasecops.app_database}: {e}")
+        sys.exit(1)
+    finally:
+        sf_service.close()
+    targets = [DbtTarget(**{k: v for k, v in t.items() if k in DbtTarget.model_fields})
+               for t in raw.get("targets") or []]
+    try:
+        resolved = pipeline_env.resolve_target(targets, branch=branch, target=target)
+    except pipeline_env.TargetNotFound as e:
+        error_line(str(e))
+        sys.exit(1)
+    values = pipeline_env.env_values(resolved)
+    platform = pipeline_env.detect_platform() if fmt == "auto" else fmt
+    for line in pipeline_env.export(values, platform):
+        click.echo(line)
+    click.echo("dbt environment: " + ", ".join(f"{k}={v}" for k, v in values.items()), err=True)
+
+
 # ── model commands ─────────────────────────────────────────────────────────
 
 
@@ -478,6 +528,9 @@ def _run_setup(project_dir: Path):
         "MONITORIAL_ORBIT"
     )
 
+    # --- Sandpit name (for your personal local-dev database) ---
+    sandpit_name = get_input_string("Enter your name (used to build your personal sandpit database, e.g. 'jon'): ")
+
     # --- CoCo agent connection ---
     cortex_connection = ""
     if shutil.which("cortex") and connection_names:
@@ -588,46 +641,15 @@ def _run_setup(project_dir: Path):
             # --- Ensure profiles.yml has the profile entry ---
             if profile_name and matched_profile and not project_settings_loaded:
                 warning_line(f"Skipping profiles.yml for '{profile_name}': PROJECT framework config "
-                             "was not loaded, so the correct CI identity/targets aren't known")
+                             "was not loaded, so the local-dev role/warehouse aren't known")
             elif profile_name and matched_profile:
-                import yaml
-                from datasecops_cli.utilities.file_utils import ensure_dir, write_file
-                from datasecops_cli.services.dbt_project_generator import generate_profiles_yml
+                from datasecops_cli.services.dbt_project_generator import write_profiles_yml
+                from datasecops_cli.services.snowflake_service import get_connection_params
 
                 profiles_dir = Path(project_settings.profile_dir).expanduser()
-                profiles_path = profiles_dir / "profiles.yml"
-
-                # Read existing profiles.yml or start fresh
-                existing_profiles = {}
-                if profiles_path.exists():
-                    with open(profiles_path, "r", encoding="utf-8") as f:
-                        existing_profiles = yaml.safe_load(f) or {}
-
-                account_info = temp_sf.get_account_info()
-                account = account_info.get("ACCOUNT", account_info.get("account", ""))
-                user = account_info.get("USER_NAME", account_info.get("user_name", ""))
-                new_profiles_yaml = generate_profiles_yml(matched_profile, project_settings, account, user)
-                new_profiles = yaml.safe_load(new_profiles_yaml) or {}
-                new_outputs = new_profiles.get(profile_name, {}).get("outputs", {})
-
-                if profile_name in existing_profiles:
-                    # Refresh the CI/deployment targets every run (so a CI identity change in
-                    # the native app takes effect), but keep an existing local-dev target as
-                    # the developer may have customized it.
-                    existing_outputs = existing_profiles[profile_name].setdefault("outputs", {})
-                    for name, output in new_outputs.items():
-                        if name == "local-dev" and name in existing_outputs:
-                            continue
-                        existing_outputs[name] = output
-                    existing_profiles[profile_name].setdefault("target", new_profiles[profile_name]["target"])
-                    write_file(profiles_path, yaml.dump(existing_profiles, default_flow_style=False, sort_keys=False))
-                    success_line(f"Updated deployment targets for profile '{profile_name}' in {profiles_path}")
-                else:
-                    info_line(f"Adding profile '{profile_name}' to {profiles_path}...")
-                    existing_profiles.update(new_profiles)
-                    ensure_dir(profiles_dir)
-                    write_file(profiles_path, yaml.dump(existing_profiles, default_flow_style=False, sort_keys=False))
-                    success_line(f"Profile '{profile_name}' written to {profiles_path}")
+                connection_params = get_connection_params(connection_name)
+                status = write_profiles_yml(profiles_dir, profile_name, matched_profile, project_settings, connection_params, sandpit_name)
+                success_line(f"Profile '{profile_name}' {status} in {profiles_dir / 'profiles.yml'}")
         except Exception as e:
             warning_line(f"Could not resolve the project profile / write profiles.yml: {e}")
             warning_line("You can set profile_name manually in .datasecops.yml")
@@ -643,6 +665,7 @@ def _run_setup(project_dir: Path):
         "app_database": app_database,
         "profile_name": profile_name,
         "cortex_connection": cortex_connection,
+        "sandpit_name": sandpit_name,
     }
     write_datasecops_config(project_dir, config_data)
     success_line(f".datasecops.yml written to {project_dir / '.datasecops.yml'}")
@@ -838,11 +861,11 @@ def _connect_and_load(config: Config, interactive: bool = True,
                     break
             # Save the selected profile back to .datasecops.yml
             if config.profile:
-                config_data = {
-                    "connection_name": config.datasecops.connection_name,
-                    "app_database": config.datasecops.app_database,
-                    "profile_name": config.profile_name,
-                }
+                from datasecops_cli.utilities.yaml_utils import read_datasecops_config
+                config_data = read_datasecops_config(config.project_dir) or {}
+                config_data["connection_name"] = config.datasecops.connection_name
+                config_data["app_database"] = config.datasecops.app_database
+                config_data["profile_name"] = config.profile_name
                 write_datasecops_config(config.project_dir, config_data)
                 success_line(f"Saved profile_name '{config.profile_name}' to .datasecops.yml")
         else:
@@ -850,11 +873,11 @@ def _connect_and_load(config: Config, interactive: bool = True,
             info_line(f"Using default profile: {config.profile_name}")
     elif not profile_was_set and len(config.all_profiles) == 1:
         # Single profile auto-selected — persist it so future runs don't re-prompt
-        config_data = {
-            "connection_name": config.datasecops.connection_name,
-            "app_database": config.datasecops.app_database,
-            "profile_name": config.profile_name,
-        }
+        from datasecops_cli.utilities.yaml_utils import read_datasecops_config
+        config_data = read_datasecops_config(config.project_dir) or {}
+        config_data["connection_name"] = config.datasecops.connection_name
+        config_data["app_database"] = config.datasecops.app_database
+        config_data["profile_name"] = config.profile_name
         write_datasecops_config(config.project_dir, config_data)
         success_line(f"Saved profile_name '{config.profile_name}' to .datasecops.yml")
 
@@ -1274,7 +1297,7 @@ def _show_main_menu(profile_name: str, git_service: GitService = None,
 
 def _run_bootstrap(config: Config):
     """Run the bootstrap command to initialise a new project."""
-    from datasecops_cli.utilities.display import get_input_true_false, select_multiple_from_list
+    from datasecops_cli.utilities.display import get_input_string, get_input_true_false, select_multiple_from_list
 
     sf_service = _connect_and_load(config)
 
@@ -1285,6 +1308,15 @@ def _run_bootstrap(config: Config):
         platform = config.source_control.source_control_platform.lower()
         info_line(f"Platform: {platform}")
 
+        # Sandpit name (for your personal local-dev database) — ask once, then persist
+        if not config.datasecops.sandpit_name:
+            from datasecops_cli.utilities.yaml_utils import read_datasecops_config, write_datasecops_config
+            config.datasecops.sandpit_name = get_input_string(
+                "Enter your name (used to build your personal sandpit database, e.g. 'jon'): ")
+            config_data = read_datasecops_config(config.project_dir) or {}
+            config_data["sandpit_name"] = config.datasecops.sandpit_name
+            write_datasecops_config(config.project_dir, config_data)
+
         # Ask about optional steps
         install_skills = get_input_true_false("Install AI coding skills?")
         skill_targets = None

{datasecops_cli-0.7.0b2 → datasecops_cli-0.7.0b5}/src/datasecops_cli/models/project_config.py

@@ -10,6 +10,7 @@ class DatasecopsConfig(BaseModel):
     app_database: str = ""
     profile_name: str = ""
     cortex_connection: str = ""
+    sandpit_name: str = ""  # personal name used to build your local-dev sandpit database (e.g. "jon")
     dbt_engine: str = "dbt_core"  # "dbt2" for dbt 2.0 (pip install dbt), "dbt_core" for dbt Core 1.x (pip install dbt-core + dbt-snowflake)
     dcm_enabled: bool = False
     dcm_security_dir: str = "security"
@@ -40,9 +41,6 @@ class ProjectSettings(BaseModel):
     execution_mode: str = "dbt_cli"
     dbt_engine: str = "dbt_core"
     ai_agents_enabled: bool = False
-    ci_username: str = ""
-    ci_role: str = ""
-    ci_warehouse: str = ""
     targets: list[DbtTarget] = Field(default_factory=lambda: [
         DbtTarget(target_name="dev", branch_name="dev", target_role="DEVELOPERS", target_warehouse="DEV_WH", is_default=True),
         DbtTarget(target_name="test", branch_name="test", target_role="DATAOPS_ADMIN", target_warehouse="DATAOPS_WH"),

{datasecops_cli-0.7.0b2 → datasecops_cli-0.7.0b5}/src/datasecops_cli/services/dbt_project_generator.py

@@ -46,6 +46,7 @@ _SAFE_PATTERNS = {
     "profile name": re.compile(r"^[A-Za-z_][A-Za-z0-9_-]*$"),
     "dbt version": re.compile(r"^[0-9]+(\.[0-9]+)*$"),
     "warehouse name": re.compile(r"^[A-Za-z_][A-Za-z0-9_$]*$"),
+    "sandpit name": re.compile(r"^[A-Za-z0-9_]+$"),
 }
 
 
@@ -193,74 +194,106 @@ vars:
     )
 
 
-def generate_profiles_yml(
+def generate_local_dev_target(
     profile: ProjectProfile,
     project_settings: ProjectSettings,
-    account: str,
-    user: str = "",
-) -> str:
-    """Generate profiles.yml from framework targets.
-
-    Every target but `local-dev` runs as the framework's CI/deployment identity
-    (Deployment settings: CI Username/Role/Warehouse), not the local developer -
-    these targets are what CI pipelines deploy with. `local-dev` is always the
-    profile's default target (even if a framework target of that name exists or
-    the framework marks a different target as its own default), so a plain
-    `dbt run` with no `--target` always uses the developer's own identity.
+    connection_params: dict[str, Any],
+    sandpit_name: str,
+) -> dict[str, Any]:
+    """Build the `local-dev` profiles.yml output for a developer's own machine.
+
+    Connection identity (account/user/authenticator/private key/password) is taken as-is
+    from the developer's own Snowflake CLI connection (`~/.snowflake/connections.toml`) -
+    whatever auth method they already use to connect works for dbt too, with no CLI code
+    needing to know about it. `role`/`warehouse`/`schema` always come from the project's
+    dbt targets config, never from connections.toml. `database` is the developer's personal
+    sandpit database, built from their name so multiple developers never collide.
 
     Args:
         profile: The project profile.
-        project_settings: The project settings with targets and CI identity.
-        account: Snowflake account name.
-        user: Fallback Snowflake user name, used only when ci_username is not configured.
+        project_settings: The project settings with dbt targets.
+        connection_params: Raw params for the developer's Snowflake CLI connection, as
+            returned by `snowflake_service.get_connection_params()`.
+        sandpit_name: The developer's name, used to build their personal sandpit database.
 
     Returns:
-        The YAML content as a string.
+        The `local-dev` output dict (to embed under `outputs.local-dev` in profiles.yml).
     """
-    outputs: dict[str, Any] = {}
-
-    # Add local-dev target
-    outputs["local-dev"] = {
-        "type": "snowflake",
-        "account": account,
-        "authenticator": "externalbrowser",
-        "role": project_settings.targets[0].target_role if project_settings.targets else "DEVELOPERS",
-        "warehouse": project_settings.targets[0].target_warehouse if project_settings.targets else "DEV_WH",
-        "database": profile.target_database,
-        "schema": project_settings.targets[0].target_schema if project_settings.targets else "PUBLIC",
-        "threads": 4,
-    }
-
-    # Add all framework (CI/deployment) targets, run as the CI identity. `local-dev` is
-    # reserved for interactive human use (added above with externalbrowser auth) - a
-    # framework target of that name must never overwrite it with the CI identity.
-    ci_user = project_settings.ci_username or user or "{{ env_var('SNOWFLAKE_USER') }}"
-    for target in project_settings.targets:
-        if target.target_name == "local-dev":
-            continue
-        branch_suffix = target.branch_name.upper()
-        outputs[target.target_name] = {
-            "type": "snowflake",
-            "account": account,
-            "user": ci_user,
-            "role": project_settings.ci_role or target.target_role,
-            "warehouse": project_settings.ci_warehouse or target.target_warehouse,
-            "database": f"{profile.target_database}_{branch_suffix}",
-            "schema": target.target_schema,
-            "threads": 4,
-        }
+    target = project_settings.targets[0] if project_settings.targets else None
+
+    output: dict[str, Any] = {"type": "snowflake"}
+    for key in (
+        "account", "user", "authenticator", "password",
+        "private_key_path", "private_key", "private_key_passphrase", "token",
+    ):
+        if connection_params.get(key):
+            output[key] = connection_params[key]
+
+    # Names can contain spaces, apostrophes, hyphens etc. (e.g. "O'Brien") - sanitize down to
+    # a safe database-identifier suffix rather than rejecting anything but a bare word.
+    sanitized_name = re.sub(r"[^A-Za-z0-9]+", "_", sandpit_name.strip()).strip("_").upper()
+    if not sanitized_name:
+        raise ValueError(f"sandpit name must contain at least one letter or digit: {sandpit_name!r}")
+    sandpit_suffix = _validate_safe(sanitized_name, "sandpit name")
+    output["role"] = target.target_role if target else "DEVELOPERS"
+    output["warehouse"] = target.target_warehouse if target else "DEV_WH"
+    output["database"] = f"{profile.target_database}SANDPIT_{sandpit_suffix}"
+    output["schema"] = target.target_schema if target else "PUBLIC"
+    output["threads"] = 4
+    return output
+
+
+def write_profiles_yml(
+    profiles_dir,
+    profile_name: str,
+    profile: ProjectProfile,
+    project_settings: ProjectSettings,
+    connection_params: dict[str, Any],
+    sandpit_name: str,
+) -> str:
+    """Add or update only the `local-dev` target for `profile_name` in
+    `<profiles_dir>/profiles.yml`, creating the `.dbt` folder and the file itself if
+    they don't already exist.
 
-    # The profile's default target is always `local-dev`: a bare `dbt run` with no --target
-    # must use the developer's own identity, never the CI identity on a deployment target -
-    # even if the framework's own default target (used for CI/deploy commands) differs.
-    profiles_data = {
-        profile.profile_name: {
+    dbt reads `~/.dbt/profiles.yml` (per `project_settings.profile_dir`), never a copy
+    inside the dbt project folder. Deployment/CI targets (dev/test/prod) are never written
+    here - those belong to the native app's CI pipeline, not a developer's machine. Any
+    other targets already present in the file (added manually, or by a framework target
+    sharing this profile's name) are left untouched.
+
+    Returns "added" or "updated".
+    """
+    from pathlib import Path
+
+    profiles_dir = Path(profiles_dir).expanduser()
+    profiles_path = profiles_dir / "profiles.yml"
+
+    existing_profiles: dict[str, Any] = {}
+    if profiles_path.exists():
+        existing_profiles = yaml.safe_load(profiles_path.read_text(encoding="utf-8")) or {}
+
+    local_dev_output = generate_local_dev_target(profile, project_settings, connection_params, sandpit_name)
+
+    if profile_name in existing_profiles:
+        entry = existing_profiles[profile_name]
+        entry.setdefault("outputs", {})
+        entry["outputs"]["local-dev"] = local_dev_output
+        # Always force the default target to local-dev: a bare `dbt run` with no --target
+        # must use the developer's own identity, never a deployment target that happened
+        # to be marked as the file's default (e.g. by a previous run or manual edit).
+        entry["target"] = "local-dev"
+        status = "updated"
+    else:
+        existing_profiles[profile_name] = {
             "target": "local-dev",
-            "outputs": outputs,
+            "outputs": {"local-dev": local_dev_output},
         }
-    }
+        status = "added"
 
-    return yaml.dump(profiles_data, default_flow_style=False, sort_keys=False)
+    profiles_dir.mkdir(parents=True, exist_ok=True)
+    profiles_path.write_text(
+        yaml.dump(existing_profiles, default_flow_style=False, sort_keys=False), encoding="utf-8")
+    return status
 
 
 def generate_packages_yml(

{datasecops_cli-0.7.0b2 → datasecops_cli-0.7.0b5}/src/datasecops_cli/services/download_service.py

@@ -308,10 +308,11 @@ class DownloadService:
         """
         from datasecops_cli.services.dbt_project_generator import (
             generate_dbt_project_yml,
-            generate_profiles_yml,
+            write_profiles_yml,
             generate_gitignore,
         )
         from datasecops_cli.services.directory_scaffolder import scaffold_directories
+        from datasecops_cli.services.snowflake_service import get_connection_params
 
         info_line("Scaffolding dbt project from framework configuration...")
 
@@ -324,13 +325,16 @@ class DownloadService:
         write_file(dbt_project_dir / "dbt_project.yml", dbt_project_content)
         success_line(f"dbt_project.yml written to {dbt_project_dir}")
 
-        # Generate profiles.yml
-        account_info = self.sf.get_account_info()
-        account = account_info.get("ACCOUNT", account_info.get("account", ""))
-        user = account_info.get("USER_NAME", account_info.get("user_name", ""))
-        profiles_content = generate_profiles_yml(profile, project_settings, account, user)
-        write_file(dbt_project_dir / "profiles.yml", profiles_content)
-        success_line(f"profiles.yml written to {dbt_project_dir}")
+        # profiles.yml is never written inside the dbt project: dbt reads it from
+        # project_settings.profile_dir (default ~/.dbt). Only the local-dev target is ever
+        # written there, using the developer's own Snowflake CLI connection and personal
+        # sandpit database - deployment targets belong to the native app's CI pipeline.
+        connection_params = get_connection_params(self.sf.config.connection_name)
+        profiles_dir = Path(project_settings.profile_dir).expanduser()
+        status = write_profiles_yml(
+            profiles_dir, profile.profile_name, profile, project_settings,
+            connection_params, self.sf.config.sandpit_name)
+        success_line(f"profiles.yml {status} in {profiles_dir}")
 
         # Generate .gitignore
         write_file(dbt_project_dir / ".gitignore", generate_gitignore())