datasecops-cli 0.2.6__tar.gz → 0.2.8__tar.gz

{datasecops_cli-0.2.6 → datasecops_cli-0.2.8}/CHANGELOG.md

@@ -2,6 +2,31 @@
 
 All notable changes to the DataSecOps CLI are documented in this file.
 
+## [0.2.8] - 2026-05-14
+
+### Added
+
+- **Non-interactive `download` subcommand** — `datasecops download <items>` downloads framework config without prompts, designed for CI/CD pipelines. Supports `sqlfluff`, `pipelines`, `packages`, `macros`, or `all`
+- **Runtime `dbtf` check** — the CLI now warns at startup if dbt Fusion (`dbtf`) is not found on PATH
+
+### Changed
+
+- **Platform auto-detected from framework config** — pipeline downloads, bootstrap, and the downloads menu now read `source_control_platform` from the native app instead of prompting the user to select GitHub or Azure DevOps
+- **Case-insensitive platform matching** — `download_pipelines()` now compares platform values case-insensitively, fixing a bug where pipelines stored as `"GitHub"` would not match the lowercase `"github"` filter
+- **All dbt commands use `dbtf`** — `DbtRunner` and setup scripts now consistently use the `dbtf` binary (dbt Fusion) instead of `dbt`
+- **Refactored CLI entry point** — `main()` now uses `argparse` with subcommands (`bootstrap`, `download`) instead of manual `sys.argv` parsing. Shared connection/config logic extracted to `_connect_and_load()`
+- **Development menu option 14** relabelled to clarify it installs dbt-core/dbt-snowflake for linting, not dbt Fusion
+
+## [0.2.7] - 2026-05-12
+
+### Changed
+
+- **Separate SQLFluff and dbt installs** — `get_sqlfluff_requirements()` now returns only `sqlfluff` and `sqlfluff-templater-dbt`; new `get_dbt_requirements()` returns only `dbt-core` and `dbt-snowflake`. The lint menu install option and the new development menu install option each handle their own packages independently
+- **Reorganised development menu** — added `[7] parse` for `dbt parse`, added `[14] install dbt` for explicit dbt-core/dbt-snowflake installation, removed standalone retry (available in run and test submenus)
+- **Expanded test submenu** — added unit tests (`test_type:unit`), data tests (`test_type:data`), and failed test retry options alongside all tests and specific selector
+- **Clearer input prompts** — run, test, and lint specific-file prompts now include examples of valid input (e.g. `my_model+`, `tag:nightly`, `models/staging/stg_orders.sql`)
+- **Bootstrap shows separate install hints** — step 7 now lists SQLFluff and dbt requirements separately with individual `uv pip install` commands
+
 ## [0.2.6] - 2026-05-11
 
 ### Fixed

{datasecops_cli-0.2.6 → datasecops_cli-0.2.8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: datasecops-cli
-Version: 0.2.6
+Version: 0.2.8
 Summary: DataSecOps Framework CLI for Snowflake Native App
 License-Expression: MIT
 License-File: LICENSE
@@ -55,7 +55,7 @@ Requires Python 3.10 or later.
 
 Optional:
 
-- **dbt Fusion** (or dbt-core with dbt-snowflake) for dbt commands
+- **dbt Fusion** — required for dbt commands (`dbtf`). Install from https://docs.getdbt.com/docs/core/installation
 - **Cortex Code** for skill downloads
 - **Node.js 18+** for GitHub/Azure DevOps MCP servers
 
@@ -96,6 +96,58 @@ datasecops
 | **Git** | Branch create/checkout/delete, commit & push, rebase, squash, deploy to environment branches, cherry-pick |
 | **Downloads** | SQLFluff config, CI/CD pipelines (GitHub Actions / Azure DevOps), dbt packages, Cortex Code skills |
 
+## Non-Interactive Mode (CI/CD)
+
+The `download` subcommand lets you pull framework config in CI/CD pipelines without interactive prompts:
+
+```bash
+# Download specific items
+datasecops download sqlfluff
+datasecops download sqlfluff packages
+datasecops download pipelines macros
+
+# Download everything
+datasecops download all
+```
+
+Available items: `sqlfluff`, `pipelines`, `packages`, `macros`, `all`
+
+The pipeline platform (GitHub / Azure DevOps) is auto-detected from the native app's source control configuration.
+
+### Pipeline Setup
+
+Your pipeline needs two things:
+
+1. **A `.datasecops.yml`** in the repo (already committed — contains no secrets):
+
+   ```yaml
+   connection_name: "ci"
+   app_database: "DATA_ENGINEERS_DATASECOPS_FRAMEWORK"
+   ```
+
+2. **A Snowflake connection** in `~/.snowflake/connections.toml` for the CI service account:
+
+   ```yaml
+   # GitHub Actions example
+   - name: Configure Snowflake connection
+     run: |
+       mkdir -p ~/.snowflake
+       cat > ~/.snowflake/connections.toml << EOF
+       [ci]
+       account = "${{ vars.SNOWFLAKE_ACCOUNT }}"
+       user = "${{ vars.SNOWFLAKE_USER }}"
+       authenticator = "snowflake_jwt"
+       private_key_file = "/tmp/rsa_key.p8"
+       warehouse = "CI_WH"
+       role = "CI_ROLE"
+       EOF
+
+   - name: Download SQLFluff config
+     run: datasecops download sqlfluff
+   ```
+
+The exit code is `0` on success, `1` if any download fails.
+
 ## MCP Server
 
 The package includes an MCP (Model Context Protocol) server that exposes your framework's governance configuration to AI coding assistants. Instead of static skill files, the MCP server gives AI tools live access to your native app's current rules.
@@ -154,12 +206,6 @@ app_database: "DATA_ENGINEERS_DATASECOPS_FRAMEWORK"
 
 Project profiles, linting rules, pipeline templates, and deployment targets are all managed centrally in the native app and pulled down by the CLI.
 
-## Documentation
-
-- [Getting Started Guide](docs/getting-started.md) — install CLI, configure MCP servers for VS Code/Cursor/Cortex Code with Snowflake, dbt, and GitHub/Azure DevOps
-- [MCP Server Reference](docs/mcp-server.md) — full tool documentation, architecture, and usage examples
-- [Development Guide](DEVELOPMENT.md) — project structure, setup scripts, native app API reference, and publishing details
-
 ## License
 
 MIT

{datasecops_cli-0.2.6 → datasecops_cli-0.2.8}/README.md

@@ -35,7 +35,7 @@ Requires Python 3.10 or later.
 
 Optional:
 
-- **dbt Fusion** (or dbt-core with dbt-snowflake) for dbt commands
+- **dbt Fusion** — required for dbt commands (`dbtf`). Install from https://docs.getdbt.com/docs/core/installation
 - **Cortex Code** for skill downloads
 - **Node.js 18+** for GitHub/Azure DevOps MCP servers
 
@@ -76,6 +76,58 @@ datasecops
 | **Git** | Branch create/checkout/delete, commit & push, rebase, squash, deploy to environment branches, cherry-pick |
 | **Downloads** | SQLFluff config, CI/CD pipelines (GitHub Actions / Azure DevOps), dbt packages, Cortex Code skills |
 
+## Non-Interactive Mode (CI/CD)
+
+The `download` subcommand lets you pull framework config in CI/CD pipelines without interactive prompts:
+
+```bash
+# Download specific items
+datasecops download sqlfluff
+datasecops download sqlfluff packages
+datasecops download pipelines macros
+
+# Download everything
+datasecops download all
+```
+
+Available items: `sqlfluff`, `pipelines`, `packages`, `macros`, `all`
+
+The pipeline platform (GitHub / Azure DevOps) is auto-detected from the native app's source control configuration.
+
+### Pipeline Setup
+
+Your pipeline needs two things:
+
+1. **A `.datasecops.yml`** in the repo (already committed — contains no secrets):
+
+   ```yaml
+   connection_name: "ci"
+   app_database: "DATA_ENGINEERS_DATASECOPS_FRAMEWORK"
+   ```
+
+2. **A Snowflake connection** in `~/.snowflake/connections.toml` for the CI service account:
+
+   ```yaml
+   # GitHub Actions example
+   - name: Configure Snowflake connection
+     run: |
+       mkdir -p ~/.snowflake
+       cat > ~/.snowflake/connections.toml << EOF
+       [ci]
+       account = "${{ vars.SNOWFLAKE_ACCOUNT }}"
+       user = "${{ vars.SNOWFLAKE_USER }}"
+       authenticator = "snowflake_jwt"
+       private_key_file = "/tmp/rsa_key.p8"
+       warehouse = "CI_WH"
+       role = "CI_ROLE"
+       EOF
+
+   - name: Download SQLFluff config
+     run: datasecops download sqlfluff
+   ```
+
+The exit code is `0` on success, `1` if any download fails.
+
 ## MCP Server
 
 The package includes an MCP (Model Context Protocol) server that exposes your framework's governance configuration to AI coding assistants. Instead of static skill files, the MCP server gives AI tools live access to your native app's current rules.
@@ -134,12 +186,6 @@ app_database: "DATA_ENGINEERS_DATASECOPS_FRAMEWORK"
 
 Project profiles, linting rules, pipeline templates, and deployment targets are all managed centrally in the native app and pulled down by the CLI.
 
-## Documentation
-
-- [Getting Started Guide](docs/getting-started.md) — install CLI, configure MCP servers for VS Code/Cursor/Cortex Code with Snowflake, dbt, and GitHub/Azure DevOps
-- [MCP Server Reference](docs/mcp-server.md) — full tool documentation, architecture, and usage examples
-- [Development Guide](DEVELOPMENT.md) — project structure, setup scripts, native app API reference, and publishing details
-
 ## License
 
 MIT

{datasecops_cli-0.2.6 → datasecops_cli-0.2.8}/docs/getting-started.md

@@ -14,7 +14,7 @@ Before you begin, ensure you have:
 - **Git** configured with access to your repository
 
 Optional:
-- **dbt Fusion** (or dbt-core with dbt-snowflake) for dbt commands
+- **dbt Fusion** — required for dbt commands (`dbtf`). Install from https://docs.getdbt.com/docs/core/installation. The CLI invokes `dbtf` for all dbt operations.
 - **A GitHub Personal Access Token** (for GitHub MCP server)
 - **An Azure DevOps PAT** (for Azure DevOps MCP server)
 

{datasecops_cli-0.2.6 → datasecops_cli-0.2.8}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "datasecops-cli"
-version = "0.2.6"
+version = "0.2.8"
 description = "DataSecOps Framework CLI for Snowflake Native App"
 readme = "README.md"
 requires-python = ">=3.10"

{datasecops_cli-0.2.6 → datasecops_cli-0.2.8}/setup.ps1

@@ -16,13 +16,14 @@ try {
     $env:Path = "$env:USERPROFILE\.local\bin;$env:Path"
 }
 
-# Check for dbt
+# Check for dbt Fusion (dbtf)
 try {
-    $null = Get-Command dbt -ErrorAction Stop
-    Write-Host "[OK] dbt found" -ForegroundColor Green
+    $null = Get-Command dbtf -ErrorAction Stop
+    Write-Host "[OK] dbt Fusion (dbtf) found" -ForegroundColor Green
 } catch {
-    Write-Host "WARNING: dbt not found on PATH." -ForegroundColor Yellow
-    Write-Host "  Install dbt Fusion or: pip install dbt-snowflake" -ForegroundColor Yellow
+    Write-Host "WARNING: dbt Fusion (dbtf) not found on PATH." -ForegroundColor Yellow
+    Write-Host "  Install dbt Fusion: https://docs.getdbt.com/docs/core/installation" -ForegroundColor Yellow
+    Write-Host "  The CLI uses 'dbtf' for all dbt commands." -ForegroundColor Yellow
 }
 
 # Check for cortex (optional)

{datasecops_cli-0.2.6 → datasecops_cli-0.2.8}/setup.sh

@@ -16,12 +16,12 @@ if ! command -v uv &> /dev/null; then
 fi
 echo "[OK] uv $(uv --version 2>/dev/null || echo 'installed')"
 
-# Check for dbt
-if ! command -v dbt &> /dev/null; then
+# Check for dbt Fusion (dbtf)
+if ! command -v dbtf &> /dev/null; then
     echo ""
-    echo "WARNING: dbt not found on PATH."
+    echo "WARNING: dbt Fusion (dbtf) not found on PATH."
     echo "  Install dbt Fusion: https://docs.getdbt.com/docs/core/installation"
-    echo "  Or install via pip: pip install dbt-snowflake"
+    echo "  The CLI uses 'dbtf' for all dbt commands."
     echo ""
 fi
 

{datasecops_cli-0.2.6 → datasecops_cli-0.2.8}/src/datasecops_cli/main.py

@@ -1,3 +1,5 @@
+import argparse
+import shutil
 import sys
 from pathlib import Path
 
@@ -18,22 +20,55 @@ from datasecops_cli.utilities.display import (
 )
 
 
+DOWNLOAD_ITEMS = ["sqlfluff", "pipelines", "packages", "macros", "all"]
+
+
+def _build_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        prog="datasecops",
+        description="DataSecOps Framework CLI for Snowflake Native App",
+    )
+    sub = parser.add_subparsers(dest="command")
+
+    sub.add_parser("bootstrap", help="Set up a new dbt project with all framework config")
+
+    dl = sub.add_parser("download", help="Download framework config non-interactively (for CI/CD)")
+    dl.add_argument(
+        "items",
+        nargs="+",
+        choices=DOWNLOAD_ITEMS,
+        help="Item(s) to download: sqlfluff, pipelines, packages, macros, or all",
+    )
+
+    return parser
+
+
 def main():
     """Main entry point for the datasecops CLI."""
+    parser = _build_parser()
+    args = parser.parse_args()
+
     config = Config()
 
-    # Handle 'bootstrap' subcommand
-    if len(sys.argv) > 1 and sys.argv[1] == "bootstrap":
+    if args.command == "bootstrap":
         _run_bootstrap(config)
-        return
+    elif args.command == "download":
+        _run_download(config, args.items)
+    else:
+        _run_interactive(config)
 
+
+def _connect_and_load(config: Config) -> SnowflakeService:
+    """Load config, connect to Snowflake, and load native app settings.
+    
+    Returns the connected SnowflakeService, or calls sys.exit on failure.
+    """
     if not config.load():
         sys.exit(1)
-    
-    # Connect to Snowflake
+
     sf_config = config.datasecops
     sf_service = SnowflakeService(sf_config)
-    
+
     try:
         info_line(f"Connecting to Snowflake ({sf_config.connection_name})...")
         sf_service.connect()
@@ -41,38 +76,85 @@ def main():
     except Exception as e:
         error_line(f"Failed to connect to Snowflake: {e}")
         sys.exit(1)
-    
+
+    info_line("Loading framework configuration...")
+    config.load_from_native_app(sf_service)
+
+    if not config.profile:
+        error_line(f"Profile '{config.profile_name}' not found in native app")
+        sf_service.close()
+        sys.exit(1)
+
+    success_line(f"Profile: {config.profile.project_name} ({config.profile_name})")
+    return sf_service
+
+
+def _run_interactive(config: Config):
+    """Run the interactive menu-driven CLI."""
+    sf_service = _connect_and_load(config)
+
     try:
-        # Load config from native app
-        info_line("Loading framework configuration...")
-        config.load_from_native_app(sf_service)
-        
-        if not config.profile:
-            error_line(f"Profile '{config.profile_name}' not found in native app")
-            sys.exit(1)
-        
-        success_line(f"Profile: {config.profile.project_name} ({config.profile_name})")
-        
+        # Check for dbt Fusion (dbtf) on PATH
+        if not shutil.which("dbtf"):
+            from datasecops_cli.utilities.display import warning_line
+            warning_line("dbt Fusion (dbtf) not found on PATH — dbt commands will not work.")
+            warning_line("Install dbt Fusion: https://docs.getdbt.com/docs/core/installation")
+
         # Initialize services
         dbt_runner = DbtRunner(
             project_dir=config.dbt_project_dir,
             profiles_dir=config.get_dbt_profiles_dir(),
             target=config.project_settings.get_default_target().target_name if config.get_default_target() else "dev"
         )
-        
+
         try:
             git_service = GitService(config.project_dir)
         except Exception:
             git_service = None
-        
+
         linting_service = LintingService(config.dbt_project_dir)
         download_service = DownloadService(sf_service, config.project_dir)
         skill_service = SkillService(sf_service)
-        
+
         # Main menu loop
-        _main_menu(config, dbt_runner, git_service, linting_service, 
+        _main_menu(config, dbt_runner, git_service, linting_service,
                     download_service, skill_service, sf_service)
-    
+    finally:
+        sf_service.close()
+
+
+def _run_download(config: Config, items: list[str]):
+    """Run non-interactive downloads for CI/CD pipelines."""
+    sf_service = _connect_and_load(config)
+
+    try:
+        download_service = DownloadService(sf_service, config.project_dir)
+        profiles_dir = str(config.get_dbt_profiles_dir())
+
+        if "all" in items:
+            items = ["sqlfluff", "pipelines", "packages", "macros"]
+
+        failed = False
+        for item in items:
+            info_line("")
+            if item == "sqlfluff":
+                if not download_service.download_sqlfluff_config(
+                    profiles_dir=profiles_dir, dbt_project_dir=config.dbt_project_dir
+                ):
+                    failed = True
+            elif item == "pipelines":
+                platform = config.source_control.source_control_platform.lower()
+                info_line(f"Platform: {platform}")
+                if not download_service.download_pipelines(platform=platform):
+                    failed = True
+            elif item == "packages":
+                if not download_service.download_dbt_packages(config.dbt_project_dir):
+                    failed = True
+            elif item == "macros":
+                if not download_service.download_macros(config.profile_name, config.dbt_project_dir):
+                    failed = True
+
+        sys.exit(1 if failed else 0)
     finally:
         sf_service.close()
 
@@ -110,12 +192,14 @@ def _main_menu(config: Config, dbt_runner: DbtRunner, git_service: GitService,
                 profile_name, config.dbt_project_dir,
                 project_settings=config.project_settings,
                 profile=config.profile,
+                source_control=config.source_control,
             )
             dl_menu.show()
         
         elif option == 4:
-            from datasecops_cli.utilities.display import select_from_list, get_input_true_false
-            platform = select_from_list(["github", "azuredevops"], "source control platform", add_back=False)
+            from datasecops_cli.utilities.display import get_input_true_false
+            platform = config.source_control.source_control_platform.lower()
+            info_line(f"Platform: {platform}")
             install_skills = get_input_true_false("Install Cortex Code skills?")
             run_deps = get_input_true_false("Run dbt deps after downloading packages?")
             bootstrap = BootstrapService(
@@ -145,35 +229,16 @@ def _show_main_menu(profile_name: str, git_service: GitService = None):
 
 def _run_bootstrap(config: Config):
     """Run the bootstrap command to initialise a new project."""
-    from datasecops_cli.utilities.display import select_from_list, get_input_true_false
+    from datasecops_cli.utilities.display import get_input_true_false
 
-    if not config.load():
-        sys.exit(1)
-
-    sf_config = config.datasecops
-    sf_service = SnowflakeService(sf_config)
-
-    try:
-        info_line(f"Connecting to Snowflake ({sf_config.connection_name})...")
-        sf_service.connect()
-        success_line("Connected")
-    except Exception as e:
-        error_line(f"Failed to connect to Snowflake: {e}")
-        sys.exit(1)
+    sf_service = _connect_and_load(config)
 
     try:
-        info_line("Loading framework configuration...")
-        config.load_from_native_app(sf_service)
-
-        if not config.profile:
-            error_line(f"Profile '{config.profile_name}' not found in native app")
-            sys.exit(1)
-
-        success_line(f"Profile: {config.profile.project_name} ({config.profile_name})")
         info_line("")
 
-        # Ask for platform
-        platform = select_from_list(["github", "azuredevops"], "source control platform", add_back=False)
+        # Use platform from framework config
+        platform = config.source_control.source_control_platform.lower()
+        info_line(f"Platform: {platform}")
 
         # Ask about optional steps
         install_skills = get_input_true_false("Install Cortex Code skills?")

{datasecops_cli-0.2.6 → datasecops_cli-0.2.8}/src/datasecops_cli/menus/development.py

@@ -48,33 +48,35 @@ class DevelopmentMenu:
                 self.dbt.deps()
                 complete_action()
             elif option == 7:
+                display_action_header("dbt parse")
+                self.dbt.parse()
+                complete_action()
+            elif option == 8:
                 display_action_header("dbt compile")
                 self.dbt.compile()
                 complete_action()
-            elif option == 8:
+            elif option == 9:
                 display_action_header("dbt snapshot")
                 self.dbt.snapshot()
                 complete_action()
-            elif option == 9:
+            elif option == 10:
                 display_action_header("dbt source freshness")
                 self.dbt.source_freshness()
                 complete_action()
-            elif option == 10:
+            elif option == 11:
                 display_action_header("dbt clean")
                 self.dbt.clean()
                 complete_action()
-            elif option == 11:
+            elif option == 12:
                 display_action_header("dbt debug")
                 self.dbt.debug()
                 complete_action()
-            elif option == 12:
+            elif option == 13:
                 display_action_header("dbt list")
                 self.dbt.list_models()
                 complete_action()
-            elif option == 13:
-                display_action_header("dbt retry")
-                self.dbt.retry()
-                complete_action()
+            elif option == 14:
+                self._install_dbt_requirements()
             self._menu()
             option = get_input_number("Choose an option: ")
     
@@ -87,13 +89,14 @@ class DevelopmentMenu:
         menu_option(4,  "docs          - Generate & serve dbt docs")
         menu_option(5,  "seed          - Load seed data")
         menu_option(6,  "deps          - Install dbt packages")
-        menu_option(7,  "compile       - Compile dbt models")
-        menu_option(8,  "snapshot      - Run dbt snapshots")
-        menu_option(9,  "freshness     - Check source freshness")
-        menu_option(10, "clean         - Clean dbt target")
-        menu_option(11, "debug         - Debug dbt connection")
-        menu_option(12, "list          - List dbt resources")
-        menu_option(13, "retry         - Retry failed dbt run")
+        menu_option(7,  "parse         - Parse dbt project")
+        menu_option(8,  "compile       - Compile dbt models")
+        menu_option(9,  "snapshot      - Run dbt snapshots")
+        menu_option(10, "freshness     - Check source freshness")
+        menu_option(11, "clean         - Clean dbt target")
+        menu_option(12, "debug         - Debug dbt connection")
+        menu_option(13, "list          - List dbt resources")
+        menu_option(14, "install lint  - Install dbt-core & dbt-snowflake for linting (not dbt Fusion)")
         menu_option(0,  "back          - Return to main menu")
     
     def _run_menu(self) -> None:
@@ -113,7 +116,7 @@ class DevelopmentMenu:
             full_refresh = get_input_true_false("Full refresh?", "n")
             self.dbt.run(modified_only=True, full_refresh=full_refresh)
         elif option == 3:
-            select = get_input_string("Enter model selector: ")
+            select = get_input_string("Enter model selector (e.g. my_model, my_model+, tag:nightly, path:models/...): ")
             if select != "0":
                 self.dbt.run(select=select)
         elif option == 4:
@@ -129,13 +132,22 @@ class DevelopmentMenu:
         clear()
         display_action_header("dbt Test Options")
         menu_option(1, "all tests     - Run all tests")
-        menu_option(2, "specific      - Run specific test(s)")
+        menu_option(2, "unit tests    - Run unit tests only")
+        menu_option(3, "data tests    - Run data tests only")
+        menu_option(4, "failed        - Retry failed tests")
+        menu_option(5, "specific      - Run specific test(s)")
         menu_option(0, "back          - Return to development menu")
         option = get_input_number("Choose an option: ")
         if option == 1:
             self.dbt.test()
         elif option == 2:
-            select = get_input_string("Enter test selector: ")
+            self.dbt.test(select="test_type:unit")
+        elif option == 3:
+            self.dbt.test(select="test_type:data")
+        elif option == 4:
+            self.dbt.retry()
+        elif option == 5:
+            select = get_input_string("Enter test selector (e.g. test_name, model_name, tag:nightly): ")
             if select != "0":
                 self.dbt.test(select=select)
         complete_action()
@@ -179,7 +191,7 @@ class DevelopmentMenu:
         elif option == 4:
             self.linting.lint_file(fix=True)
         elif option == 5:
-            path = get_input_string("Enter file path: ")
+            path = get_input_string("Enter SQL file path (e.g. models/staging/stg_orders.sql): ")
             if path != "0":
                 self.linting.lint_file(file_path=path, fix=False)
         elif option == 6:
@@ -192,11 +204,11 @@ class DevelopmentMenu:
             error_line("Download service not available")
             return
         
-        # Show current versions
+        # Show current sqlfluff versions
         installed = self.linting.get_installed_versions()
         info_line("Currently installed:")
-        for pkg, ver in installed.items():
-            info_line(f"  {pkg}: {ver or 'not installed'}")
+        for pkg in ("sqlfluff", "sqlfluff-templater-dbt"):
+            info_line(f"  {pkg}: {installed.get(pkg) or 'not installed'}")
         
         # Fetch required versions from framework
         packages = self.downloads.get_sqlfluff_requirements()
@@ -210,3 +222,32 @@ class DevelopmentMenu:
         
         info_line("")
         self.linting.install_requirements(packages)
+
+    def _install_dbt_requirements(self) -> None:
+        """Install dbt-core and dbt-snowflake at versions defined by the framework."""
+        display_action_header("Install dbt Requirements")
+        if not self.downloads:
+            error_line("Download service not available")
+            complete_action()
+            return
+        
+        # Show current dbt versions
+        installed = self.linting.get_installed_versions()
+        info_line("Currently installed:")
+        for pkg in ("dbt-core", "dbt-snowflake"):
+            info_line(f"  {pkg}: {installed.get(pkg) or 'not installed'}")
+        
+        # Fetch required dbt versions from framework
+        packages = self.downloads.get_dbt_requirements()
+        if not packages:
+            complete_action()
+            return
+        
+        info_line("")
+        info_line("Framework requires:")
+        for pkg in packages:
+            info_line(f"  {pkg}")
+        
+        info_line("")
+        self.linting.install_requirements(packages)
+        complete_action()

{datasecops_cli-0.2.6 → datasecops_cli-0.2.8}/src/datasecops_cli/menus/downloads.py

@@ -1,6 +1,6 @@
 from pathlib import Path
 
-from datasecops_cli.models.project_config import ProjectProfile, ProjectSettings
+from datasecops_cli.models.project_config import ProjectProfile, ProjectSettings, SourceControl
 from datasecops_cli.services.download_service import DownloadService
 from datasecops_cli.services.skill_service import SkillService
 from datasecops_cli.services.dbt_runner import DbtRunner
@@ -14,7 +14,8 @@ from datasecops_cli.utilities.display import (
 class DownloadsMenu:
     def __init__(self, download_service: DownloadService, skill_service: SkillService,
                  dbt_runner: DbtRunner, profile_name: str, dbt_project_dir: Path,
-                 project_settings: ProjectSettings = None, profile: ProjectProfile = None):
+                 project_settings: ProjectSettings = None, profile: ProjectProfile = None,
+                 source_control: SourceControl = None):
         self.downloads = download_service
         self.skills = skill_service
         self.dbt = dbt_runner
@@ -22,6 +23,7 @@ class DownloadsMenu:
         self.dbt_project_dir = dbt_project_dir
         self.project_settings = project_settings
         self.profile = profile
+        self.source_control = source_control
     
     def show(self) -> None:
         self._menu()
@@ -34,7 +36,8 @@ class DownloadsMenu:
                 complete_action()
             elif option == 2:
                 display_action_header("Download Pipeline Files")
-                platform = select_from_list(["github", "azuredevops"], "platform", add_back=False)
+                platform = self.source_control.source_control_platform.lower() if self.source_control else "github"
+                info_line(f"Platform: {platform}")
                 self.downloads.download_pipelines(platform=platform)
                 complete_action()
             elif option == 3:

{datasecops_cli-0.2.6 → datasecops_cli-0.2.8}/src/datasecops_cli/services/bootstrap_service.py

@@ -109,14 +109,18 @@ class BootstrapService:
                 info_line("  (dbt deps failed - run manually after fixing packages.yml)")
                 steps_passed += 1
 
-        # Step 7: Download SQLFluff requirements
+        # Step 7: Check SQLFluff & dbt version requirements
         info_line("")
         step_num = 7 if run_deps else 6
-        info_line(f"[{step_num}] Checking SQLFluff version requirements...")
-        requirements = self.download_service.get_sqlfluff_requirements()
-        if requirements:
-            info_line(f"  Required: {', '.join(requirements)}")
-            info_line("  (install with: uv pip install " + " ".join(requirements) + ")")
+        info_line(f"[{step_num}] Checking SQLFluff & dbt version requirements...")
+        sqlfluff_requirements = self.download_service.get_sqlfluff_requirements()
+        if sqlfluff_requirements:
+            info_line(f"  SQLFluff: {', '.join(sqlfluff_requirements)}")
+            info_line("  (install with: uv pip install " + " ".join(sqlfluff_requirements) + ")")
+        dbt_requirements = self.download_service.get_dbt_requirements()
+        if dbt_requirements:
+            info_line(f"  dbt:      {', '.join(dbt_requirements)}")
+            info_line("  (install with: uv pip install " + " ".join(dbt_requirements) + ")")
         steps_passed += 1
 
         # Step 8: Install Cortex Code skills
@@ -134,7 +138,7 @@ class BootstrapService:
         info_line("Your project is ready. Next steps:")
         info_line(f"  cd {dbt_project_dir}")
         info_line("  datasecops          — run the framework CLI")
-        info_line("  dbt debug           — verify Snowflake connection")
-        info_line("  dbt run             — run your first models")
+        info_line("  dbtf debug          — verify Snowflake connection")
+        info_line("  dbtf run            — run your first models")
         info_line("")
         return True

{datasecops_cli-0.2.6 → datasecops_cli-0.2.8}/src/datasecops_cli/services/dbt_runner.py

@@ -21,7 +21,7 @@ class DbtRunner:
         ]
     
     def _run_command(self, command: str, extra_args: list[str] = None) -> subprocess.CompletedProcess:
-        cmd = ["dbt", command] + (extra_args or []) + self._default_args()
+        cmd = ["dbtf", command] + (extra_args or []) + self._default_args()
         info_line(f"Running: {' '.join(cmd)}")
         result = subprocess.run(cmd, capture_output=False)
         if result.returncode != 0:
@@ -94,7 +94,7 @@ class DbtRunner:
         return self._run_command("docs", ["generate", f"--target={self.target}"])
     
     def docs_serve(self) -> subprocess.Popen:
-        cmd = ["dbt", "docs", "serve"] + self._default_args()
+        cmd = ["dbtf", "docs", "serve"] + self._default_args()
         info_line(f"Running: {' '.join(cmd)}")
         return subprocess.Popen(cmd)
     
@@ -117,6 +117,9 @@ class DbtRunner:
         self._copy_manifest()
         return result
     
+    def parse(self) -> subprocess.CompletedProcess:
+        return self._run_command("parse")
+
     def run_operation(self, macro: str, args_str: str = None) -> subprocess.CompletedProcess:
         cmd_args = [f"--target={self.target}"]
         if args_str:

{datasecops_cli-0.2.6 → datasecops_cli-0.2.8}/src/datasecops_cli/services/download_service.py

@@ -159,7 +159,12 @@ class DownloadService:
                 section_name = self.RULE_SECTION_MAP.get(code)
                 if not section_name:
                     continue
-                opts = entry.get("options", {})
+                opts = dict(entry.get("options", {}))
+                # For aliasing.length, 0 means "no limit" which sqlfluff expects as None
+                if section_name == "aliasing.length":
+                    for len_key in ("min_alias_length", "max_alias_length"):
+                        if len_key in opts and opts[len_key] == 0:
+                            opts[len_key] = None
                 self._emit_section(lines, f"sqlfluff:rules:{section_name}", opts)
 
         content = "\n".join(lines)
@@ -168,17 +173,16 @@ class DownloadService:
         success_line(f"SQLFluff config written to {dest}")
         return True
     
-    def get_sqlfluff_requirements(self) -> list[str]:
-        """Fetch active sqlfluff and dbt package versions from the framework."""
-        info_line("Fetching SQLFluff requirements from framework...")
+    def _fetch_framework_versions(self) -> dict[str, str]:
+        """Fetch all active package versions from the framework's DBT_VERSIONS config.
+        
+        Returns a dict mapping pip package name to pinned spec, e.g. {"dbt-core": "dbt-core==1.9.0", ...}.
+        """
         raw = self.sf.get_framework_config("DBT_VERSIONS")
         if not raw:
             error_line("No DBT_VERSIONS configuration found in native app")
-            return []
-        
-        packages = []
+            return {}
         
-        # Map of config key -> pip package name
         version_keys = {
             "sqlfluff_versions": "sqlfluff",
             "sqlfluff_templater_versions": "sqlfluff-templater-dbt",
@@ -186,16 +190,35 @@ class DownloadService:
             "dbt_snowflake_versions": "dbt-snowflake",
         }
         
+        result: dict[str, str] = {}
         for config_key, pkg_name in version_keys.items():
             for entry in raw.get(config_key, []):
                 if entry.get("active"):
-                    packages.append(f"{pkg_name}=={entry['version']}")
+                    result[pkg_name] = f"{pkg_name}=={entry['version']}"
                     break
         
-        if not packages:
+        if not result:
             error_line("No active versions found in framework configuration")
         
-        return packages
+        return result
+
+    def get_sqlfluff_requirements(self) -> list[str]:
+        """Fetch active sqlfluff and sqlfluff-templater-dbt versions from the framework."""
+        info_line("Fetching SQLFluff requirements from framework...")
+        versions = self._fetch_framework_versions()
+        sqlfluff_packages = [versions[k] for k in ("sqlfluff", "sqlfluff-templater-dbt") if k in versions]
+        if not sqlfluff_packages:
+            error_line("No active SQLFluff versions found in framework configuration")
+        return sqlfluff_packages
+
+    def get_dbt_requirements(self) -> list[str]:
+        """Fetch active dbt-core and dbt-snowflake versions from the framework."""
+        info_line("Fetching dbt requirements from framework...")
+        versions = self._fetch_framework_versions()
+        dbt_packages = [versions[k] for k in ("dbt-core", "dbt-snowflake") if k in versions]
+        if not dbt_packages:
+            error_line("No active dbt versions found in framework configuration")
+        return dbt_packages
     
     def download_pipelines(self, platform: str = "github") -> bool:
         info_line(f"Downloading {platform} pipeline configurations...")
@@ -207,12 +230,12 @@ class DownloadService:
         pipelines = raw.get("pipelines", [])
         count = 0
         for pipe in pipelines:
-            if pipe.get("platform", "github") != platform or not pipe.get("enabled", True):
+            if pipe.get("platform", "github").lower() != platform.lower() or not pipe.get("enabled", True):
                 continue
             filename = pipe.get("filename", "")
             yaml_content = pipe.get("yaml_content", "")
             if filename and yaml_content:
-                if platform == "github":
+                if platform.lower() == "github":
                     dest = self.project_dir / ".github" / "workflows" / filename
                 else:
                     dest = self.project_dir / filename

{datasecops_cli-0.2.6 → datasecops_cli-0.2.8}/src/datasecops_cli/services/linting_service.py

@@ -10,10 +10,16 @@ class LintingService:
     def __init__(self, project_dir: Path):
         self.project_dir = project_dir
     
-    def get_installed_versions(self) -> dict[str, str]:
-        """Get currently installed versions of sqlfluff packages."""
+    def get_installed_versions(self, packages: list[str] = None) -> dict[str, str]:
+        """Get currently installed versions of the specified packages.
+        
+        Args:
+            packages: Package names to check. Defaults to sqlfluff, sqlfluff-templater-dbt, dbt-core, dbt-snowflake.
+        """
+        if packages is None:
+            packages = ["sqlfluff", "sqlfluff-templater-dbt", "dbt-core", "dbt-snowflake"]
         versions = {}
-        for package in ["sqlfluff", "sqlfluff-templater-dbt", "dbt-core", "dbt-snowflake"]:
+        for package in packages:
             result = subprocess.run(
                 ["uv", "pip", "show", package],
                 capture_output=True, text=True
@@ -28,7 +34,7 @@ class LintingService:
         return versions
     
     def install_requirements(self, packages: list[str]) -> bool:
-        """Install sqlfluff packages at specified versions."""
+        """Install packages at specified versions via uv pip."""
         if not packages:
             warning_line("No packages to install")
             return False
@@ -37,7 +43,7 @@ class LintingService:
         info_line(f"Installing: {', '.join(packages)}")
         result = subprocess.run(cmd, capture_output=False)
         if result.returncode == 0:
-            success_line("SQLFluff requirements installed successfully")
+            success_line("Packages installed successfully")
             return True
         else:
             error_line(f"uv pip install failed with exit code {result.returncode}")
@@ -75,8 +81,8 @@ class LintingService:
                 return True
             return self.install_requirements(to_install)
 
-        # No pinned versions — just ensure packages are present
-        missing = [pkg for pkg in ["sqlfluff", "sqlfluff-templater-dbt", "dbt-core", "dbt-snowflake"] if not installed.get(pkg)]
+        # No pinned versions — just ensure sqlfluff packages are present
+        missing = [pkg for pkg in ["sqlfluff", "sqlfluff-templater-dbt"] if not installed.get(pkg)]
         if not missing:
             return True
         warning_line(f"Missing packages: {', '.join(missing)}")

datasecops_cli-0.2.8/tests/test_main.py

@@ -0,0 +1,207 @@
+"""Tests for datasecops_cli.main argument parsing and download dispatch."""
+import pytest
+from unittest.mock import MagicMock, patch, call
+from pathlib import Path
+
+from datasecops_cli.main import _build_parser, _run_download, DOWNLOAD_ITEMS
+from datasecops_cli.config import Config
+
+
+class TestBuildParser:
+    """Tests for the argparse configuration."""
+
+    def test_no_args_gives_no_command(self):
+        parser = _build_parser()
+        args = parser.parse_args([])
+        assert args.command is None
+
+    def test_bootstrap_command(self):
+        parser = _build_parser()
+        args = parser.parse_args(["bootstrap"])
+        assert args.command == "bootstrap"
+
+    def test_download_single_item(self):
+        parser = _build_parser()
+        args = parser.parse_args(["download", "sqlfluff"])
+        assert args.command == "download"
+        assert args.items == ["sqlfluff"]
+
+    def test_download_multiple_items(self):
+        parser = _build_parser()
+        args = parser.parse_args(["download", "sqlfluff", "packages", "macros"])
+        assert args.command == "download"
+        assert args.items == ["sqlfluff", "packages", "macros"]
+
+    def test_download_all(self):
+        parser = _build_parser()
+        args = parser.parse_args(["download", "all"])
+        assert args.command == "download"
+        assert args.items == ["all"]
+
+    def test_download_invalid_item_exits(self):
+        parser = _build_parser()
+        with pytest.raises(SystemExit):
+            parser.parse_args(["download", "invalid"])
+
+    def test_download_no_items_exits(self):
+        parser = _build_parser()
+        with pytest.raises(SystemExit):
+            parser.parse_args(["download"])
+
+    def test_all_download_items_accepted(self):
+        parser = _build_parser()
+        for item in DOWNLOAD_ITEMS:
+            args = parser.parse_args(["download", item])
+            assert args.items == [item]
+
+
+class TestRunDownload:
+    """Tests for _run_download dispatch logic using mocked services."""
+
+    def _make_config(self, tmp_path):
+        """Create a Config with minimal valid state."""
+        config = Config()
+        config.project_dir = tmp_path
+        config.dbt_project_dir = tmp_path
+        config.profile_name = "test_profile"
+        # source_control defaults to GitHub
+        return config
+
+    @patch("datasecops_cli.main._connect_and_load")
+    def test_download_sqlfluff(self, mock_connect, tmp_path):
+        config = self._make_config(tmp_path)
+        mock_sf = MagicMock()
+        mock_connect.return_value = mock_sf
+
+        with patch("datasecops_cli.main.DownloadService") as MockDS:
+            mock_ds = MockDS.return_value
+            mock_ds.download_sqlfluff_config.return_value = True
+
+            with pytest.raises(SystemExit) as exc:
+                _run_download(config, ["sqlfluff"])
+
+            assert exc.value.code == 0
+            mock_ds.download_sqlfluff_config.assert_called_once()
+            mock_ds.download_pipelines.assert_not_called()
+            mock_ds.download_dbt_packages.assert_not_called()
+            mock_ds.download_macros.assert_not_called()
+
+    @patch("datasecops_cli.main._connect_and_load")
+    def test_download_pipelines(self, mock_connect, tmp_path):
+        config = self._make_config(tmp_path)
+        mock_sf = MagicMock()
+        mock_connect.return_value = mock_sf
+
+        with patch("datasecops_cli.main.DownloadService") as MockDS:
+            mock_ds = MockDS.return_value
+            mock_ds.download_pipelines.return_value = True
+
+            with pytest.raises(SystemExit) as exc:
+                _run_download(config, ["pipelines"])
+
+            assert exc.value.code == 0
+            mock_ds.download_pipelines.assert_called_once_with(platform="github")
+
+    @patch("datasecops_cli.main._connect_and_load")
+    def test_download_packages(self, mock_connect, tmp_path):
+        config = self._make_config(tmp_path)
+        mock_sf = MagicMock()
+        mock_connect.return_value = mock_sf
+
+        with patch("datasecops_cli.main.DownloadService") as MockDS:
+            mock_ds = MockDS.return_value
+            mock_ds.download_dbt_packages.return_value = True
+
+            with pytest.raises(SystemExit) as exc:
+                _run_download(config, ["packages"])
+
+            assert exc.value.code == 0
+            mock_ds.download_dbt_packages.assert_called_once_with(tmp_path)
+
+    @patch("datasecops_cli.main._connect_and_load")
+    def test_download_macros(self, mock_connect, tmp_path):
+        config = self._make_config(tmp_path)
+        mock_sf = MagicMock()
+        mock_connect.return_value = mock_sf
+
+        with patch("datasecops_cli.main.DownloadService") as MockDS:
+            mock_ds = MockDS.return_value
+            mock_ds.download_macros.return_value = True
+
+            with pytest.raises(SystemExit) as exc:
+                _run_download(config, ["macros"])
+
+            assert exc.value.code == 0
+            mock_ds.download_macros.assert_called_once_with("test_profile", tmp_path)
+
+    @patch("datasecops_cli.main._connect_and_load")
+    def test_download_all_expands_to_all_items(self, mock_connect, tmp_path):
+        config = self._make_config(tmp_path)
+        mock_sf = MagicMock()
+        mock_connect.return_value = mock_sf
+
+        with patch("datasecops_cli.main.DownloadService") as MockDS:
+            mock_ds = MockDS.return_value
+            mock_ds.download_sqlfluff_config.return_value = True
+            mock_ds.download_pipelines.return_value = True
+            mock_ds.download_dbt_packages.return_value = True
+            mock_ds.download_macros.return_value = True
+
+            with pytest.raises(SystemExit) as exc:
+                _run_download(config, ["all"])
+
+            assert exc.value.code == 0
+            mock_ds.download_sqlfluff_config.assert_called_once()
+            mock_ds.download_pipelines.assert_called_once()
+            mock_ds.download_dbt_packages.assert_called_once()
+            mock_ds.download_macros.assert_called_once()
+
+    @patch("datasecops_cli.main._connect_and_load")
+    def test_download_failure_exits_1(self, mock_connect, tmp_path):
+        config = self._make_config(tmp_path)
+        mock_sf = MagicMock()
+        mock_connect.return_value = mock_sf
+
+        with patch("datasecops_cli.main.DownloadService") as MockDS:
+            mock_ds = MockDS.return_value
+            mock_ds.download_sqlfluff_config.return_value = False  # failure
+
+            with pytest.raises(SystemExit) as exc:
+                _run_download(config, ["sqlfluff"])
+
+            assert exc.value.code == 1
+
+    @patch("datasecops_cli.main._connect_and_load")
+    def test_download_partial_failure_exits_1(self, mock_connect, tmp_path):
+        """If one item fails but another succeeds, exit code is still 1."""
+        config = self._make_config(tmp_path)
+        mock_sf = MagicMock()
+        mock_connect.return_value = mock_sf
+
+        with patch("datasecops_cli.main.DownloadService") as MockDS:
+            mock_ds = MockDS.return_value
+            mock_ds.download_sqlfluff_config.return_value = True
+            mock_ds.download_dbt_packages.return_value = False
+
+            with pytest.raises(SystemExit) as exc:
+                _run_download(config, ["sqlfluff", "packages"])
+
+            assert exc.value.code == 1
+
+    @patch("datasecops_cli.main._connect_and_load")
+    def test_download_pipelines_uses_configured_platform(self, mock_connect, tmp_path):
+        """Pipeline download reads platform from source_control config."""
+        config = self._make_config(tmp_path)
+        config.source_control.source_control_platform = "AzureDevOps"
+        mock_sf = MagicMock()
+        mock_connect.return_value = mock_sf
+
+        with patch("datasecops_cli.main.DownloadService") as MockDS:
+            mock_ds = MockDS.return_value
+            mock_ds.download_pipelines.return_value = True
+
+            with pytest.raises(SystemExit) as exc:
+                _run_download(config, ["pipelines"])
+
+            assert exc.value.code == 0
+            mock_ds.download_pipelines.assert_called_once_with(platform="azuredevops")