datasecops-cli 0.1.0__tar.gz → 0.2.1__tar.gz

datasecops_cli-0.2.1/CHANGELOG.md

@@ -0,0 +1,92 @@
+# Changelog
+
+All notable changes to the DataSecOps CLI are documented in this file.
+
+## [0.2.1] - 2026-05-10
+
+### Fixed
+
+- **SQLFluff config now includes `[sqlfluff:templater:dbt]` section** with `project_dir` and `profiles_dir` when downloaded from the framework, eliminating the need for `${DBT_PROFILES_DIR}` environment variable substitution
+- **Auto-download `.sqlfluff` before linting** — the lint menu now automatically downloads the config from the framework if it doesn't exist locally, instead of failing silently or erroring
+- **CI lint action handles missing `.sqlfluffconfig`** — the `tasks/lint-sql` action now only runs `envsubst` if `.sqlfluffconfig` exists, and accepts a pre-generated `.sqlfluff` file directly
+
+### Changed
+
+- `download_sqlfluff_config()` accepts an optional `profiles_dir` parameter to embed in the generated config
+- `DevelopmentMenu` and `DownloadsMenu` pass the resolved `profiles_dir` through to the download service
+- `.sqlfluff` no longer needs to be committed to client repos — it is generated on-the-fly from the framework
+
+## [0.2.0] - 2026-05-09
+
+### Added
+
+- **MCP Server** — new `datasecops-mcp` server exposes framework governance config to AI coding tools (VS Code, Cursor, Cortex Code, Claude Code)
+  - `get_branching_rules` — branch types, naming conventions, ticket requirements
+  - `get_linting_rules` — SQLFluff rules from the native app
+  - `get_dbt_packages` — approved packages with pinned versions
+  - `get_pipeline_config` — CI/CD pipeline templates (GitHub/Azure DevOps)
+  - `get_dbt_versions` — mandated dbt and SQLFluff versions
+  - `get_project_settings` — project paths, execution mode, targets
+  - `get_project_profiles` — all registered dbt projects
+  - `get_deployment_targets` — environment targets with roles and warehouses
+  - `get_available_skills` — Cortex Code skills metadata
+  - `get_default_macros` — default macros for a project profile
+  - `validate_branch_name` — validates branch names against framework conventions
+  - `get_deployment_workflow` — environment promotion flow
+  - `lint_sql` — lint a SQL file using project's SQLFluff config
+  - `fix_sql` — auto-fix linting issues in a SQL file
+  - `lint_project` — lint or fix all models in the project
+
+- **Bootstrap command** — `datasecops bootstrap` sets up a new dbt project in one step:
+  1. Runs `dbtf init` and generates `profiles.yml` from framework targets
+  2. Downloads default macros for the project profile
+  3. Downloads `.sqlfluff` linting configuration
+  4. Downloads CI/CD pipeline files (GitHub Actions or Azure DevOps)
+  5. Downloads `packages.yml` with approved versions
+  6. Runs `dbt deps`
+  7. Checks SQLFluff version requirements
+  8. Installs Cortex Code skills
+  - Also available from main menu as option [4]
+
+- **Macro downloads** — new `api.get_macros_for_profile` integration
+  - Downloads default macros (e.g., `generate_schema_name`) to `macros/` directory
+  - Available in downloads menu as option [4]
+  - Included in bootstrap process
+
+- **MCP server configuration in setup scripts** — `setup.sh` and `setup.ps1` now:
+  - Check for Node.js (needed for GitHub/Azure DevOps MCP servers)
+  - Install with MCP support (`datasecops-cli[mcp]`)
+  - Optionally install `dbt-mcp`
+  - Interactively configure MCP servers for VS Code, Cursor, or Cortex Code
+  - Support GitHub and Azure DevOps PAT configuration
+
+- **MCP server integration guide** (`mcp-servers.json`) — example config for running framework, dbt, GitHub, and Azure DevOps MCP servers together
+
+### Changed
+
+- `pyproject.toml` — added `mcp` optional dependency group and `datasecops-mcp` script entry point
+- Main menu now shows option [4] for bootstrap
+- Downloads menu reordered: macros is option [4], skills is [5], new project is [6]
+- Setup scripts install `datasecops-cli[mcp]` by default instead of `datasecops-cli`
+
+### Documentation
+
+- `docs/getting-started.md` — full setup guide for CLI + MCP servers with VS Code, Cursor, Cortex Code
+- `docs/mcp-server.md` — MCP server reference with tool documentation, architecture, and usage examples
+- `README.md` — updated with MCP server section, recommended server stack, and new documentation links
+
+## [0.1.0] - 2025-03-05
+
+### Added
+
+- Initial release of the DataSecOps CLI
+- Interactive menu-driven interface for dbt development, git operations, and configuration downloads
+- dbt commands via dbt Fusion subprocess (run, build, test, lint, compile, snapshot, freshness, docs, seed, deps, clean, debug, list, retry)
+- Git operations via GitPython (branching, commit, push, pull, rebase, squash, deploy, cherry-pick)
+- Configuration downloads from native app (SQLFluff rules, CI/CD pipelines, dbt packages, Cortex Code skills)
+- Framework-enforced branch naming conventions
+- Environment branch deployment (dev/test/prod)
+- SQLFluff linting with framework-managed rules
+- Snowflake Native App integration via stored procedures
+- Cross-platform support (Windows, Linux, macOS)
+- PyPI distribution with GitHub Actions OIDC publishing

datasecops_cli-0.2.1/PKG-INFO

@@ -0,0 +1,165 @@
+Metadata-Version: 2.4
+Name: datasecops-cli
+Version: 0.2.1
+Summary: DataSecOps Framework CLI for Snowflake Native App
+License-Expression: MIT
+License-File: LICENSE
+Requires-Python: >=3.10
+Requires-Dist: colorama>=0.4
+Requires-Dist: gitpython>=3.1
+Requires-Dist: pydantic>=2.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: snowflake-connector-python>=3.0
+Requires-Dist: sqlfluff>=3.0
+Provides-Extra: mcp
+Requires-Dist: mcp>=1.0; extra == 'mcp'
+Provides-Extra: test
+Requires-Dist: pytest-cov>=4.0; extra == 'test'
+Requires-Dist: pytest>=7.0; extra == 'test'
+Description-Content-Type: text/markdown
+
+# DataSecOps CLI
+
+A command-line interface for the [Data Engineers DataSecOps Native App](https://app.snowflake.com/marketplace) on Snowflake. Streamlines dbt development, source control workflows, and framework configuration management for teams using the DataSecOps Framework.
+
+## What is this?
+
+The DataSecOps CLI is the local developer companion to the **Data Engineers DataSecOps Native App** — a Snowflake Native App that provides governance, configuration management, and standardised development workflows for data teams.
+
+This CLI connects to the native app and gives developers:
+
+- **dbt development commands** — run, build, test, lint, and manage dbt projects via dbt Fusion
+- **Source control operations** — branching, committing, rebasing, and deploying via GitPython with naming conventions enforced by the framework
+- **Configuration downloads** — pull SQLFluff rules, CI/CD pipelines, dbt packages, and Cortex Code skills from the native app to your local project
+- **MCP server** — expose framework governance rules to AI coding assistants (VS Code, Cursor, Cortex Code, Claude Code)
+
+## Installation
+
+```bash
+pip install datasecops-cli
+```
+
+With MCP server support:
+
+```bash
+pip install "datasecops-cli[mcp]"
+```
+
+Requires Python 3.10 or later.
+
+## Prerequisites
+
+- A Snowflake connection configured in `~/.snowflake/connections.toml`
+- The **Data Engineers DataSecOps Native App** installed in your Snowflake account
+- A project profile created in the native app
+
+Optional:
+
+- **dbt Fusion** (or dbt-core with dbt-snowflake) for dbt commands
+- **Cortex Code** for skill downloads
+- **Node.js 18+** for GitHub/Azure DevOps MCP servers
+
+## Quick Start
+
+### 1. Run the setup script
+
+The setup script creates a virtual environment, installs the CLI, and writes your local configuration.
+
+**Linux / macOS:**
+
+```bash
+chmod +x setup.sh && ./setup.sh
+```
+
+**Windows (PowerShell):**
+
+```powershell
+.\setup.ps1
+```
+
+You'll be prompted for your Snowflake connection name and the native app database name.
+
+### 2. Activate the virtual environment and run
+
+```bash
+source .venv/bin/activate   # Linux/macOS
+.\.venv\Scripts\Activate.ps1  # Windows
+
+datasecops
+```
+
+## Features
+
+| Menu | Capabilities |
+|------|-------------|
+| **Development** | dbt run, build, test, lint (SQLFluff), deps, seed, compile, snapshot, freshness, docs |
+| **Git** | Branch create/checkout/delete, commit & push, rebase, squash, deploy to environment branches, cherry-pick |
+| **Downloads** | SQLFluff config, CI/CD pipelines (GitHub Actions / Azure DevOps), dbt packages, Cortex Code skills |
+
+## MCP Server
+
+The package includes an MCP (Model Context Protocol) server that exposes your framework's governance configuration to AI coding assistants. Instead of static skill files, the MCP server gives AI tools live access to your native app's current rules.
+
+### Available Tools
+
+| Category | Tools |
+|----------|-------|
+| **Configuration** | `get_branching_rules`, `get_linting_rules`, `get_dbt_packages`, `get_pipeline_config`, `get_dbt_versions` |
+| **Project** | `get_project_settings`, `get_project_profiles`, `get_deployment_targets`, `get_available_skills` |
+| **Linting** | `lint_sql`, `fix_sql`, `lint_project` |
+| **Source Control** | `validate_branch_name`, `get_deployment_workflow` |
+
+### Setup for AI Tools
+
+**Cortex Code:**
+
+```bash
+cortex mcp add datasecops-framework -- datasecops-mcp
+```
+
+**VS Code / Cursor** — add to `.vscode/mcp.json` or `.cursor/mcp.json`:
+
+```json
+{
+  "mcpServers": {
+    "datasecops-framework": {
+      "command": "datasecops-mcp",
+      "args": []
+    }
+  }
+}
+```
+
+### Recommended MCP Server Stack
+
+For full integration with Snowflake, dbt, and your source control platform, configure these MCP servers alongside the framework server:
+
+| Server | Purpose | Install |
+|--------|---------|---------|
+| **datasecops-framework** | Governance rules, linting, branch validation | Included (`datasecops-mcp`) |
+| **dbt** | Lineage, model discovery, codegen, semantic layer | `pip install dbt-mcp` |
+| **GitHub** | PRs, issues, CI checks, releases | `npx -y @modelcontextprotocol/server-github` |
+| **Azure DevOps** | PRs, pipelines, work items, boards | `npx -y @tiberriver256/mcp-server-azure-devops` |
+
+See the [Getting Started Guide](docs/getting-started.md) for full configuration instructions for each editor and platform.
+
+## Configuration
+
+The CLI reads from a `.datasecops.yml` file in your project root (created by the setup script):
+
+```yaml
+connection_name: "my_connection"
+app_database: "DATA_ENGINEERS_DATASECOPS_FRAMEWORK"
+```
+
+Project profiles, linting rules, pipeline templates, and deployment targets are all managed centrally in the native app and pulled down by the CLI.
+
+## Documentation
+
+- [Getting Started Guide](docs/getting-started.md) — install CLI, configure MCP servers for VS Code/Cursor/Cortex Code with Snowflake, dbt, and GitHub/Azure DevOps
+- [MCP Server Reference](docs/mcp-server.md) — full tool documentation, architecture, and usage examples
+- [Development Guide](DEVELOPMENT.md) — project structure, setup scripts, native app API reference, and publishing details
+
+## License
+
+MIT

datasecops_cli-0.2.1/README.md

@@ -0,0 +1,145 @@
+# DataSecOps CLI
+
+A command-line interface for the [Data Engineers DataSecOps Native App](https://app.snowflake.com/marketplace) on Snowflake. Streamlines dbt development, source control workflows, and framework configuration management for teams using the DataSecOps Framework.
+
+## What is this?
+
+The DataSecOps CLI is the local developer companion to the **Data Engineers DataSecOps Native App** — a Snowflake Native App that provides governance, configuration management, and standardised development workflows for data teams.
+
+This CLI connects to the native app and gives developers:
+
+- **dbt development commands** — run, build, test, lint, and manage dbt projects via dbt Fusion
+- **Source control operations** — branching, committing, rebasing, and deploying via GitPython with naming conventions enforced by the framework
+- **Configuration downloads** — pull SQLFluff rules, CI/CD pipelines, dbt packages, and Cortex Code skills from the native app to your local project
+- **MCP server** — expose framework governance rules to AI coding assistants (VS Code, Cursor, Cortex Code, Claude Code)
+
+## Installation
+
+```bash
+pip install datasecops-cli
+```
+
+With MCP server support:
+
+```bash
+pip install "datasecops-cli[mcp]"
+```
+
+Requires Python 3.10 or later.
+
+## Prerequisites
+
+- A Snowflake connection configured in `~/.snowflake/connections.toml`
+- The **Data Engineers DataSecOps Native App** installed in your Snowflake account
+- A project profile created in the native app
+
+Optional:
+
+- **dbt Fusion** (or dbt-core with dbt-snowflake) for dbt commands
+- **Cortex Code** for skill downloads
+- **Node.js 18+** for GitHub/Azure DevOps MCP servers
+
+## Quick Start
+
+### 1. Run the setup script
+
+The setup script creates a virtual environment, installs the CLI, and writes your local configuration.
+
+**Linux / macOS:**
+
+```bash
+chmod +x setup.sh && ./setup.sh
+```
+
+**Windows (PowerShell):**
+
+```powershell
+.\setup.ps1
+```
+
+You'll be prompted for your Snowflake connection name and the native app database name.
+
+### 2. Activate the virtual environment and run
+
+```bash
+source .venv/bin/activate   # Linux/macOS
+.\.venv\Scripts\Activate.ps1  # Windows
+
+datasecops
+```
+
+## Features
+
+| Menu | Capabilities |
+|------|-------------|
+| **Development** | dbt run, build, test, lint (SQLFluff), deps, seed, compile, snapshot, freshness, docs |
+| **Git** | Branch create/checkout/delete, commit & push, rebase, squash, deploy to environment branches, cherry-pick |
+| **Downloads** | SQLFluff config, CI/CD pipelines (GitHub Actions / Azure DevOps), dbt packages, Cortex Code skills |
+
+## MCP Server
+
+The package includes an MCP (Model Context Protocol) server that exposes your framework's governance configuration to AI coding assistants. Instead of static skill files, the MCP server gives AI tools live access to your native app's current rules.
+
+### Available Tools
+
+| Category | Tools |
+|----------|-------|
+| **Configuration** | `get_branching_rules`, `get_linting_rules`, `get_dbt_packages`, `get_pipeline_config`, `get_dbt_versions` |
+| **Project** | `get_project_settings`, `get_project_profiles`, `get_deployment_targets`, `get_available_skills` |
+| **Linting** | `lint_sql`, `fix_sql`, `lint_project` |
+| **Source Control** | `validate_branch_name`, `get_deployment_workflow` |
+
+### Setup for AI Tools
+
+**Cortex Code:**
+
+```bash
+cortex mcp add datasecops-framework -- datasecops-mcp
+```
+
+**VS Code / Cursor** — add to `.vscode/mcp.json` or `.cursor/mcp.json`:
+
+```json
+{
+  "mcpServers": {
+    "datasecops-framework": {
+      "command": "datasecops-mcp",
+      "args": []
+    }
+  }
+}
+```
+
+### Recommended MCP Server Stack
+
+For full integration with Snowflake, dbt, and your source control platform, configure these MCP servers alongside the framework server:
+
+| Server | Purpose | Install |
+|--------|---------|---------|
+| **datasecops-framework** | Governance rules, linting, branch validation | Included (`datasecops-mcp`) |
+| **dbt** | Lineage, model discovery, codegen, semantic layer | `pip install dbt-mcp` |
+| **GitHub** | PRs, issues, CI checks, releases | `npx -y @modelcontextprotocol/server-github` |
+| **Azure DevOps** | PRs, pipelines, work items, boards | `npx -y @tiberriver256/mcp-server-azure-devops` |
+
+See the [Getting Started Guide](docs/getting-started.md) for full configuration instructions for each editor and platform.
+
+## Configuration
+
+The CLI reads from a `.datasecops.yml` file in your project root (created by the setup script):
+
+```yaml
+connection_name: "my_connection"
+app_database: "DATA_ENGINEERS_DATASECOPS_FRAMEWORK"
+```
+
+Project profiles, linting rules, pipeline templates, and deployment targets are all managed centrally in the native app and pulled down by the CLI.
+
+## Documentation
+
+- [Getting Started Guide](docs/getting-started.md) — install CLI, configure MCP servers for VS Code/Cursor/Cortex Code with Snowflake, dbt, and GitHub/Azure DevOps
+- [MCP Server Reference](docs/mcp-server.md) — full tool documentation, architecture, and usage examples
+- [Development Guide](DEVELOPMENT.md) — project structure, setup scripts, native app API reference, and publishing details
+
+## License
+
+MIT