thothctl 0.16.2__tar.gz → 0.16.3__tar.gz

{thothctl-0.16.2 → thothctl-0.16.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: thothctl
-Version: 0.16.2
+Version: 0.16.3
 Summary: A CLI for Developer Control Plane. Accelerate your cloud IaC deployments.
 Project-URL: SourceCode, https://github.com/thothforge/thothctl
 Project-URL: HomePage, https://github.com/thothforge/thothctl

{thothctl-0.16.2 → thothctl-0.16.3}/docs/framework/commands/check/check_project_iac.md

@@ -17,9 +17,99 @@ Options:
                                   Type of IaC check to perform [default: structure]
   -p, --project-type [stack|module]
                                   Project type: stack or module [default: stack]
+  --org-policy TEXT               Organization policy source (Git URL or local path)
+  --enforcement [soft|hard]       Enforcement mode: soft (report) or hard (fail pipeline)
   --help                          Show this message and exit.
 ```
 
+## Organization Policy Enforcement
+
+ThothCTL can enforce organizational standards that projects **cannot override**. This ensures all projects in your organization follow the same structure, naming, and tagging rules — regardless of what individual `.thothcf.toml` files contain.
+
+### How It Works
+
+1. Set `THOTH_ORG_POLICY` to your org policy Git repo (or pass `--org-policy`)
+2. The repo contains `rules/base.toml` + `rules/<project_type>.toml`
+3. ThothCTL merges org rules with project rules — **mandatory org rules cannot be weakened**
+4. Violations are reported with enforcement level (mandatory = fail, recommended = warn)
+
+### Usage
+
+```bash
+# Via env var (CI/CD recommended)
+export THOTH_ORG_POLICY=https://github.com/myorg/org-policies.git
+thothctl check project iac --enforcement hard
+
+# Pin to a version
+export THOTH_ORG_POLICY=https://github.com/myorg/org-policies.git@v1.0
+thothctl check project iac --enforcement hard
+
+# Via flag
+thothctl check project iac --org-policy /path/to/org-policies --enforcement hard
+
+# Local path (development)
+thothctl check project iac --org-policy ../org-iac-policies
+```
+
+### Enforcement Levels
+
+| Level | Behavior | Project Can Override? |
+|-------|----------|---------------------|
+| `mandatory` | Fails pipeline with `--enforcement hard` | ❌ No |
+| `recommended` | Warning only | ⚠️ Can opt-out |
+| `informational` | Report only | ✅ Yes |
+
+### Org Policy Repo Structure
+
+```
+org-policies/
+├── rules/                    # ThothCTL structural rules
+│   ├── base.toml             # All project types
+│   ├── terraform-terragrunt.toml
+│   ├── terraform_module.toml
+│   └── cdkv2.toml
+├── shared/policy/            # OPA/Rego policies (used by scan iac -t opa)
+│   ├── naming.rego
+│   ├── tagging.rego
+│   └── regions.rego
+└── README.md
+```
+
+The same repo serves both:
+- **`thothctl check project iac`** → reads `rules/`
+- **`thothctl scan iac -t opa`** → reads `shared/policy/` (auto-discovered via `THOTH_ORG_POLICY`)
+
+### Example Output
+
+```
+📜 Loading org policy from: https://github.com/myorg/org-policies.git
+
+❌ Mandatory Violations
+┌────────────────────────────────────┬─────────────────┬─────────┐
+│ Rule                               │ Expected        │ Found   │
+├────────────────────────────────────┼─────────────────┼─────────┤
+│ project_structure.folders.docs     │ docs/ exists    │ missing │
+│ project_structure.root_files       │ .pre-commit...  │ missing │
+└────────────────────────────────────┴─────────────────┴─────────┘
+
+⚠️ Recommendations
+┌────────────────────────────────────┬─────────────────┬─────────┐
+│ Rule                               │ Expected        │ Found   │
+├────────────────────────────────────┼─────────────────┼─────────┤
+│ project_structure.folders.common   │ common/ exists  │ missing │
+└────────────────────────────────────┴─────────────────┴─────────┘
+```
+
+### CI/CD Integration
+
+```yaml
+# GitHub Actions
+- name: Check org compliance
+  run: thothctl check project iac --enforcement hard
+  env:
+    THOTH_ORG_POLICY: https://github.com/myorg/org-policies.git@v1.0
+```
+
 ## Project Types
 
 ### Stack Projects (`-p stack`)

{thothctl-0.16.2 → thothctl-0.16.3}/docs/framework/commands/scan/scan_iac.md

@@ -58,13 +58,101 @@ thothctl scan iac -t kics
 
 ### Terraform-compliance
 
-[Terraform-compliance](https://terraform-compliance.com/) is a lightweight, security and compliance focused test framework against Terraform that enables negative testing capability for your infrastructure-as-code.
+[Terraform-compliance](https://terraform-compliance.com/) (v1.15.1) is a lightweight BDD test framework that evaluates `tfplan.json` files against Gherkin `.feature` files. It enables human-readable compliance scenarios like "S3 buckets must have encryption enabled."
+
+#### Prerequisites
 
 ```bash
-# Scan with Terraform-compliance
+pip install terraform-compliance
+```
+
+Requires `tfplan.json` files in your project (generated with `terraform show -json tfplan.binary > tfplan.json`).
+
+#### Usage
+
+```bash
+# Local features directory
+thothctl scan iac -t terraform-compliance -o "features_dir=features"
+
+# Git repository with subpath (//subpath syntax)
+thothctl scan iac -t terraform-compliance -o "features_dir=https://github.com/myorg/org-policies.git//compliance/features"
+
+# Git repository (auto-discovers compliance/features/ or features/)
+thothctl scan iac -t terraform-compliance -o "features_dir=https://github.com/myorg/org-policies.git"
+
+# SSH Git URL with subpath
+thothctl scan iac -t terraform-compliance -o "features_dir=git@github.com:myorg/compliance.git//features"
+
+# Auto-discover from THOTH_ORG_POLICY (looks in compliance/features/)
+export THOTH_ORG_POLICY=https://github.com/myorg/org-policies.git
 thothctl scan iac -t terraform-compliance
 ```
 
+#### Features Resolution
+
+| Priority | Source | Example |
+|----------|--------|---------|
+| 1 | **Git URL with `//subpath`** | `https://github.com/myorg/policies.git//compliance/features` |
+| 2 | **Git URL** (auto-discovers `compliance/features/` or `features/`) | `https://github.com/myorg/policies.git` |
+| 3 | **Relative to project** | `features/` |
+| 4 | **Absolute path** | `/shared/compliance/features` |
+| 5 | **`THOTH_ORG_POLICY` env** → `compliance/features/` | Auto-discovered |
+
+#### Writing Feature Files
+
+Feature files use Gherkin syntax:
+
+```gherkin
+Feature: Ensure encryption is enabled for all storage resources
+
+  Scenario: S3 buckets must have encryption
+    Given I have aws_s3_bucket defined
+    Then it must have server_side_encryption_configuration
+
+  Scenario: RDS instances must be encrypted
+    Given I have aws_db_instance defined
+    Then it must have storage_encrypted
+    And its value must be true
+```
+
+```gherkin
+Feature: Ensure all resources have required tags
+
+  Scenario Outline: Resources must have mandatory tags
+    Given I have resource that supports tags defined
+    Then it must have tags
+    And it must contain <tag>
+
+    Examples:
+      | tag         |
+      | Environment |
+      | Owner       |
+      | Project     |
+```
+
+#### Per-Stack Scanning
+
+ThothCTL finds all `tfplan.json` files in your project and runs terraform-compliance against each one individually. Results are aggregated into:
+
+- Per-stack HTML reports at `Reports/terraform-compliance/html_reports/`
+- Unified `scan_report.html` alongside other tools
+
+#### Organization Policy Integration
+
+Store feature files in your org policy repo alongside OPA policies:
+
+```
+org-policies/
+├── compliance/features/     ← terraform-compliance features
+│   ├── encryption.feature
+│   ├── tagging.feature
+│   └── networking.feature
+├── shared/policy/           ← OPA/Rego policies
+└── rules/                   ← ThothCTL structure rules
+```
+
+Both tools share the same `THOTH_ORG_POLICY` env var.
+
 ### OPA / Conftest
 
 [Open Policy Agent (OPA)](https://www.openpolicyagent.org/) is a CNCF-graduated general-purpose policy engine. ThothCTL integrates OPA through two modes:

{thothctl-0.16.2 → thothctl-0.16.3}/docs/framework/commands/scan/scan_overview.md

@@ -79,10 +79,12 @@ ThothCTL automatically tracks scan results in `~/.thothcf/scan_history.db` (SQLi
 | **Checkov** | Static analysis with built-in rules | `checkov` binary |
 | **Trivy** | Vulnerability and misconfiguration detection | `trivy` binary |
 | **KICS** | Static analysis via Docker | Docker |
-| **Terraform-compliance** | BDD-style compliance testing | `terraform-compliance` binary |
+| **Terraform-compliance** | BDD-style compliance testing against tfplan.json | `terraform-compliance` (pip) |
 | **OPA/Conftest** | Custom policy evaluation with Rego | `conftest` and/or `opa` binary |
 
-Each tool has its own strengths. Combine built-in rule scanners (Checkov, Trivy) with custom policy tools (OPA) for comprehensive coverage.
+Each tool has its own strengths. Combine built-in rule scanners (Checkov, Trivy) with custom policy tools (OPA, Terraform-compliance) for comprehensive coverage.
+
+**Organization Policy Repo**: Set `THOTH_ORG_POLICY` env var to point all policy tools (OPA, terraform-compliance, project structure rules) to a single centralized governance repository.
 
 ## Next Steps
 

{thothctl-0.16.2 → thothctl-0.16.3}/docs/framework/use_cases/tasks/create_template.md

@@ -269,8 +269,8 @@ content = ["main.tf", "variables.tf", "outputs.tf"]
 
 ## Related Documentation
 
-- [Template Engine Overview](../../template_engine/template_engine.md)
-- [GitHub Templates](../../template_engine/github_templates.md)
-- [Project Convert](../commands/project/project_convert.md)
-- [Project Upgrade](../commands/project/project_upgrade.md)
+- [Template Engine Overview](../../../template_engine/template_engine.md)
+- [GitHub Templates](../../../template_engine/github_templates.md)
+- [Project Convert](../../commands/project/project_convert.md)
+- [Project Upgrade](../../commands/project/project_upgrade.md)
 - [Platform Engineering Templates](../platform_engineering_templates.md)

{thothctl-0.16.2 → thothctl-0.16.3}/docs/index.md

@@ -52,6 +52,8 @@ pip install thothctl
 | **Security** | [Checkov](https://www.checkov.io/) | Native (pip) |
 | **Security** | [Trivy](https://trivy.dev/) | CLI binary |
 | **Security** | [KICS](https://docs.kics.io/) | Docker container |
+| **Compliance** | [Terraform-compliance](https://terraform-compliance.com/) | CLI binary |
+| **Policy** | [OPA/Conftest](https://www.openpolicyagent.org/) | CLI binary |
 | **Docs** | [Terraform-docs](https://terraform-docs.io/) | CLI binary |
 | **AI** | [OpenAI](https://platform.openai.com/) | GPT-4 Turbo |
 | **AI** | [AWS Bedrock](https://aws.amazon.com/bedrock/) | Claude Sonnet (InvokeModel + Agent) |

{thothctl-0.16.2 → thothctl-0.16.3}/src/thothctl/commands/check/commands/project/iac.py

@@ -1,4 +1,5 @@
 import logging
+import os
 import click
 import sys
 import io
@@ -173,6 +174,8 @@ class CheckProjectIaCCommand(ClickCommand):
         ctx = click.get_current_context()
         directory = ctx.obj.get("CODE_DIRECTORY", ".")
         project_type = kwargs.get('project_type', 'stack')
+        org_policy = kwargs.get('org_policy')
+        enforcement = kwargs.get('enforcement', 'soft')
         
         # Create header with project type
         header_text = f"🏗️ Infrastructure as Code {'Module' if project_type == 'module' else 'Stack'} Structure Check"
@@ -184,6 +187,9 @@ class CheckProjectIaCCommand(ClickCommand):
         ))
         
         try:
+            # Run org policy check if source is provided
+            org_violations = self._check_org_policy(directory, project_type, org_policy)
+
             # Capture stdout to format it nicely
             captured_output = io.StringIO()
             
@@ -196,13 +202,19 @@ class CheckProjectIaCCommand(ClickCommand):
                         project_type=project_type
                     )
                 except SystemExit as e:
-                    # Handle sys.exit() from validation service
                     result = e.code == 0
             
             # Format and display the captured output
             output = captured_output.getvalue()
             if output.strip():
                 self._format_validation_output(output)
+
+            # Display org policy results
+            if org_violations is not None:
+                self._display_org_violations(org_violations, enforcement)
+                mandatory_fails = [v for v in org_violations if v.enforcement == "mandatory"]
+                if mandatory_fails and enforcement == "hard":
+                    result = False
             
             # Create summary
             if result:
@@ -224,8 +236,8 @@ class CheckProjectIaCCommand(ClickCommand):
             self.console.print()
             self.console.print(summary_panel)
             
-            # Only exit with error code in strict mode
-            if not result and kwargs.get('mode') == 'strict':
+            # Exit with error code if hard enforcement has mandatory violations
+            if not result and (kwargs.get('mode') == 'strict' or enforcement == 'hard'):
                 exit(1)
                 
         except Exception as e:
@@ -233,6 +245,69 @@ class CheckProjectIaCCommand(ClickCommand):
             self.logger.error(f"Failed to execute IaC project check: {str(e)}")
             raise
 
+    def _check_org_policy(self, directory: str, project_type: str, org_policy=None):
+        """Check project against organizational policy if available."""
+        from .....services.check.org_policy_loader import get_org_policy_path, resolve_rules_dir
+        from .....services.check.rule_merger import load_org_rules, merge_with_project, evaluate
+
+        org_path = get_org_policy_path(org_policy)
+        if not org_path:
+            return None
+
+        rules_dir = resolve_rules_dir(org_path)
+        if not rules_dir:
+            logger.info(f"No rules/ directory in org policy at {org_path}")
+            return None
+
+        self.console.print(f"[blue]📜 Loading org policy from: {org_path}[/blue]")
+
+        # Map CLI project_type to toml filename
+        type_map = {"stack": "terraform-terragrunt", "module": "terraform_module"}
+        rule_type = type_map.get(project_type, project_type)
+
+        ruleset = load_org_rules(rules_dir, rule_type)
+        project_toml = os.path.join(directory, ".thothcf.toml")
+        ruleset = merge_with_project(ruleset, project_toml)
+
+        return evaluate(ruleset, directory)
+
+    def _display_org_violations(self, violations, enforcement: str):
+        """Display org policy violations in a Rich table."""
+        if not violations:
+            self.console.print(Panel(
+                "✅ [green]Organization policy check passed[/green]",
+                title="Org Policy",
+                style="green",
+                box=box.ROUNDED,
+            ))
+            return
+
+        mandatory = [v for v in violations if v.enforcement == "mandatory"]
+        recommended = [v for v in violations if v.enforcement == "recommended"]
+        info = [v for v in violations if v.enforcement == "informational"]
+
+        if mandatory:
+            table = Table(title="❌ Mandatory Violations", box=box.ROUNDED, header_style="bold red")
+            table.add_column("Rule", style="cyan")
+            table.add_column("Expected", style="green")
+            table.add_column("Found", style="red")
+            for v in mandatory:
+                table.add_row(v.rule, v.expected, v.found)
+            self.console.print(table)
+
+        if recommended:
+            table = Table(title="⚠️ Recommendations", box=box.ROUNDED, header_style="bold yellow")
+            table.add_column("Rule", style="cyan")
+            table.add_column("Expected", style="green")
+            table.add_column("Found", style="yellow")
+            for v in recommended:
+                table.add_row(v.rule, v.expected, v.found)
+            self.console.print(table)
+
+        if info:
+            for v in info:
+                self.console.print(f"  ℹ️  {v.rule}: {v.expected} (found: {v.found})")
+
     def _validate_project_structure(self, directory: str, mode: str = "soft", check_type: str = "structure", project_type: str = "stack") -> bool:
         """Validate the IaC project structure
         
@@ -269,4 +344,15 @@ cli = CheckProjectIaCCommand.as_click_command(
         type=click.Choice(["stack", "module"], case_sensitive=False),
         default="stack"
     ),
+    click.option(
+        "--org-policy",
+        help="Organization policy source (Git URL or local path). Also reads THOTH_ORG_POLICY env var.",
+        default=None,
+    ),
+    click.option(
+        "--enforcement",
+        help="Enforcement mode: soft (report only) or hard (fail on mandatory violations)",
+        type=click.Choice(["soft", "hard"], case_sensitive=False),
+        default="soft",
+    ),
 )

{thothctl-0.16.2 → thothctl-0.16.3}/src/thothctl/core/version_tools.py

@@ -38,7 +38,7 @@ version_tools = """[
   },
   {
     "name": "terraform-compliance",
-    "version": "1.13.0"
+    "version": "1.15.1"
   },
   {
     "name": "terramate",

thothctl-0.16.3/src/thothctl/services/check/org_policy_loader.py

@@ -0,0 +1,95 @@
+"""Organizational Policy Loader — fetches and caches org policy repo."""
+import hashlib
+import logging
+import os
+from pathlib import Path
+from typing import Optional
+
+logger = logging.getLogger(__name__)
+
+CACHE_DIR = Path.home() / ".thothcf" / ".policy_cache"
+
+
+def get_org_policy_path(org_policy: Optional[str] = None) -> Optional[str]:
+    """Resolve org policy repo path. Clones/caches if Git URL.
+
+    Resolution:
+        1. Explicit --org-policy argument
+        2. THOTH_ORG_POLICY env var
+        3. None (no org policy)
+
+    Returns:
+        Absolute path to cached org policy repo, or None.
+    """
+    source = org_policy or os.environ.get("THOTH_ORG_POLICY")
+    if not source:
+        return None
+
+    # If it's already a local path
+    if os.path.isdir(source):
+        return os.path.abspath(source)
+
+    # Git URL — clone/cache
+    if _is_git_url(source):
+        return _clone_or_pull(source)
+
+    return None
+
+
+def resolve_rules_dir(org_path: str) -> Optional[str]:
+    """Get the rules/ directory from an org policy repo."""
+    rules_dir = os.path.join(org_path, "rules")
+    return rules_dir if os.path.isdir(rules_dir) else None
+
+
+def resolve_policy_dir(org_path: str) -> Optional[str]:
+    """Get the policy/ directory (OPA/Rego) from an org policy repo."""
+    # Check policy/ first, then shared/policy/ (common convention)
+    for candidate in ["policy", os.path.join("shared", "policy")]:
+        policy_dir = os.path.join(org_path, candidate)
+        if os.path.isdir(policy_dir):
+            return policy_dir
+    return None
+
+
+def _is_git_url(value: str) -> bool:
+    return value.startswith(("https://", "git@", "ssh://", "git://"))
+
+
+def _clone_or_pull(repo_url: str) -> Optional[str]:
+    """Clone or update a Git repo to local cache."""
+    try:
+        import git
+    except ImportError:
+        logger.error("GitPython required. Install: pip install gitpython")
+        return None
+
+    # Parse optional @ref
+    ref = None
+    if "@" in repo_url and not repo_url.startswith("git@"):
+        repo_url, ref = repo_url.rsplit("@", 1)
+    elif repo_url.startswith("git@") and repo_url.count("@") > 1:
+        repo_url, ref = repo_url.rsplit("@", 1)
+
+    url_hash = hashlib.sha256(repo_url.encode()).hexdigest()[:12]
+    cache_path = CACHE_DIR / url_hash
+    CACHE_DIR.mkdir(parents=True, exist_ok=True)
+
+    try:
+        if (cache_path / ".git").exists():
+            repo = git.Repo(cache_path)
+            repo.remotes.origin.fetch()
+            if ref:
+                repo.git.checkout(ref)
+            else:
+                repo.remotes.origin.pull()
+        else:
+            kwargs = {"depth": 1} if not ref else {}
+            repo = git.Repo.clone_from(repo_url, cache_path, **kwargs)
+            if ref:
+                repo.git.checkout(ref)
+
+        return str(cache_path)
+    except Exception as e:
+        logger.error(f"Failed to clone org policy repo: {e}")
+        return None