tff-core 0.2.0__tar.gz

tff_core-0.2.0/.gitignore

@@ -0,0 +1,10 @@
+.venv/
+__pycache__/
+*.egg-info/
+.pytest_cache/
+.ruff_cache/
+dist/
+build/
+.cache/
+.coverage
+coverage.xml

tff_core-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Bart Schuijt
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

tff_core-0.2.0/PKG-INFO

@@ -0,0 +1,224 @@
+Metadata-Version: 2.4
+Name: tff-core
+Version: 0.2.0
+Summary: Core rules and fitness functions engine for transformation projects
+Author-email: Bart Schuijt <schuijt.bart@gmail.com>
+License: MIT
+License-File: LICENSE
+Requires-Python: >=3.12
+Requires-Dist: pydantic>=2.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: rich>=13.0
+Requires-Dist: sqlglot>=23.0
+Description-Content-Type: text/markdown
+
+# sqlmesh-ff
+
+[![PyPI version](https://img.shields.io/pypi/v/sqlmesh-ff.svg)](https://pypi.org/project/sqlmesh-ff/)
+[![Python versions](https://img.shields.io/pypi/pyversions/sqlmesh-ff.svg)](https://pypi.org/project/sqlmesh-ff/)
+
+Configurable fitness functions plugin for [SQLMesh](https://sqlmesh.com) projects.
+
+Ships SQLMesh linter rules (classification macros, SQL complexity, metadata, naming) and architectural checks (layer integrity, custom exclusions, schema contracts, dependency graph) with a unified Rich lint report.
+
+## Installation
+
+```bash
+# Install from PyPI:
+uv add sqlmesh-ff
+
+# Or using pip:
+pip install sqlmesh-ff
+```
+
+## Quick start
+
+1. Add `fitness_functions.yaml` to your SQLMesh project root (see [Configuration](#configuration)).
+2. Add a small `config.py` bootstrap (see [Where configuration lives](#where-configuration-lives)) — SQLMesh requires the loader as a Python class and cannot load `config.py` and `config.yaml` in the same folder.
+3. Run lint:
+
+```bash
+sqlmesh-ff lint
+```
+
+## Where configuration lives
+
+There are three layers. Only the YAML/JSON files in your project are user-editable settings.
+
+| Layer | File | Role | You edit this? |
+|-------|------|------|----------------|
+| Plugin defaults | `sqlmesh_ff/config.py` (installed package) | Pydantic schema and built-in defaults (e.g. `fan_out_warn: 15`) | No — library code, never overwritten |
+| SQLMesh project | `settings.yaml` | Gateways, `linter.rules`, variables, CI/CD bot | Yes — normal SQLMesh config |
+| Fitness functions | `fitness_functions.yaml` | Thresholds, rule toggles, column naming/type rules, paths to JSON data | Yes — main FF config |
+| Loader bootstrap | `config.py` (project root) | Loads `settings.yaml` and registers `FitnessLoader` | Rarely — ~15 lines of wiring |
+| Contract data | `linter_contract_groups.json`, `linter_exclusions.json` | Repo-specific schema parity and dependency exclusions | Yes — project data |
+
+**Merge order for fitness settings:** plugin defaults → `fitness_functions.yaml` → optional `loader_kwargs` overrides in `config.py`. Your YAML always wins over plugin defaults. The project `config.py` does not hold fitness thresholds — it only points at `fitness_functions.yaml`.
+
+**Why `config.py` exists:** SQLMesh accepts `loader: FitnessLoader` only as a Python class, not as a YAML string. Because SQLMesh rejects having both `config.py` and `config.yaml` in one folder, projects use `settings.yaml` (SQLMesh settings) plus `config.py` (loader registration).
+
+Example bootstrap:
+
+```python
+from pathlib import Path
+
+from sqlmesh.core.config import Config
+from sqlmesh.utils.yaml import load as yaml_load
+from sqlmesh_ff.loader import FitnessLoader
+
+_settings = yaml_load(Path(__file__).parent / "settings.yaml")
+config = Config.parse_obj(_settings).update_with({
+    "loader": FitnessLoader,
+    "loader_kwargs": {"fitness_functions_config": "fitness_functions.yaml"},
+})
+```
+
+Enable individual SQLMesh rules in `settings.yaml` under `linter.rules` / `linter.warn_rules`.
+
+## Configuration
+
+Fitness function settings live in `fitness_functions.yaml` at the project root. Override the file path or individual keys via `loader_kwargs` in `config.py` (advanced — most projects only set `fitness_functions_config`).
+
+### Example `fitness_functions.yaml`
+
+```yaml
+contract_groups_path: linter_contract_groups.json
+exclusions_path: linter_exclusions.json
+
+layers:
+  order: [sources, derived, core, marts, export]
+
+checks:
+  layer_integrity: { enabled: true }
+  custom_exclusions: { enabled: true }
+  schema_contracts: { enabled: true }
+  dependency_graph:
+    enabled: true
+    fan_out_warn: 15
+    fan_out_fail: 25
+    fan_in_warn: 10
+
+rules:
+  classification_macros:
+    enabled: true
+    skip_layers: [sources]
+    columns:
+      product_type: "@product_type\\b|@PRODUCT_TYPE\\b"
+  sql_complexity:
+    enabled: true
+    thresholds:
+      decision_points: [15, 25]
+      cte_count: [8, 12]
+      join_count: [8, 12]
+      line_count: [250, 400]
+  mart_naming:
+    enabled: true
+    layer_name: marts
+    rule: prefix_with_subdirectory
+  column_names:
+    enabled: true
+    replacements: {}
+  column_types:
+    enabled: true
+    rules: []
+    equivalent_types:
+      text: [text, varchar]
+  metadata:
+    owner: true
+    description: true
+    grain: true
+  filename_equals_modelname:
+    enabled: true
+```
+
+### Project-specific JSON
+
+Keep repo-specific contract and exclusion data in your project:
+
+- `linter_contract_groups.json` — cross-model schema parity groups
+- `linter_exclusions.json` — blocked dependency patterns and allowed exceptions
+
+Reference their paths from `fitness_functions.yaml`. The plugin ships generic engines only; examples live in this README.
+
+### Rule name mapping
+
+SQLMesh uses lowercase class names in `linter.rules`:
+
+| Config key | SQLMesh rule name |
+|------------|-------------------|
+| `classification_macros` | `classificationmacros` |
+| `sql_complexity` | `sqlcomplexity` |
+| `mart_naming` | `martmodelnamingconvention` |
+| `column_names` | `columnnames` |
+| `column_types` | `columntypes` |
+| `metadata.owner` | `nomissingowner` |
+| `metadata.description` | `nomissingdescription` |
+| `metadata.grain` | `nomissinggrain` |
+| `filename_equals_modelname` | `filenameequalsmodelname` |
+
+## CLI
+
+```
+sqlmesh-ff lint [--project PATH] [--config PATH] [--checks CHECK,...] [--fail-level error|warning] [--group-by connascence|model]
+```
+
+- **Default:** all enabled checks plus SQLMesh linter rules
+- **`--checks layer_integrity,custom_exclusions`:** run subset (for pre-push hooks)
+- **`--fail-level warning`:** treat warnings as failures
+- **`--group-by connascence|model`:** change how violations are grouped in the report (default: `connascence`)
+
+## Integration example
+
+Example overrides. `api_request` should always be named `api_call`. `_id` columns should always be of type `text` and `is_` columns should always be of type `boolean`.
+
+```yaml
+column_names:
+  replacements:
+    api_request: api_call
+column_types:
+  rules:
+    - name: id_is_text
+      pattern: "_id$"
+      data_type: text
+    - name: boolean
+      pattern: "^is_"
+      data_type: boolean
+```
+
+## Examples
+
+A complete, runnable example project showcasing the configuration of `sqlmesh-ff` rules, exclusions, contracts, and a continuous integration workflow is located in the [examples/](file:///Users/bartschuijt/git/sqlmesh-ff/examples/) directory.
+
+To run the linter against the example project locally, run:
+```bash
+sqlmesh-ff lint --project examples/minimal-sqlmesh-project
+```
+
+See [examples/minimal-sqlmesh-project/fitness_functions.yaml](file:///Users/bartschuijt/git/sqlmesh-ff/examples/minimal-sqlmesh-project/fitness_functions.yaml) to inspect the configured rules.
+
+## Development
+
+Initialize your local environment and configure the Git pre-push hook:
+```bash
+make init
+```
+
+Run linter, tests, or check diff coverage:
+```bash
+make lint
+make test
+make coverage
+```
+
+### Releases and PR titles
+
+Releases are automated with [release-please](https://github.com/googleapis/release-please) on merges to `main`. Use [Conventional Commits](https://www.conventionalcommits.org/) in PR titles so changelog entries and semver bumps are correct.
+
+PR titles must start with a type prefix, for example:
+
+- `feat: add dependency graph fan-in check`
+- `fix: remove unused import in loader tests`
+- `docs: document fitness_functions.yaml merge order`
+- `ci: add release-please workflow`
+
+Supported types include `feat`, `fix`, `docs`, `style`, `refactor`, `perf`, `test`, `build`, `ci`, and `chore`. The PR title check in CI enforces this format.

tff_core-0.2.0/README.md

@@ -0,0 +1,210 @@
+# sqlmesh-ff
+
+[![PyPI version](https://img.shields.io/pypi/v/sqlmesh-ff.svg)](https://pypi.org/project/sqlmesh-ff/)
+[![Python versions](https://img.shields.io/pypi/pyversions/sqlmesh-ff.svg)](https://pypi.org/project/sqlmesh-ff/)
+
+Configurable fitness functions plugin for [SQLMesh](https://sqlmesh.com) projects.
+
+Ships SQLMesh linter rules (classification macros, SQL complexity, metadata, naming) and architectural checks (layer integrity, custom exclusions, schema contracts, dependency graph) with a unified Rich lint report.
+
+## Installation
+
+```bash
+# Install from PyPI:
+uv add sqlmesh-ff
+
+# Or using pip:
+pip install sqlmesh-ff
+```
+
+## Quick start
+
+1. Add `fitness_functions.yaml` to your SQLMesh project root (see [Configuration](#configuration)).
+2. Add a small `config.py` bootstrap (see [Where configuration lives](#where-configuration-lives)) — SQLMesh requires the loader as a Python class and cannot load `config.py` and `config.yaml` in the same folder.
+3. Run lint:
+
+```bash
+sqlmesh-ff lint
+```
+
+## Where configuration lives
+
+There are three layers. Only the YAML/JSON files in your project are user-editable settings.
+
+| Layer | File | Role | You edit this? |
+|-------|------|------|----------------|
+| Plugin defaults | `sqlmesh_ff/config.py` (installed package) | Pydantic schema and built-in defaults (e.g. `fan_out_warn: 15`) | No — library code, never overwritten |
+| SQLMesh project | `settings.yaml` | Gateways, `linter.rules`, variables, CI/CD bot | Yes — normal SQLMesh config |
+| Fitness functions | `fitness_functions.yaml` | Thresholds, rule toggles, column naming/type rules, paths to JSON data | Yes — main FF config |
+| Loader bootstrap | `config.py` (project root) | Loads `settings.yaml` and registers `FitnessLoader` | Rarely — ~15 lines of wiring |
+| Contract data | `linter_contract_groups.json`, `linter_exclusions.json` | Repo-specific schema parity and dependency exclusions | Yes — project data |
+
+**Merge order for fitness settings:** plugin defaults → `fitness_functions.yaml` → optional `loader_kwargs` overrides in `config.py`. Your YAML always wins over plugin defaults. The project `config.py` does not hold fitness thresholds — it only points at `fitness_functions.yaml`.
+
+**Why `config.py` exists:** SQLMesh accepts `loader: FitnessLoader` only as a Python class, not as a YAML string. Because SQLMesh rejects having both `config.py` and `config.yaml` in one folder, projects use `settings.yaml` (SQLMesh settings) plus `config.py` (loader registration).
+
+Example bootstrap:
+
+```python
+from pathlib import Path
+
+from sqlmesh.core.config import Config
+from sqlmesh.utils.yaml import load as yaml_load
+from sqlmesh_ff.loader import FitnessLoader
+
+_settings = yaml_load(Path(__file__).parent / "settings.yaml")
+config = Config.parse_obj(_settings).update_with({
+    "loader": FitnessLoader,
+    "loader_kwargs": {"fitness_functions_config": "fitness_functions.yaml"},
+})
+```
+
+Enable individual SQLMesh rules in `settings.yaml` under `linter.rules` / `linter.warn_rules`.
+
+## Configuration
+
+Fitness function settings live in `fitness_functions.yaml` at the project root. Override the file path or individual keys via `loader_kwargs` in `config.py` (advanced — most projects only set `fitness_functions_config`).
+
+### Example `fitness_functions.yaml`
+
+```yaml
+contract_groups_path: linter_contract_groups.json
+exclusions_path: linter_exclusions.json
+
+layers:
+  order: [sources, derived, core, marts, export]
+
+checks:
+  layer_integrity: { enabled: true }
+  custom_exclusions: { enabled: true }
+  schema_contracts: { enabled: true }
+  dependency_graph:
+    enabled: true
+    fan_out_warn: 15
+    fan_out_fail: 25
+    fan_in_warn: 10
+
+rules:
+  classification_macros:
+    enabled: true
+    skip_layers: [sources]
+    columns:
+      product_type: "@product_type\\b|@PRODUCT_TYPE\\b"
+  sql_complexity:
+    enabled: true
+    thresholds:
+      decision_points: [15, 25]
+      cte_count: [8, 12]
+      join_count: [8, 12]
+      line_count: [250, 400]
+  mart_naming:
+    enabled: true
+    layer_name: marts
+    rule: prefix_with_subdirectory
+  column_names:
+    enabled: true
+    replacements: {}
+  column_types:
+    enabled: true
+    rules: []
+    equivalent_types:
+      text: [text, varchar]
+  metadata:
+    owner: true
+    description: true
+    grain: true
+  filename_equals_modelname:
+    enabled: true
+```
+
+### Project-specific JSON
+
+Keep repo-specific contract and exclusion data in your project:
+
+- `linter_contract_groups.json` — cross-model schema parity groups
+- `linter_exclusions.json` — blocked dependency patterns and allowed exceptions
+
+Reference their paths from `fitness_functions.yaml`. The plugin ships generic engines only; examples live in this README.
+
+### Rule name mapping
+
+SQLMesh uses lowercase class names in `linter.rules`:
+
+| Config key | SQLMesh rule name |
+|------------|-------------------|
+| `classification_macros` | `classificationmacros` |
+| `sql_complexity` | `sqlcomplexity` |
+| `mart_naming` | `martmodelnamingconvention` |
+| `column_names` | `columnnames` |
+| `column_types` | `columntypes` |
+| `metadata.owner` | `nomissingowner` |
+| `metadata.description` | `nomissingdescription` |
+| `metadata.grain` | `nomissinggrain` |
+| `filename_equals_modelname` | `filenameequalsmodelname` |
+
+## CLI
+
+```
+sqlmesh-ff lint [--project PATH] [--config PATH] [--checks CHECK,...] [--fail-level error|warning] [--group-by connascence|model]
+```
+
+- **Default:** all enabled checks plus SQLMesh linter rules
+- **`--checks layer_integrity,custom_exclusions`:** run subset (for pre-push hooks)
+- **`--fail-level warning`:** treat warnings as failures
+- **`--group-by connascence|model`:** change how violations are grouped in the report (default: `connascence`)
+
+## Integration example
+
+Example overrides. `api_request` should always be named `api_call`. `_id` columns should always be of type `text` and `is_` columns should always be of type `boolean`.
+
+```yaml
+column_names:
+  replacements:
+    api_request: api_call
+column_types:
+  rules:
+    - name: id_is_text
+      pattern: "_id$"
+      data_type: text
+    - name: boolean
+      pattern: "^is_"
+      data_type: boolean
+```
+
+## Examples
+
+A complete, runnable example project showcasing the configuration of `sqlmesh-ff` rules, exclusions, contracts, and a continuous integration workflow is located in the [examples/](file:///Users/bartschuijt/git/sqlmesh-ff/examples/) directory.
+
+To run the linter against the example project locally, run:
+```bash
+sqlmesh-ff lint --project examples/minimal-sqlmesh-project
+```
+
+See [examples/minimal-sqlmesh-project/fitness_functions.yaml](file:///Users/bartschuijt/git/sqlmesh-ff/examples/minimal-sqlmesh-project/fitness_functions.yaml) to inspect the configured rules.
+
+## Development
+
+Initialize your local environment and configure the Git pre-push hook:
+```bash
+make init
+```
+
+Run linter, tests, or check diff coverage:
+```bash
+make lint
+make test
+make coverage
+```
+
+### Releases and PR titles
+
+Releases are automated with [release-please](https://github.com/googleapis/release-please) on merges to `main`. Use [Conventional Commits](https://www.conventionalcommits.org/) in PR titles so changelog entries and semver bumps are correct.
+
+PR titles must start with a type prefix, for example:
+
+- `feat: add dependency graph fan-in check`
+- `fix: remove unused import in loader tests`
+- `docs: document fitness_functions.yaml merge order`
+- `ci: add release-please workflow`
+
+Supported types include `feat`, `fix`, `docs`, `style`, `refactor`, `perf`, `test`, `build`, `ci`, and `chore`. The PR title check in CI enforces this format.

tff_core-0.2.0/pyproject.toml

@@ -0,0 +1,23 @@
+[project]
+name = "tff-core"
+version = "0.2.0"
+description = "Core rules and fitness functions engine for transformation projects"
+readme = "README.md"
+requires-python = ">=3.12"
+license = { text = "MIT" }
+authors = [
+    { name = "Bart Schuijt", email = "schuijt.bart@gmail.com" }
+]
+dependencies = [
+    "pydantic>=2.0",
+    "pyyaml>=6.0",
+    "rich>=13.0",
+    "sqlglot>=23.0",
+]
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/tff"]

tff_core-0.2.0/src/tff/core/checks/custom_exclusions.py

@@ -0,0 +1,174 @@
+"""Custom dependency exclusion rules for layer/domain boundaries."""
+
+from __future__ import annotations
+
+import json
+import logging
+from pathlib import Path
+
+from tff.core.model import ModelRepresentation
+from tff.core.config import FitnessFunctionsConfig, resolve_project_path
+from tff.core.report import LintFinding
+from tff.core.utils.paths import get_layer_and_domain, model_path_relative
+
+logger = logging.getLogger(__name__)
+
+
+class CustomExclusionsChecker:
+    """Enforce custom exclusions for model dependencies between layers."""
+
+    def __init__(self, models: dict[str, ModelRepresentation], exclusions_path: Path):
+        self.models = models
+        self.exclusions_path = exclusions_path
+        self.exclusions = self._load_exclusions()
+
+    def _load_exclusions(self) -> dict:
+        if not self.exclusions_path.exists():
+            logger.warning(
+                "Config file %s not found. No exclusions will be enforced.",
+                self.exclusions_path,
+            )
+            return {}
+
+        try:
+            with open(self.exclusions_path, encoding="utf-8") as f:
+                return json.load(f)
+        except (json.JSONDecodeError, OSError) as e:
+            logger.warning(
+                "Could not load exclusions config from %s: %s",
+                self.exclusions_path,
+                e,
+            )
+            return {}
+
+    def _normalize_model_name(self, name: str) -> str:
+        parts = name.replace('"', "").split(".")
+        if len(parts) >= 2:
+            return f"{parts[-2]}.{parts[-1]}"
+        return name
+
+    def _is_allowed_exception(self, model_name: str, dependency_name: str) -> bool:
+        normalized_model = self._normalize_model_name(model_name)
+        normalized_dependency = self._normalize_model_name(dependency_name)
+
+        for exception in self.exclusions.get("allowed_exceptions", []):
+            if (
+                exception.get("model") == normalized_model
+                and exception.get("dependency") == normalized_dependency
+            ):
+                return True
+        return False
+
+    def _is_excluded_dependency(
+        self,
+        source_layer: str,
+        source_domain: str,
+        target_layer: str,
+        target_domain: str,
+        model_name: str | None = None,
+        dependency_name: str | None = None,
+    ) -> bool:
+        if model_name and dependency_name:
+            if self._is_allowed_exception(model_name, dependency_name):
+                return False
+
+        for exclusion in self.exclusions.get("exclusions", []):
+            source_match = True
+            if (
+                "source_layer" in exclusion
+                and exclusion["source_layer"] != source_layer
+            ):
+                source_match = False
+            if (
+                "source_domain" in exclusion
+                and exclusion["source_domain"] != source_domain
+            ):
+                source_match = False
+
+            target_match = True
+            if (
+                "target_layer" in exclusion
+                and exclusion["target_layer"] != target_layer
+            ):
+                target_match = False
+            if (
+                "target_domain" in exclusion
+                and exclusion["target_domain"] != target_domain
+            ):
+                target_match = False
+
+            if source_match and target_match:
+                return True
+
+        return False
+
+    def check_model(self, model: ModelRepresentation) -> list[str]:
+        if model.is_symbolic:
+            return []
+
+        violations = []
+        model_layer, model_domain = get_layer_and_domain(model.path)
+        if not model_layer:
+            return []
+
+        for dependency_name in model.depends_on:
+            try:
+                dependency_model = self.models.get(dependency_name)
+                if not dependency_model:
+                    continue
+
+                dep_layer, dep_domain = get_layer_and_domain(dependency_model.path)
+                if not dep_layer:
+                    continue
+
+                if self._is_excluded_dependency(
+                    source_layer=dep_layer,
+                    source_domain=dep_domain or "",
+                    target_layer=model_layer,
+                    target_domain=model_domain or "",
+                    model_name=str(model.name),
+                    dependency_name=str(dependency_name),
+                ):
+                    violations.append(
+                        f"Model '{model.name}' in layer '{model_layer}"
+                        f"{f'/{model_domain}' if model_domain else ''}' "
+                        f"depends on '{dependency_name}' in layer '{dep_layer}"
+                        f"{f'/{dep_domain}' if dep_domain else ''}', "
+                        f"which is not allowed by custom exclusions"
+                    )
+            except Exception as e:
+                logger.error(
+                    "Unexpected error checking dependency %s for model %s: %s",
+                    dependency_name,
+                    model.name,
+                    e,
+                    exc_info=True,
+                )
+                continue
+
+        return violations
+
+
+def collect_custom_exclusion_findings(
+    models: dict[str, ModelRepresentation], config: FitnessFunctionsConfig
+) -> list[LintFinding]:
+    exclusions_path = resolve_project_path(config, config.exclusions_path)
+    checker = CustomExclusionsChecker(models, exclusions_path)
+    findings: list[LintFinding] = []
+
+    for model_name, model in models.items():
+        if model.is_symbolic:
+            continue
+
+        for message in checker.check_model(model):
+            findings.append(
+                LintFinding(
+                    check="custom_exclusions",
+                    severity="error",
+                    model=str(model.name),
+                    path=model_path_relative(model),
+                    message=message.removeprefix(f"Model '{model.name}' ").strip(),
+                )
+            )
+
+    return findings