data-contract-validator 1.0.4b0__tar.gz → 1.1.0__tar.gz

data_contract_validator-1.1.0/CHANGELOG.md

@@ -0,0 +1,121 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [1.1.0] - 2026-06-30
+
+This release is focused on **accuracy** — making a red check always mean a real
+problem and a green check genuinely safe, so the tool can be trusted to gate a
+deploy.
+
+### Added
+- **Canonical type system** (`core/types.py`): every extractor now normalizes
+  its native types (warehouse SQL types, Python hints) into a shared, neutral
+  vocabulary (`CanonicalType`). The validator compares canonical types instead
+  of raw strings, eliminating the bulk of false "type mismatch" warnings
+  (e.g. dbt `varchar` vs Pydantic `str` are now correctly equal).
+  - Dialect-aware normalization: Snowflake `NUMBER(38,0)`→bigint, BigQuery
+    `INT64`/`FLOAT64`, Redshift `SUPER`, Postgres `jsonb`, and more.
+- **Tiered dbt extraction** with graceful degradation:
+  1. `catalog.json` — real warehouse types (high confidence).
+  2. `sqlglot` — a proper SQL parser. Handles CTEs, `||`, window functions, and
+     quoted identifiers that the old regex parser mangled. Detects `SELECT *`
+     and flags the schema as incomplete.
+  3. regex — last-resort best effort (low confidence, never hard-fails).
+- **Confidence-aware validation**: when source columns can't be fully resolved
+  (e.g. `SELECT *`), a missing column is reported as a **warning, not a
+  build-blocking critical**. Type warnings are suppressed for low-confidence
+  (regex-tier) sources. This is the core false-positive guard.
+- **Explicit mapping config** (`mapping:` in `.retl-validator.yml`) for when
+  name heuristics aren't enough — map a target table/column to a differently
+  named source model/column:
+  ```yaml
+  mapping:
+    tables:
+      user_analytics: user_analytics_summary
+    columns:
+      user_analytics:
+        userId: user_id
+  ```
+- **Name normalization**: tables/columns now match across snake_case, camelCase
+  and casing differences (`userId` == `user_id` == `USER_ID`).
+
+### Changed
+- `Schema` now carries `confidence` and `is_complete` (via `metadata`).
+- `BaseExtractor` no longer contains Python-specific type mapping; type
+  normalization lives in the canonical type system. Added `_make_column` helper.
+- Added `sqlglot` as a dependency (imported optionally; falls back to regex if
+  absent).
+
+### Fixed
+- Hardened GitHub API rate-limit handling against non-dict response headers
+  (previously could raise when headers weren't a mapping).
+
+## [1.0.5] - 2025-01-24
+
+### Fixed
+- **CRITICAL**: Fixed missing return statement in `DBTExtractor.extract_schemas()` that could return `None` instead of dictionary
+  - Added fallback to SQL file parsing when manifest.json is unavailable
+  - Now works reliably with or without DBT CLI installed
+- **HIGH**: Fixed function signature mismatch in `_test_configuration()` causing TypeError on `--dry-run` command
+  - Added missing `disable_manifest` parameter
+  - Enhanced to display manifest parsing status
+- **MEDIUM**: Replaced bare exception handler in `_try_compile_dbt()` with specific exception types
+  - Now properly handles TimeoutExpired, FileNotFoundError
+  - Provides helpful error messages instead of silent failures
+  - Respects keyboard interrupts
+- **MEDIUM**: Removed unused `fastapi_directory` parameter from CLI
+  - Simplified API - use `--fastapi-local` for both files and directories
+- **MEDIUM**: Added comprehensive YAML error handling with user-friendly messages
+  - Catches malformed YAML files with helpful suggestions
+  - Validates required configuration sections
+  - Provides clear error messages instead of Python tracebacks
+- **LOW**: Added GitHub API rate limiting detection and handling
+  - Monitors rate limit headers and warns when limits are low
+  - Provides helpful guidance to use GITHUB_TOKEN for higher limits
+  - Better error messages for 403 and 404 responses
+
+### Improved
+- Enhanced error messages throughout the application
+- Better support for different use-cases:
+  - DBT projects with or without manifest.json
+  - Local files and directories for FastAPI models
+  - GitHub repositories with rate limit awareness
+  - Configuration validation with clear error reporting
+
+## [1.0.0] - 2025-01-XX
+
+### Added
+- Initial release of Data Contract Validator
+- DBT schema extraction from SQL files and manifest.json
+- FastAPI/Pydantic model extraction from local files and GitHub repos
+- Command-line interface with multiple output formats
+- GitHub Actions integration
+- Contract validation with critical/warning/info severity levels
+- Support for multiple repositories and complex validation scenarios
+
+### Features
+- ✅ DBT model schema extraction
+- ✅ FastAPI/Pydantic schema extraction
+- ✅ Cross-repository validation
+- ✅ GitHub Actions workflows
+- ✅ Multiple output formats (terminal, JSON, GitHub Actions)
+- ✅ Comprehensive error reporting with suggested fixes
+- ✅ Type compatibility checking
+- ✅ Missing table/column detection
+
+### Known Limitations
+- Only supports DBT and FastAPI currently
+- Requires manual installation of DBT CLI
+- Limited type inference from SQL
+- No support for complex nested types
+
+[Unreleased]: https://github.com/OGsiji/data-contract-validator/compare/v1.1.0...HEAD
+[1.1.0]: https://github.com/OGsiji/data-contract-validator/releases/tag/v1.1.0
+[1.0.5]: https://github.com/OGsiji/data-contract-validator/releases/tag/v1.0.5
+[1.0.0]: https://github.com/OGsiji/data-contract-validator/releases/tag/v1.0.0

data_contract_validator-1.1.0/PKG-INFO

@@ -0,0 +1,336 @@
+Metadata-Version: 2.4
+Name: data-contract-validator
+Version: 1.1.0
+Summary: Validate data contracts between dbt models and FastAPI/Pydantic APIs with accurate, low-false-positive schema checks
+Author-email: Ogunniran Siji <ogunniransiji@gmail.com>
+Maintainer-email: Ogunniran Siji <ogunniransiji@gmail.com>
+License: MIT
+Project-URL: Homepage, https://github.com/OGsiji/data-contract-validator
+Project-URL: Documentation, https://github.com/OGsiji/data-contract-validator/blob/main/README.md
+Project-URL: Repository, https://github.com/OGsiji/data-contract-validator
+Project-URL: Bug Reports, https://github.com/OGsiji/data-contract-validator/issues
+Project-URL: Changelog, https://github.com/OGsiji/data-contract-validator/blob/main/CHANGELOG.md
+Keywords: dbt,fastapi,contract-testing,api-validation,data-engineering,schema-validation,ci-cd,devops
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Quality Assurance
+Classifier: Topic :: Software Development :: Testing
+Classifier: Topic :: Database
+Classifier: Topic :: Internet :: WWW/HTTP :: HTTP Servers
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: PyYAML>=6.0
+Requires-Dist: requests>=2.25.0
+Requires-Dist: click>=8.0.0
+Requires-Dist: sqlglot>=20.0.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
+Requires-Dist: black>=22.0.0; extra == "dev"
+Requires-Dist: flake8>=4.0.0; extra == "dev"
+Requires-Dist: mypy>=0.991; extra == "dev"
+Requires-Dist: pre-commit>=2.20.0; extra == "dev"
+Requires-Dist: build>=0.8.0; extra == "dev"
+Requires-Dist: twine>=4.0.0; extra == "dev"
+Provides-Extra: test
+Requires-Dist: pytest>=7.0.0; extra == "test"
+Requires-Dist: pytest-cov>=4.0.0; extra == "test"
+Requires-Dist: pytest-mock>=3.8.0; extra == "test"
+Dynamic: license-file
+
+# 🛡️ Data Contract Validator
+
+> **Catch breaking changes between your dbt models and your FastAPI/Pydantic APIs — before they hit production.**
+
+[![PyPI version](https://badge.fury.io/py/data-contract-validator.svg)](https://badge.fury.io/py/data-contract-validator)
+[![Tests](https://github.com/OGsiji/data-contract-validator/workflows/Tests/badge.svg)](https://github.com/OGsiji/data-contract-validator/actions)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+## 🎯 What it solves
+
+Your analytics team changes a dbt model. Your API team's FastAPI service still
+expects the old shape. Nobody notices until production 500s at 2 AM.
+
+This tool sits on that boundary. It extracts the schema your **dbt models
+produce** and the schema your **Pydantic models expect**, compares them, and
+fails CI when the data side can no longer satisfy the API side.
+
+```
+   dbt models                 Data Contract Validator                FastAPI / Pydantic
+(what the pipeline   ──▶   extract → normalize → compare   ◀──   (what the API expects)
+    produces)                     ↓
+                          critical issues block the build
+```
+
+### Built for trust
+
+A check that gates a deploy is only useful if it doesn't cry wolf. v1.1
+re-architected extraction around that principle:
+
+- **Canonical types** — dbt `varchar` and Pydantic `str` are understood to be
+  the same thing, so you don't get drowned in fake "type mismatch" warnings.
+- **A real SQL parser** (`sqlglot`) instead of regex — CTEs, `||`
+  concatenation, window functions and quoted identifiers are parsed correctly.
+- **Confidence-aware** — if the tool can't fully resolve a model's columns
+  (e.g. `SELECT *`), it will **warn** rather than falsely **block** your build.
+
+## ⚡ Quick start
+
+```bash
+pip install data-contract-validator
+```
+
+```bash
+# Initialize config + CI workflow in your dbt project
+contract-validator init --interactive
+
+# Sanity-check the setup
+contract-validator test
+
+# Validate
+contract-validator validate
+```
+
+### One-off validation (no config file)
+
+```bash
+# Local dbt project against a local Pydantic models file or directory
+contract-validator validate \
+  --dbt-project ./my-dbt-project \
+  --fastapi-local ./my-api/app/models.py
+
+# dbt project against models in another GitHub repo (microservices)
+contract-validator validate \
+  --dbt-project . \
+  --fastapi-repo "my-org/my-api" \
+  --fastapi-path "app/models.py"
+```
+
+## 🔍 How extraction works (and why it's accurate)
+
+### dbt side — tiered, best-source-wins
+
+| Tier | Source | Types | Confidence | Notes |
+|---|---|---|---|---|
+| 1 | `target/catalog.json` | **Real warehouse types** | high | Produced by `dbt docs generate`. Most accurate. |
+| 2 | `sqlglot` SQL parse | Inferred (often unknown) | medium | Trusted column **names**; enriched with documented types from `manifest.json`. Detects `SELECT *`. |
+| 3 | regex parse | Guessed | low | Last resort. Never used to hard-fail a build. |
+
+The tool auto-detects what's available and degrades gracefully — so it works
+offline in pre-commit **and** with full type fidelity in a warehouse-connected
+CI job.
+
+> 💡 **Tip:** run `dbt docs generate` in CI before validating to unlock Tier 1
+> (real types). Without it, you still get accurate column-presence checks from
+> Tier 2.
+
+### FastAPI side
+
+Pydantic / SQLModel classes are parsed from source with Python's `ast` (no
+imports executed). `Optional[...]` controls whether a field is required;
+`table=True` SQLModel classes (DB tables, not API contracts) are skipped.
+
+## 🚦 What gets flagged
+
+| Severity | Meaning | Example |
+|---|---|---|
+| 🚨 **Critical** | Blocks the build | API requires a column the dbt model no longer produces |
+| ⚠️ **Warning** | Worth a look, non-blocking | A real type mismatch, or a missing column on a model we couldn't fully resolve |
+
+```bash
+$ contract-validator validate
+
+🛡️ Data Contract Validation Results:
+Status: ❌ FAILED
+Critical: 1 | Warnings: 0
+
+🚨 Critical Issues (Must Fix):
+  💥 user_analytics
+     Column: total_orders
+     Problem: Target REQUIRES column 'total_orders' but source doesn't provide it
+     🔧 Fix: Add column 'total_orders' to source model for table 'user_analytics'
+```
+
+## 🔧 Configuration (`.retl-validator.yml`)
+
+```yaml
+version: "1.0"
+name: "my-project-contracts"
+
+source:
+  dbt:
+    project_path: "."
+    auto_compile: true
+    # Force Tier 2/3 SQL parsing even if catalog/manifest exist:
+    disable_manifest: false
+
+target:
+  fastapi:
+    # GitHub repo:
+    type: "github"
+    repo: "my-org/my-api"
+    path: "app/models.py"
+    # ...or local:
+    # type: "local"
+    # path: "../my-api/app/models.py"
+
+# Optional: explicit mapping for when names don't line up by convention.
+mapping:
+  tables:
+    # target (Pydantic) table : source (dbt) model
+    user_analytics: user_analytics_summary
+  columns:
+    user_analytics:
+      # target column : source column
+      userId: user_id
+
+validation:
+  fail_on: ["missing_tables", "missing_required_columns"]
+  warn_on: ["type_mismatches", "missing_optional_columns"]
+```
+
+### When do I need `mapping`?
+
+By default, names are matched across `snake_case` / `camelCase` / casing
+(`UserAnalytics` → `user_analytics`, `userId` → `user_id`). Reach for `mapping`
+only when a model or column is named so differently that the convention can't
+bridge it (e.g. Pydantic `user_id` ↔ dbt `customer_identifier`).
+
+## 🐍 Python API
+
+```python
+from data_contract_validator import ContractValidator, DBTExtractor, FastAPIExtractor
+
+dbt = DBTExtractor(project_path="./dbt-project")
+fastapi = FastAPIExtractor.from_github_repo("my-org/my-api", "app/models.py")
+
+validator = ContractValidator(
+    source_extractor=dbt,
+    target_extractor=fastapi,
+    mapping={"tables": {"user_analytics": "user_analytics_summary"}},  # optional
+)
+result = validator.validate()
+
+if not result.success:
+    for issue in result.critical_issues:
+        print(f"💥 {issue.table}.{issue.column}: {issue.message}")
+```
+
+## 🪝 CI / pre-commit integration
+
+### GitHub Actions
+
+`contract-validator init` generates a workflow for you. Minimal version:
+
+```yaml
+name: 🛡️ Data Contract Validation
+on:
+  pull_request:
+    paths: ["models/**/*.sql", "dbt_project.yml", "**/*models*.py"]
+jobs:
+  validate-contracts:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v4
+        with: { python-version: "3.11" }
+      - run: pip install data-contract-validator
+      # Optional: `dbt docs generate` here for real warehouse types (Tier 1)
+      - run: contract-validator validate --output github
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+```
+
+### Pre-commit
+
+```bash
+contract-validator setup-precommit --install-hooks
+```
+
+```yaml
+repos:
+  - repo: https://github.com/OGsiji/data-contract-validator
+    rev: v1.1.0
+    hooks:
+      - id: contract-validation
+```
+
+## 🧪 Output formats
+
+```bash
+contract-validator validate --output terminal   # human-friendly (default)
+contract-validator validate --output json        # machine-readable for CI
+contract-validator validate --output github       # GitHub Actions annotations
+```
+
+## 🚀 Supported frameworks
+
+**Source:** dbt (all adapters — Snowflake, BigQuery, Redshift, Postgres, …).
+**Target:** FastAPI (Pydantic v2 + SQLModel).
+
+The extractor architecture is intentionally pluggable (`BaseExtractor` →
+`Dict[str, Schema]` with canonical types), so additional sources/targets can be
+added without touching the validator. [Open an issue](https://github.com/OGsiji/data-contract-validator/issues)
+to request one.
+
+## 🛠️ Development & testing
+
+```bash
+git clone https://github.com/OGsiji/data-contract-validator
+cd data-contract-validator
+
+python -m venv .venv && source .venv/bin/activate
+pip install -e ".[dev]"     # or: pip install -e ".[test]"
+
+# Run the suite
+pytest
+
+# Lint / format
+black data_contract_validator tests
+```
+
+The test suite covers the canonical type system (`tests/test_core/test_types.py`),
+the tiered dbt extractor including sqlglot CTE handling and `catalog.json`
+(`tests/test_extractors/test_dbt.py`), and the confidence/mapping behavior of
+the validator (`tests/test_core/test_validator.py`).
+
+### Adding an extractor
+
+```python
+from data_contract_validator.extractors.base import BaseExtractor
+from data_contract_validator.core.types import CanonicalType
+
+class MyExtractor(BaseExtractor):
+    def extract_schemas(self):
+        # return Dict[str, Schema]; use self._make_column(...) so each column
+        # carries a canonical_type the validator can compare.
+        ...
+```
+
+## 🗺️ Roadmap
+
+- Real compatibility semantics (nullability, additive vs. breaking changes)
+- Reporter/logging abstraction (quiet/embeddable core)
+- A canonical, language-neutral contract artifact + baseline/snapshot diffing
+- More targets (Django, SQLAlchemy, GraphQL, OpenAPI)
+
+## 📄 License
+
+MIT — see [LICENSE](LICENSE).
+
+## 🆘 Support
+
+- 🐛 Issues: https://github.com/OGsiji/data-contract-validator/issues
+- 📧 Email: ogunniransiji@gmail.com
+
+If this saves you a production incident, please ⭐ the repo.