@caddis/cli 0.1.0 → 0.2.0

package/bundles/antigravity-plugin-extras/skills/data-contract-pipeline/references/layer-responsibilities.md

@@ -0,0 +1,333 @@
+# Layer Responsibilities
+
+Detailed specification for each layer in the data contract pipeline. When generating code for a layer, follow these rules exactly.
+
+---
+
+## Layer 1 — Source Adapter
+
+**Location**: `src/services/adapters/` or `src/services/`  
+**Responsibility**: Fetch raw data from external source and return it as a Python dict or list.
+
+### Rules
+
+- The adapter is the ONLY layer that knows about the source (file path, API URL, DB query)
+- Returns raw `dict` / `list[dict]` — no Pydantic models, no transformations
+- Handles connection errors, retries, and timeouts
+- Logs the fetch operation (loguru, never print)
+- Uses `pathlib.Path` for file paths, never hardcoded strings
+- Environment-specific config (URLs, credentials) comes from `pydantic-settings`, never hardcoded
+
+### Template
+
+```python
+from pathlib import Path
+from loguru import logger
+
+class {{SourceName}}Adapter:
+    """Fetches raw {{source_name}} data from {{source_type}}."""
+
+    def __init__(self, source_path: Path) -> None:
+        self._source_path = source_path
+
+    def fetch(self) -> dict:
+        """Load raw payload. Returns unvalidated dict."""
+        logger.debug(f"Fetching {{source_name}} from {self._source_path}")
+        # Source-specific fetch logic here
+        ...
+```
+
+### Anti-patterns
+
+- ❌ Adapter validates or transforms data (that's the normalizer's job)
+- ❌ Adapter returns a Pydantic model (that couples adapter to schema)
+- ❌ Adapter catches and silently swallows errors
+
+---
+
+## Layer 2 — Ingestion Model
+
+**Location**: `src/models/ingestion/`  
+**Responsibility**: Pydantic model that matches the raw source payload shape exactly (1:1).
+
+### Rules
+
+- Every key in the source payload MUST have a corresponding field
+- Field types match the source data types (str, int, float, list, dict, None)
+- Use `Optional[T]` for keys that may be null or absent in some payloads
+- Use `Field(alias=...)` only if the source key name is not valid Python (e.g., contains spaces or special chars)
+- Use `extra="allow"` during initial development to catch unexpected keys
+- Nested objects get their own ingestion model
+- This model is NOT the API response — it exists to validate raw data
+- **Share a `_FLEX` config constant** across all ingest DTOs so the `extra="allow"` policy is centralized
+
+### Template
+
+```python
+from pydantic import BaseModel, ConfigDict, Field
+
+# Shared config for all data-ingest DTOs.  extra="allow" means unknown keys
+# land in model_extra and flow through — nothing is silently dropped.
+_FLEX = ConfigDict(populate_by_name=True, extra="allow")
+
+
+class {{SourceName}}Payload(BaseModel):
+    """1:1 representation of raw {{source_name}} payload."""
+    model_config = _FLEX
+
+    {{field_name}}: {{raw_type}}
+    # ... one field per source key
+```
+
+### Anti-patterns
+
+- ❌ Dropping fields that "aren't needed yet" (breaks contract completeness)
+- ❌ Applying business logic (renaming, computing, filtering)
+- ❌ Using `extra="ignore"` (hides new source fields)
+- ❌ Declaring `ConfigDict(extra="allow")` inline on every model instead of sharing a `_FLEX` constant
+
+---
+
+## Layer 3 — Normalizer
+
+**Location**: `src/services/normalizers/`  
+**Responsibility**: Transform ingestion model into display DTO. This is where business logic lives.
+
+### Rules
+
+- Input: ingestion model instance. Output: display DTO instance
+- Handles field renaming, unit conversion, computed fields, filtering
+- Every transformation is explicit — no implicit field passing
+- Pure function where possible (no side effects, no DB calls)
+- If a field is intentionally dropped, document WHY with a comment
+
+### Template
+
+```python
+from src.models.ingestion.{{source_slug}} import {{SourceName}}Payload
+from src.models.responses.{{source_slug}} import {{DtoName}}
+
+def normalize_{{source_slug}}(raw: {{SourceName}}Payload) -> {{DtoName}}:
+    """Transform raw {{source_name}} payload into display DTO."""
+    return {{DtoName}}(
+        {{dto_field}}=raw.{{source_field}},
+        # Computed fields
+        {{computed_field}}=_compute_{{computed_field}}(raw),
+    )
+```
+
+### Anti-patterns
+
+- ❌ Normalizer fetches data (that's the adapter's job)
+- ❌ Normalizer returns dict instead of DTO (loses type safety)
+- ❌ Implicit field dropping without documentation
+
+---
+
+## Layer 4 — Display DTO
+
+**Location**: `src/models/responses/`  
+**Responsibility**: Frozen Pydantic model defining the API boundary contract.
+
+### Rules
+
+- `frozen=True` — immutable after construction
+- `populate_by_name=True` — can construct with Python names or aliases
+- `extra="forbid"` — no silent data loss (use "allow" only during migration, with logging)
+- `Field(alias="camelCaseName")` for every field exposed to frontend
+- All fields have type annotations
+- Optional fields have explicit defaults (`Field(default_factory=list)`)
+- This is the API contract — changes here require frontend coordination
+
+### Template
+
+```python
+from pydantic import BaseModel, ConfigDict, Field
+
+class {{DtoName}}(BaseModel):
+    """API response DTO for {{source_name}}."""
+    model_config = ConfigDict(
+        frozen=True,
+        populate_by_name=True,
+        extra="forbid",
+    )
+
+    {{field_name}}: {{python_type}} = Field(alias="{{camelCaseName}}")
+```
+
+### Anti-patterns
+
+- ❌ `extra="ignore"` — silently drops fields, causes data loss bugs
+- ❌ Mutable DTO — allows post-construction tampering
+- ❌ Missing aliases — forces frontend to use snake_case
+- ❌ Any field — loses type safety at the API boundary
+
+---
+
+## Layer 5 — API Router
+
+**Location**: `src/api/routers/`  
+**Responsibility**: FastAPI endpoint that returns typed `response_model`.
+
+### Rules
+
+- Always set `response_model=DtoName` on the endpoint
+- Router calls service layer, never adapter or repository directly
+- Returns DTO instance, not dict
+- Error handling returns structured error responses, not stack traces
+- Follows REST conventions for path naming
+
+### Template
+
+```python
+from fastapi import APIRouter, Depends
+from src.models.responses.{{source_slug}} import {{DtoName}}
+from src.services.{{source_slug}}_service import get_{{source_slug}}
+
+router = APIRouter(prefix="/api/{{source-slug}}", tags=["{{source_name}}"])
+
+@router.get("/", response_model={{DtoName}})
+async def get_data(
+    # query params here
+):
+    """Get {{source_name}} data."""
+    return await get_{{source_slug}}()
+```
+
+### Anti-patterns
+
+- ❌ No `response_model` — FastAPI can't validate the response shape
+- ❌ Returning raw dict — bypasses DTO validation
+- ❌ Business logic in router — should be in service layer
+
+---
+
+## Layer 6 — Frontend Types
+
+**Location**: `frontend/src/types/`  
+**Responsibility**: TypeScript interfaces that mirror the display DTO.
+
+### Rules
+
+- One interface per DTO, field names match DTO aliases (camelCase)
+- Use strict TypeScript types — no `any`
+- Nested objects get their own interface
+- Export from a barrel file (`index.ts`) for clean imports
+- When DTO changes, TypeScript interface changes in the same PR
+
+### Template
+
+```typescript
+export interface {{DtoName}} {
+  {{camelCaseName}}: {{tsType}};
+}
+```
+
+### Type Mapping
+
+| Python Type | TypeScript Type |
+|------------|----------------|
+| `str` | `string` |
+| `int`, `float` | `number` |
+| `bool` | `boolean` |
+| `list[T]` | `T[]` |
+| `dict[str, T]` | `Record<string, T>` |
+| `Optional[T]` | `T \| null` |
+| `datetime` | `string` (ISO format) |
+| `Decimal` | `number` |
+
+---
+
+## Layer 7 — Frontend Service
+
+**Location**: `frontend/src/services/`  
+**Responsibility**: Typed fetch/axios functions that call the API and return typed data.
+
+### Rules
+
+- Every function has a return type matching the TypeScript interface
+- Base URL from environment config, never hardcoded
+- Error handling with typed error responses
+- Uses the project's HTTP client (axios, fetch, React Query, etc.)
+
+### Template
+
+```typescript
+import type { {{DtoName}} } from '@/types/{{source-slug}}';
+import { apiClient } from '@/lib/api-client';
+
+export async function get{{DtoName}}(): Promise<{{DtoName}}> {
+  const { data } = await apiClient.get<{{DtoName}}>('/api/{{source-slug}}');
+  return data;
+}
+```
+
+### Anti-patterns
+
+- ❌ `any` return type — loses type safety
+- ❌ Raw `fetch()` without error handling
+- ❌ Hardcoded API URL
+
+---
+
+## Layer 8 — Contract Tests
+
+**Location**: `tests/unit/test_contract_*.py`  
+**Responsibility**: Validate alignment between layers using golden sample payloads.
+
+### Rules
+
+- At minimum: payload → DTO validation, uncovered keys check
+- Tests run fast (no network, no database)
+- Tests use golden sample from `tests/fixtures/payloads/`
+- Each data source gets its own test file
+- Test names describe what contract they validate
+
+### Test Categories
+
+1. **Structural**: Does the payload parse into the DTO?
+2. **Coverage**: Does the DTO cover all payload keys?
+3. **Alias**: Do aliases match the camelCase keys the frontend expects?
+4. **Optional**: Do fields with defaults handle absent keys?
+5. **Cross-layer**: Do TypeScript interface properties match DTO aliases?
+
+See [drift-check-catalog.md](./drift-check-catalog.md) for the complete check list.
+
+---
+
+## Schema Evolution Lifecycle
+
+Data pipelines rarely start with a frozen schema. This lifecycle describes how the three-layer defence adapts as the upstream schema matures.
+
+### Three-Layer Defence Summary
+
+| Layer | Mechanism | Purpose |
+|-------|-----------|----------|
+| **1 — Passthrough** | `extra="allow"` on all ingest DTOs (`_FLEX`) | Unknown keys flow through `model_extra` instead of being silently dropped |
+| **2 — Drift detection** | Recursive `_collect_dropped_keys()` in contract tests | Compares raw JSON keys vs DTO field aliases at every nesting depth |
+| **3 — Exclusion allow-list** | `KNOWN_EXCLUDED_KEYS` dict with fnmatch globs | Intentionally untyped keys don't pollute drift reports |
+
+### Workflow by Schema State
+
+| Pipeline state | `extra=` config | Drift test mode | Action |
+|---|---|---|---|
+| **Schema evolving daily** | `extra="allow"` (`_FLEX`) | Soft warning — `warnings.warn()`, CI passes | New keys flow through `model_extra`; drift reports show untyped keys as non-blocking warnings |
+| **Schema stabilised / frozen** | `extra="forbid"` on display DTO; `extra="allow"` on ingest if you want to catch new keys | Hard assert — `assert dropped == []`, CI blocks | Type all remaining `model_extra` keys; set `CONTRACT_SCHEMA_FROZEN=1` (preferred), or `_contract_state.json` (`{"schema_frozen": true}`), or rely on CI fallback |
+| **Key intentionally untyped** | n/a | Both modes | Add to `KNOWN_EXCLUDED_KEYS` with fnmatch glob + ticket ref; key is silently excluded from drift detection |
+
+### Decision Checklist for New JSON Keys
+
+1. **Is the key in every payload?** → Add typed field to the DTO now.
+2. **Is it experimental / may be removed?** → Leave untyped; `extra="allow"` forwards it. Add to `KNOWN_EXCLUDED_KEYS` if the warning is noisy.
+3. **Is it deprecated upstream?** → Add to `KNOWN_EXCLUDED_KEYS` permanently with ticket ref.
+4. **Is the schema frozen?** → Switch drift test to hard assert via precedence: `CONTRACT_SCHEMA_FROZEN=1` (preferred), then `_contract_state.json`, then CI fallback. Any new untyped key is a CI failure.
+
+### Anti-Patterns
+
+| Anti-pattern | Why it fails | Correct approach |
+|---|---|---|
+| `extra='ignore'` on ingest DTOs | Silently drops evolving fields | `extra='allow'` — data flows through |
+| Shape-only tests (`len(surveys) == 3`) | Pass even when half the fields are dropped | Recursive key-vs-field comparison |
+| Hardcoded expected-key lists in tests | Stale within a week | Read actual JSON files, compare dynamically |
+| One giant DTO with all fields optional | No validation, no IDE support | Nested models with required + optional typed fields |
+| `model_dump(exclude_unset=True)` to hide extras | Hides the problem, doesn’t fix it | Let `model_extra` flow through; type when ready |

package/bundles/antigravity-plugin-extras/skills/data-contract-pipeline/scripts/discover_connections.py

@@ -0,0 +1,170 @@
+"""Discover connection configuration in an app repository.
+
+This script scans common config files for DB/API/file connection hints and
+emits a masked connection_manifest.json. It never prints secret values.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import re
+from dataclasses import asdict, dataclass
+from datetime import datetime, timezone
+from pathlib import Path
+
+
+SECRET_RE = re.compile(
+    r"(password|passwd|pwd|secret|token|api[_-]?key|access[_-]?key)",
+    re.IGNORECASE,
+)
+KEY_VALUE_RE = re.compile(r"^\s*([A-Z0-9_.-]+)\s*[:=]\s*(.+?)\s*$", re.IGNORECASE)
+CONNECTION_KEYS = (
+    "DB_",
+    "DATABASE",
+    "SQL",
+    "MSSQL",
+    "POSTGRES",
+    "MYSQL",
+    "ODBC",
+    "JDBC",
+    "CONNECTION",
+    "API_",
+    "FILE_",
+)
+DEFAULT_SUFFIXES = {".ini", ".toml", ".yaml", ".yml", ".json", ".config"}
+CONFIG_NAME_HINTS = (
+    ".env",
+    "settings",
+    "config",
+    "docker-compose",
+    "compose",
+    "alembic",
+    "database",
+    "connection",
+    "deploy",
+    "deployment",
+    "readme",
+    "runbook",
+)
+SKIP_DIRS = {
+    ".git",
+    ".venv",
+    "venv",
+    "env",
+    "node_modules",
+    "dist",
+    "build",
+    "__pycache__",
+    ".next",
+    ".turbo",
+}
+
+
+@dataclass
+class ConnectionHint:
+    file: str
+    key: str
+    value: str
+    secret: bool
+
+
+def is_candidate_file(path: Path) -> bool:
+    if any(part in SKIP_DIRS for part in path.parts):
+        return False
+    name = path.name.lower()
+    if any(hint in name for hint in CONFIG_NAME_HINTS):
+        return True
+    if path.suffix.lower() in DEFAULT_SUFFIXES:
+        return True
+    return False
+
+
+def mask_value(key: str, value: str) -> tuple[str, bool]:
+    is_secret = bool(SECRET_RE.search(key))
+    if is_secret:
+        return "***", True
+    if SECRET_RE.search(value):
+        return re.sub(r"(?i)(pwd|password|token|secret)=([^;\\s]+)", r"\1=***", value), True
+    return value.strip().strip('"').strip("'"), False
+
+
+def scan_file(path: Path, root: Path) -> list[ConnectionHint]:
+    hints: list[ConnectionHint] = []
+    try:
+        text = path.read_text(encoding="utf-8", errors="ignore")
+    except OSError:
+        return hints
+
+    for line in text.splitlines():
+        match = KEY_VALUE_RE.match(line)
+        if not match:
+            continue
+        key, raw_value = match.group(1), match.group(2)
+        if raw_value.strip() in {"(", "{", "["}:
+            continue
+        if not key.upper().startswith(CONNECTION_KEYS) and "CONNECTION" not in key.upper():
+            continue
+        value, secret = mask_value(key, raw_value)
+        hints.append(
+            ConnectionHint(
+                file=str(path.relative_to(root)),
+                key=key,
+                value=value,
+                secret=secret,
+            )
+        )
+    return hints
+
+
+def infer_summary(hints: list[ConnectionHint]) -> dict[str, str]:
+    by_key = {h.key.upper(): h.value for h in hints if not h.secret}
+    return {
+        "host": by_key.get("DB_HOST") or by_key.get("DATABASE_HOST") or "",
+        "port": by_key.get("DB_PORT") or by_key.get("DATABASE_PORT") or "",
+        "database": by_key.get("DB_NAME") or by_key.get("DATABASE") or "",
+        "driver": by_key.get("DB_DRIVER") or by_key.get("ODBC_DRIVER") or "",
+        "auth_mode": (
+            "trusted"
+            if by_key.get("DB_TRUSTED_CONNECTION", "").lower() in {"yes", "true", "1"}
+            else "credential-or-unknown"
+        ),
+    }
+
+
+def discover(app_path: Path) -> dict[str, object]:
+    root = app_path.resolve()
+    hints: list[ConnectionHint] = []
+    for path in root.rglob("*"):
+        if path.is_file() and is_candidate_file(path):
+            hints.extend(scan_file(path, root))
+
+    return {
+        "generated_at": datetime.now(timezone.utc).isoformat(),
+        "app_path": str(root),
+        "summary": infer_summary(hints),
+        "connection_hints": [asdict(h) for h in hints],
+        "notes": [
+            "Secret values are masked.",
+            "Validate stale or conflicting configs before using a connection.",
+        ],
+    }
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description="Discover masked connection config in an app repo.")
+    parser.add_argument("--app", type=Path, required=True, help="Application repository path")
+    parser.add_argument("--output", type=Path, help="Output connection_manifest.json path")
+    args = parser.parse_args()
+
+    manifest = discover(args.app)
+    text = json.dumps(manifest, indent=2)
+    if args.output:
+        args.output.parent.mkdir(parents=True, exist_ok=True)
+        args.output.write_text(text + "\n", encoding="utf-8")
+    else:
+        print(text)
+
+
+if __name__ == "__main__":
+    main()

package/bundles/antigravity-plugin-extras/skills/data-contract-pipeline/scripts/discover_sources.py

@@ -0,0 +1,161 @@
+"""Discover source schemas from files or a SQL Server connection string.
+
+The script emits a normalized source_manifest.json. DB discovery uses pyodbc
+when available and metadata-only queries by default.
+"""
+
+from __future__ import annotations
+
+import argparse
+import csv
+import json
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any
+
+
+def infer_value_type(value: Any) -> str:
+    if value is None or value == "":
+        return "null-or-empty"
+    if isinstance(value, bool):
+        return "bool"
+    if isinstance(value, int):
+        return "int"
+    if isinstance(value, float):
+        return "float"
+    text = str(value).strip()
+    for caster, name in ((int, "int"), (float, "float")):
+        try:
+            caster(text.replace(",", ""))
+            return name
+        except ValueError:
+            pass
+    return "str"
+
+
+def discover_json(path: Path) -> dict[str, object]:
+    data = json.loads(path.read_text(encoding="utf-8"))
+    sample = data[0] if isinstance(data, list) and data else data
+    fields = []
+    if isinstance(sample, dict):
+        fields = [
+            {"path": key, "type": infer_value_type(value), "example": str(value)[:120]}
+            for key, value in sample.items()
+        ]
+    return {"source": str(path), "kind": "json", "fields": fields}
+
+
+def discover_csv(path: Path) -> dict[str, object]:
+    with path.open(newline="", encoding="utf-8-sig") as handle:
+        reader = csv.DictReader(handle)
+        first = next(reader, {})
+    fields = [
+        {"path": key, "type": infer_value_type(value), "example": str(value)[:120]}
+        for key, value in first.items()
+    ]
+    return {"source": str(path), "kind": "csv", "fields": fields}
+
+
+def discover_text(path: Path) -> dict[str, object]:
+    text = path.read_text(encoding="utf-8", errors="ignore")
+    return {
+        "source": str(path),
+        "kind": path.suffix.lower().lstrip(".") or "text",
+        "line_count": len(text.splitlines()),
+        "fields": [],
+    }
+
+
+def discover_file(path: Path) -> dict[str, object]:
+    suffix = path.suffix.lower()
+    if suffix == ".json":
+        return discover_json(path)
+    if suffix in {".csv", ".tsv"}:
+        return discover_csv(path)
+    return discover_text(path)
+
+
+def discover_db(connection_string: str, schema: str | None) -> dict[str, object]:
+    try:
+        import pyodbc  # type: ignore
+    except ModuleNotFoundError:
+        return {
+            "kind": "db",
+            "status": "blocked",
+            "reason": "pyodbc is not installed",
+            "objects": [],
+        }
+
+    objects: list[dict[str, object]] = []
+    with pyodbc.connect(connection_string, timeout=10) as conn:
+        cursor = conn.cursor()
+        cursor.execute(
+            """
+            SELECT TABLE_SCHEMA, TABLE_NAME, TABLE_TYPE
+            FROM INFORMATION_SCHEMA.TABLES
+            WHERE (? IS NULL OR TABLE_SCHEMA = ?)
+            ORDER BY TABLE_SCHEMA, TABLE_NAME
+            """,
+            schema,
+            schema,
+        )
+        rows = cursor.fetchall()
+        for row in rows:
+            table_schema, table_name, table_type = row
+            cursor.execute(
+                """
+                SELECT COLUMN_NAME, DATA_TYPE, IS_NULLABLE
+                FROM INFORMATION_SCHEMA.COLUMNS
+                WHERE TABLE_SCHEMA = ? AND TABLE_NAME = ?
+                ORDER BY ORDINAL_POSITION
+                """,
+                table_schema,
+                table_name,
+            )
+            columns = [
+                {"name": c.COLUMN_NAME, "type": c.DATA_TYPE, "nullable": c.IS_NULLABLE}
+                for c in cursor.fetchall()
+            ]
+            objects.append(
+                {
+                    "schema": table_schema,
+                    "name": table_name,
+                    "type": table_type,
+                    "columns": columns,
+                }
+            )
+    return {"kind": "db", "status": "discovered", "objects": objects}
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description="Discover data sources and emit source_manifest.json.")
+    parser.add_argument("--source", type=Path, action="append", help="Source file or directory")
+    parser.add_argument("--connection-string", help="ODBC connection string for DB metadata discovery")
+    parser.add_argument("--schema", help="Optional DB schema filter")
+    parser.add_argument("--output", type=Path, required=True, help="Output source_manifest.json path")
+    args = parser.parse_args()
+
+    sources: list[dict[str, object]] = []
+    for source in args.source or []:
+        if source.is_dir():
+            for path in source.rglob("*"):
+                if path.is_file():
+                    sources.append(discover_file(path))
+        elif source.exists():
+            sources.append(discover_file(source))
+        else:
+            sources.append({"source": str(source), "status": "missing", "fields": []})
+
+    if args.connection_string:
+        sources.append(discover_db(args.connection_string, args.schema))
+
+    manifest = {
+        "generated_at": datetime.now(timezone.utc).isoformat(),
+        "sources": sources,
+    }
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    args.output.write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8")
+
+
+if __name__ == "__main__":
+    main()