@caddis/cli 0.0.0 → 0.2.0

package/bundles/antigravity-plugin-extras/skills/data-contract-pipeline/README.md

@@ -0,0 +1,248 @@
+# Data Contract Pipeline
+
+Use this skill to map DB tables, files, APIs, requirements, and UI mockups into a verified frontend data contract.
+
+It helps you:
+
+- Map DB tables/files/docs into UI safely.
+- Generate read models, normalizers, DisplayDTOs, OpenAPI-first REST contracts, MCP-first contracts, frontend types, and services.
+- Detect gaps between mockups, requirements, backend data, and final UI.
+- Generate UI lineage so every production field can be traced back to source data.
+- Catch drift when sources, DTOs, API responses, or frontend types change.
+
+The skill is generic. Any app-specific DB, such as a CDR database, should be treated as an evaluation case or implementation input, not as hardcoded skill behavior.
+
+---
+
+## Key Principle
+
+DB tables do not map directly to UI.
+
+Requirements and UI demand decide what should be displayed. DBs, files, and APIs decide what can be displayed. The DisplayDTO is the contract between backend reality and frontend rendering.
+
+The safe path is:
+
+```text
+Source -> Read Model -> Normalizer -> DisplayDTO -> OpenAPI REST + MCP Contracts -> Frontend Types -> UI -> Lineage
+```
+
+---
+
+## How It Works
+
+```mermaid
+flowchart LR
+  A[User provides app repo, docs, DB/files, mockup] --> B[Discover app config and connections]
+  B --> C[Inspect sources: DB schemas, files, APIs]
+  A --> D[Extract requirements and UI demand]
+  C --> E[Analysis summary + Q&A checkpoint]
+  D --> E
+  E --> F[Reconcile source capability vs UI demand]
+  F --> G[Read Model]
+  G --> H[Normalizer]
+  H --> I[DisplayDTO]
+  I --> J[OpenAPI REST + MCP Contracts]
+  J --> K[Frontend Types + Services + MCP Clients]
+  K --> L[UI Components]
+  F --> M[Gap Register]
+  L --> N[UI Lineage Manifest]
+  N --> O[Contract + Drift Tests]
+```
+
+The skill asks questions after it has inspected the repo, docs, sources, and UI. This keeps the questions narrow and prevents guessing.
+
+---
+
+## How The User Uses It
+
+Use natural language. These are good prompts:
+
+```text
+Use data-contract-pipeline for this app repo and this DB. Map the required UI fields and generate lineage.
+```
+
+```text
+Use this skill to map this JSON/Markdown file into the UI and create DisplayDTOs and frontend types.
+```
+
+```text
+Audit whether my mockup is backed by real data. Show mockup-only, required-unplaced, and blocked elements.
+```
+
+```text
+Generate lineage for every UI field in this app.
+```
+
+```text
+The app repo has the DB config. Discover it first, then ask me only for missing details.
+```
+
+The user should provide as many of these as they have:
+
+- App repo path.
+- DB/table/view names or file paths.
+- Requirements docs, PRDs, tickets, or runbooks.
+- UI mockup folder, HTML, JSX/TSX, or final frontend route files.
+- Existing DTO/API/frontend type files if they exist.
+
+If credentials are missing, the skill first searches app config and then asks for only the missing values.
+
+---
+
+## Stages
+
+1. **Intake**: collect app repo, source inputs, requirements, and UI inputs.
+2. **Connection Discovery**: find DB/file/API config in the app repo and mask secrets.
+3. **Source Discovery**: inspect DB schemas, files, APIs, samples, and existing DTOs.
+4. **Requirements and UI Demand Extraction**: extract what the product and UI need.
+5. **Analysis Summary + Q&A**: ask up to 3 high-impact questions after discovery.
+6. **Reconciliation**: compare source capability against requirements and UI demand.
+7. **Read Model + DTO Design**: define adapter/read model, normalizer, DisplayDTO, API, and frontend types.
+8. **Lineage Generation**: trace each UI field back to source object/field and transform.
+9. **Validation**: run drift checks, type alignment, query smoke tests, and lineage coverage.
+
+---
+
+## Status Classes
+
+Every UI/data item should receive one status:
+
+| Status | Meaning |
+|--------|---------|
+| `contract-backed` | Verified source, transform, DTO/API, and UI placement exist. |
+| `required-unplaced` | Required by docs/backend but missing from mockup/final UI placement. |
+| `source-capability-only` | Source has the data, but no requirement/UI demand uses it. |
+| `mockup-only` | Present in mockup/demo data, but not part of the backed production UI unless promoted by the user. |
+| `partially-backed` | Some fields map, but source, formula, state, or freshness is incomplete. |
+| `blocked` | Cannot map safely because of missing credentials, source, formula, or safe query. |
+| `deferred` | Explicitly out of current release scope. |
+
+These statuses prevent demo mockups and raw backend fields from becoming production contracts by accident.
+
+---
+
+## Missing UI Handling
+
+When requirements or backend data demand a UI element that is absent from the mockup:
+
+1. Mark it `required-unplaced`.
+2. Create `ui_gap_brief.md`.
+3. Include requirement/source evidence, proposed page, component type, DisplayDTO fields, and acceptance criteria.
+4. Ask the user/product owner to accept placement or defer it.
+
+Do not silently place the element in a random screen.
+
+When the mockup has a UI element with no requirement or source:
+
+1. Mark it `mockup-only`.
+2. Do not create production REST or MCP endpoints for it.
+3. Consult the user and either remove it from the production UI, replace it with a data-backed element, or promote it by adding a requirement and source contract.
+
+Most creative mockup-only elements should be removed or replaced, not marked deferred. Use `deferred` only when the user explicitly says the feature is intended for a later release.
+
+---
+
+## Standard Outputs
+
+The workflow can emit:
+
+- `connection_manifest.json`
+- `source_manifest.json`
+- `requirements_manifest.json`
+- `ui_demand_manifest.json`
+- `contract_decisions.json`
+- `read_catalog.json`
+- `openapi_contract.yaml`
+- `mcp_contract.json`
+- `mapping_manifest.json`
+- `lineage_manifest.json`
+- `gap_register.json`
+- `ui_gap_brief.md`
+
+Not every run needs every artifact. Audit runs usually produce manifests and reports. Build runs also produce DTOs, API routes, frontend types, services, and tests.
+
+---
+
+## Script Reference
+
+Existing low-level helpers:
+
+- `extract_schema.py`: extract schema from JSON, Markdown, CSV, XLSX, YAML, text, or DB metadata.
+- `drift_check.py`: compare payload, ingestion model, and DisplayDTO.
+- `ts_dto_compare.py`: compare frontend TypeScript types to Python DTOs.
+- `generate_mapping_doc.py`: generate or check field mapping documentation.
+
+Workflow helpers:
+
+- `discover_connections.py`: find DB/file/API connection config in app repos.
+- `discover_sources.py`: inspect source files or DB metadata and emit `source_manifest.json`.
+- `extract_requirements.py`: parse docs and markdown into requirement demand.
+- `extract_ui_demand.py`: parse HTML/JSX/TSX/routes/mockup files into UI demand.
+- `reconcile_contract.py`: compare requirements, UI demand, and source capability.
+- `compile_read_catalog.py`: scaffold safe read model/query catalog.
+- `generate_api_contracts.py`: scaffold OpenAPI-first REST and MCP-first service contracts from the read catalog.
+- `validate_lineage.py`: verify production UI fields have lineage.
+- `generate_gap_report.py`: produce `gap_register.json` and `ui_gap_brief.md`.
+
+Example:
+
+```bash
+python .github/skills/workflow/data-contract-pipeline/scripts/discover_connections.py \
+  --app E:/Projects/my-app \
+  --output artifacts/data-contract/connection_manifest.json
+
+python .github/skills/workflow/data-contract-pipeline/scripts/extract_ui_demand.py \
+  --ui E:/Projects/my-app/frontend/src \
+  --output artifacts/data-contract/ui_demand_manifest.json
+
+python .github/skills/workflow/data-contract-pipeline/scripts/reconcile_contract.py \
+  --sources artifacts/data-contract/source_manifest.json \
+  --requirements artifacts/data-contract/requirements_manifest.json \
+  --ui-demand artifacts/data-contract/ui_demand_manifest.json \
+  --output artifacts/data-contract/mapping_manifest.json \
+  --gaps artifacts/data-contract/gap_register.json
+
+python .github/skills/workflow/data-contract-pipeline/scripts/compile_read_catalog.py \
+  --mapping artifacts/data-contract/mapping_manifest.json \
+  --output artifacts/data-contract/read_catalog.json
+
+python .github/skills/workflow/data-contract-pipeline/scripts/generate_api_contracts.py \
+  --read-catalog artifacts/data-contract/read_catalog.json \
+  --openapi artifacts/data-contract/openapi_contract.yaml \
+  --mcp artifacts/data-contract/mcp_contract.json
+```
+
+---
+
+## Enterprise Pattern Check
+
+This workflow is directionally aligned with how mature product/data/platform teams handle data-to-UI work:
+
+- Consumer-driven contracts: UI/API needs are explicit.
+- DTO/read-model boundaries: raw DB and vendor schemas do not leak into UI.
+- Typed APIs and frontend types: generated or checked.
+- OpenAPI-first REST and MCP-first endpoints: app data can be consumed by other apps and MCP clients.
+- Data lineage: every displayed field has a traceable source.
+- Contract tests: drift is caught before release.
+- Gap register: product/design/backend mismatches are visible.
+
+Large organizations may also use GraphQL schemas, OpenAPI codegen, dbt semantic layers, data catalogs, schema registries, and internal developer portals. This skill plus deterministic scripts is the practical, app-by-app version.
+
+---
+
+## Landmines
+
+- Secret leakage: never print passwords, tokens, or full connection strings.
+- SQL performance: avoid unbounded aggregation on large LIVE tables.
+- PII exposure: classify MSISDN, IMSI, email, customer IDs, names, and account IDs.
+- Mockup hallucination: demo data is not a production contract.
+- Backend overexposure: do not expose DB-only fields just because they exist.
+- Metric ambiguity: define formula, grain, timezone, unit, null semantics, and freshness.
+- Date/time drift: decide timezone and reporting grain.
+- Currency/units drift: define cents vs euros, bytes vs MB/GB, percentages vs fractions.
+- Null semantics: distinguish zero, null, missing, unavailable, and not applicable.
+- Auth drift: app config may be stale; test and report actual connection mode.
+- Schema doc drift: live schema can differ from docs.
+- Service contract sprawl: do not expose every read model as public REST/MCP; expose only approved, backed, permissioned capabilities.
+- MCP security: tools/resources must define auth, PII classification, rate limits, and read/write behavior.
+- Small-model fragility: keep stage artifacts and statuses explicit.

package/bundles/antigravity-plugin-extras/skills/data-contract-pipeline/SKILL.md

@@ -0,0 +1,203 @@
+---
+name: data-contract-pipeline
+context: fork
+description: "**WORKFLOW SKILL** - Build, audit, and validate data-to-UI contracts for apps. Use whenever the user mentions data mapping, UI lineage, DB-to-UI, DisplayDTOs, source-to-screen mapping, data contracts, schema drift, typed API responses, frontend type alignment, mockup grounding, requirements-to-UI mapping, or asks whether a UI is backed by real DB/file data. Works for DBs, JSON, Markdown, CSV, XLSX, YAML, APIs, and UI mockups."
+---
+
+# Data Contract Pipeline
+
+Build a verified path from data sources to UI fields without leaking raw database or file structure into frontend components.
+
+```
+Requirements + UI demand decide what should be displayed.
+DBs / files / APIs decide what can be displayed.
+Ingestion model → normalizer → DisplayDTO defines the safe contract between them.
+```
+
+If your organization has a reference scaffold for this pattern, run its tests first to understand it
+before applying the pipeline below.
+
+---
+
+## Decision tree — choose your flow before Stage 1
+
+```
+Do you have DB schema docs, ERDs, or an existing API spec?
+│
+├─ YES → documented_workflow.md  (Intake → Audit → Reconcile → Generate → Validate)
+│
+└─ NO  → discovery_workflow.md   (cold start — introspect DB first, then same 5 stages)
+
+Are you only checking whether an existing UI is backed by data?
+└─ YES → Run Stage 2 (Audit) only; produce gap_register in .contracts/
+```
+
+---
+
+## Operating model
+
+1. **Inspect first, ask second.** Read the repo, DB schema, mockups, and existing DTOs before asking the user anything.
+2. **Max 3 questions at once.** Prefer concrete defaults: "I will use `config/.env.api.dev` unless you say otherwise."
+3. **Never print secrets.** Log variable names, not values.
+4. **Bounded queries only.** `TOP N` / `OFFSET+FETCH` on LIVE tables. Never unbounded `SELECT *`.
+
+---
+
+## Stage 1 — Intake
+
+Collect or infer:
+- App repo path.
+- Source inputs: DB/table/view names, connection config, JSON/Markdown/CSV files, or schema docs.
+- Consumer inputs: requirements docs, UI mockups, existing routes or component files.
+- Mode: `build` | `audit` | `drift-check` | `lineage-only`.
+
+Search `.env*`, settings modules, Docker Compose, and deployment manifests for DB config before asking.
+
+---
+
+## Stage 2 — Audit
+
+Classify every source column against the D1–D12 drift catalogue.
+
+**D1** No fields dropped from embedded JSON/markdown blobs.
+**D2** Ingestion model has `extra="allow"` (future columns are captured, not rejected).
+**D3** DisplayDTO has `extra="forbid"` (normalizer bugs surface immediately).
+**D4** DisplayDTO uses camelCase field aliases (serialised output matches frontend expectations).
+**D5** NULL semantics are preserved, not defaulted (NULL revenue ≠ $0).
+**D6** Type coercion validated: string → Decimal, string → datetime, string → bool.
+**D7** Normalizer produces correct status for each null/malformed/full payload variant.
+**D8** Golden sample round-trip: raw_row → normalizer → model_dump(by_alias=True) matches expected.
+**D9** Schema drift: live DB columns match ingestion model field list.
+**D10** API → TypeScript: DisplayDTO aliases match generated TS interface keys.
+**D11** PII columns classified and not exposed unintentionally via DisplayDTO.
+**D12** Bounded query: repository enforces page_size ≤ MAX_PAGE, no unbounded SELECT.
+
+For embedded payload columns (NVARCHAR(MAX) / TEXT):
+- JSON blobs → `embedded_json_adapter` → embedded payload model (extra="allow" at every depth)
+- Markdown blobs → `embedded_markdown_adapter` → sections + bold_kv dict
+- CSV columns → `csv_column_adapter` (RFC 4180 quoted-comma aware)
+- Free text with IDs → `embedded_text_adapter` → paren_ids + lines
+
+Produce: annotated column list; flag null semantics, embedded type, and known sub-schema.
+
+---
+
+## Stage 3 — Reconcile
+
+Map each source column to a DisplayDTO field. Assign one status per field:
+
+| Status | Meaning |
+|---|---|
+| `contract-backed` | Source → normalizer → DisplayDTO → FE type tested end-to-end |
+| `partially-backed` | Derives from nullable payload; may degrade to `blocked` at runtime |
+| `blocked` | Mandatory source column always NULL in prod — cannot populate |
+| `required-unplaced` | Frontend/requirements demand it; source not yet identified |
+| `source-capability-only` | Source has it; no requirement or UI demand exists yet |
+| `mockup-only` | In UI mockup only — no backend data path |
+| `deferred` | Explicitly out of scope for current sprint |
+
+Use the reconcile sub-prompt: `assets/prompts/reconcile_mapping.md`.
+
+Output: `.contracts/feature.contract.json`.
+
+---
+
+## Stage 4 — Generate
+
+Produce layers in this order:
+1. Ingestion model (`extra="allow"`, raw payload columns as `str | None`)
+2. Embedded payload model (each sub-model also `extra="allow"`)
+3. Adapters (scalar, csv, json, markdown, text)
+4. Normalizer (row → DisplayDTO, status classification logic)
+5. DisplayDTO (`frozen=True`, `extra="forbid"`, camelCase aliases)
+6. Repository (SQLAlchemy `text()` with named params, schema-qualified, bounded)
+7. Service (orchestrate repo → ingestion model → normalizer)
+8. FastAPI router (`response_model=` on every route, Query bounds with ge/le)
+9. **Query catalog** — `.contracts/query_catalog.md` (see below)
+10. Frontend types — regenerate from `/openapi.json` via `npx openapi-typescript`
+11. Golden fixtures + D1–D12 contract tests
+
+`_FLEX = ConfigDict(populate_by_name=True, extra="allow")` on ingestion models.
+`_FROZEN = ConfigDict(frozen=True, extra="forbid", populate_by_name=True)` on DisplayDTOs.
+
+---
+
+## Stage 5 — Validate
+
+```bash
+pytest tests/unit/ -v   # all D1-D12 checks must pass
+```
+
+For `mockup-only` or `required-unplaced` fields: generate `ui_gap_brief.md` and ask the
+user to place, promote, or defer — do not silently add to a random screen.
+
+CI rule: frozen contracts fail on unmapped DisplayDTO fields, DTO/type drift, and missing PII classification.
+
+---
+
+## Query Catalog
+
+For every DB-backed UI field, produce `.contracts/query_catalog.md`.
+
+This document must be **executable** — every SQL block should run as-is against the target DB (substitute
+`{schema}` with the real schema name, e.g. `dbo`). Its purpose is to make explicit, during data analysis
+or debugging, exactly which SQL query populates which UI element.
+
+### Format
+
+````markdown
+# Query Catalog — {Feature Name}
+# Generated: {date}
+# Target: {DB server / environment}
+
+---
+
+## {EntityName} — {repository function name}
+
+**UI fields populated:** `fieldA`, `fieldB`, `fieldC`
+**Called by:** `{ServiceClass}.{method}` → `{NormalizerFunction}` → `{DisplayDTO}`
+**Paging:** TOP {MAX_PAGE} / OFFSET+FETCH
+
+```sql
+-- Fetch {EntityName} rows, paged
+-- Parameters: :customer_id (optional filter), :offset (int), :page_size (int)
+SELECT
+    t.column_a        AS field_a,
+    t.column_b        AS field_b,
+    t.payload_column  AS payload_column   -- NVARCHAR(MAX): parsed by embedded_json_adapter
+FROM {schema}.TableName AS t
+WHERE (:customer_id IS NULL OR t.customer_id = :customer_id)
+ORDER BY t.created_at DESC
+OFFSET :offset ROWS FETCH NEXT :page_size ROWS ONLY;
+```
+
+**Derived fields from payload_column (not in SQL — extracted by normalizer):**
+| UI field | Source path | Adapter |
+|---|---|---|
+| `summaryStatus` | Derived from NULL check on `payload_column` | n/a |
+| `keyIssueCount` | `payload_column → key_issues[].length` | `embedded_json_adapter` |
+| `dashboardMetrics` | `payload_column → dashboard_metrics{}` | `embedded_json_adapter` |
+````
+
+### Rules
+
+- One `##` section per repository function (one query per section).
+- List every UI field the query feeds — both scalar columns AND fields derived from embedded payloads.
+- For embedded payload columns, add the **Derived fields** table beneath the SQL block showing the JSON path, markdown section, or CSV token that maps to each UI field.
+- Queries must use **named bind parameters** (`:param`), not f-string interpolation.
+- Include the `OFFSET+FETCH` or `TOP N` bound in every query — never unbounded.
+- If a query is schema-qualified, use `{schema}.TableName` with a comment showing the default (e.g. `-- default: dbo`).
+- After generating, verify each query runs without error against the SQLite fixture: `python fixture/seed_sqlite_fixture.py && sqlite3 fixture/sample_source.db < .contracts/query_catalog_sqlite.sql`.
+
+---
+
+## Landmines
+
+- **Secret leakage** — mask passwords and full connection strings
+- **Unbounded queries** — no `SELECT *` on large LIVE tables; always bound
+- **PII exposure** — classify MSISDN, email, customer IDs, names, account IDs before exposing
+- **Mockup hallucination** — demo data constants are not a production contract
+- **Null semantics** — distinguish zero / null / missing / unavailable / not-applicable
+- **Metric ambiguity** — block until formula, grain, timezone, unit, and freshness are confirmed
+- **Schema doc drift** — live schema wins over stale docs
+- **MCP security** — every tool/resource needs auth, PII flag, rate limit, read/write scope

package/bundles/antigravity-plugin-extras/skills/data-contract-pipeline/assets/adapter.py.template

@@ -0,0 +1,44 @@
+"""
+{{SOURCE_NAME}} Source Adapter
+
+Fetches raw {{source_name}} data from {{source_type}}.
+Part of the data contract pipeline — this adapter returns raw dict/list,
+validated downstream by the ingestion model.
+"""
+
+from pathlib import Path
+
+from loguru import logger
+
+
+class {{ClassName}}Adapter:
+    """Fetches raw {{source_name}} data from {{source_type}}."""
+
+    def __init__(self, source_path: Path) -> None:
+        self._source_path = source_path
+
+    def fetch(self) -> dict:
+        """
+        Load raw payload from source.
+
+        Returns unvalidated dict — the ingestion model validates the shape.
+
+        Raises:
+            FileNotFoundError: If the source file does not exist.
+            json.JSONDecodeError: If the source file is not valid JSON.
+        """
+        import json
+
+        logger.debug(f"Fetching {{source_name}} from {self._source_path}")
+
+        if not self._source_path.exists():
+            raise FileNotFoundError(
+                f"{{source_name}} source not found: {self._source_path}"
+            )
+
+        raw = json.loads(self._source_path.read_text(encoding="utf-8"))
+        logger.info(
+            f"Loaded {{source_name}} payload "
+            f"({len(raw) if isinstance(raw, list) else 'object'})"
+        )
+        return raw

package/bundles/antigravity-plugin-extras/skills/data-contract-pipeline/assets/api-router.py.template

@@ -0,0 +1,43 @@
+"""
+{{SOURCE_NAME}} API Router
+
+FastAPI endpoint returning typed response_model for {{source_name}} data.
+
+Part of the data contract pipeline:
+  Source → Adapter → Ingestion Model → Normalizer → Display DTO → **API Router**
+"""
+
+from fastapi import APIRouter, Depends, HTTPException, Query
+from loguru import logger
+
+from src.models.responses.{{source_slug}} import {{ClassName}}Response
+from src.services.{{source_slug}}_service import get_{{source_slug}}_data
+
+router = APIRouter(
+    prefix="/api/{{url-slug}}",
+    tags=["{{source_name}}"],
+)
+
+
+@router.get(
+    "/",
+    response_model={{ClassName}}Response,
+    summary="Get {{source_name}} data",
+)
+async def get_data(
+    # --- Add query parameters as needed ---
+    # period: str = Query(..., description="Period identifier (e.g., '2026-03')"),
+):
+    """
+    Retrieve {{source_name}} data.
+
+    Returns a typed {{ClassName}}Response validated by Pydantic.
+    """
+    try:
+        result = await get_{{source_slug}}_data()
+        return result
+    except FileNotFoundError:
+        raise HTTPException(status_code=404, detail="Data not found for requested period")
+    except Exception as e:
+        logger.error(f"Error fetching {{source_name}} data: {e}")
+        raise HTTPException(status_code=500, detail="Internal server error")