@caddis/cli 0.0.0 → 0.2.0

package/bundles/antigravity-plugin-extras/skills/data-contract-pipeline/assets/prompts/extract_ui_demand.md

@@ -0,0 +1,35 @@
+# Sub-prompt: Extract UI Demand
+
+Use this prompt to extract data demand from existing frontend components (JSX, TSX, HTML, Vue templates).
+Paste the component source after the divider, then run this prompt.
+
+---
+
+You are extracting data field demand from a frontend component or page.
+
+For each field, prop, variable, or binding that renders or filters data, produce a JSON entry:
+
+```json
+{
+  "display_label": "Text shown to the user (from label, column header, tooltip, etc.)",
+  "prop_or_key": "The prop name or object key used in code (e.g. row.customerId)",
+  "component": "Component or page file name",
+  "context": "table-column | kpi-card | chart-axis | filter | export | badge | detail-panel | other",
+  "inferred_type": "string | number | boolean | date | list | enum | unknown",
+  "nullable_in_ui": "true if the UI renders a fallback (e.g. '—', 'N/A') for missing values",
+  "current_source": "Where does the value come from? (mock constant, API call, prop, store, hardcoded)",
+  "backed": "true | false | unknown — is this connected to a real API or DB?",
+  "notes": "Anything suspicious: hardcoded data, TODO comments, demo-only flags"
+}
+```
+
+Rules:
+- Include EVERY rendered field, even if it appears backed by mock data.
+- Mark `backed: false` when value is a hardcoded string, imported mock constant, or demo fixture.
+- Mark `backed: unknown` when you cannot determine the source from the component alone.
+- Do not skip fields because they seem obvious — completeness is required.
+- Output a JSON array of demand objects, followed by a `## Mock/Unbacked Fields` section listing fields that are NOT currently connected to real data.
+
+---
+
+COMPONENT SOURCE:

package/bundles/antigravity-plugin-extras/skills/data-contract-pipeline/assets/prompts/reconcile_mapping.md

@@ -0,0 +1,66 @@
+# Sub-prompt: Reconcile Mapping
+
+Use this prompt to map source columns → normalizer transforms → DisplayDTO fields.
+Provide the source column list and requirements list, then run this prompt.
+
+---
+
+You are reconciling data source capabilities against product requirements and UI demand.
+
+## Inputs
+
+You will receive:
+1. `SOURCE_COLUMNS` — columns from the ingestion model or DB schema
+2. `REQUIREMENTS` — field requirements extracted from docs or tickets
+3. `UI_DEMAND` — fields extracted from frontend components
+4. `EXISTING_CONTRACT` — existing `.contracts/*.contract.json` (if any)
+
+## Task
+
+For each DisplayDTO field that should exist (derived from requirements + UI demand):
+
+1. Find the best matching source column(s).
+2. Determine what adapter and normalizer transform is needed.
+3. Assign a status:
+
+| Status | Assign when |
+|---|---|
+| `contract-backed` | Source column maps cleanly; transform is deterministic; test exists |
+| `partially-backed` | Derives from nullable embedded payload; may degrade to blocked at runtime |
+| `blocked` | Required column is always NULL in prod, or formula is undefined |
+| `required-unplaced` | Frontend/requirements demand it; no source column found |
+| `source-capability-only` | Source has it; no requirement or UI demand |
+| `mockup-only` | In UI mockup only; no backend path |
+| `deferred` | Explicitly out of scope |
+
+4. Output one JSON entry per DisplayDTO field:
+
+```json
+{
+  "field_key": "camelCase DTO key",
+  "source_column": "column_name (or null if required-unplaced)",
+  "adapter": "scalar | csv | embedded_json | embedded_markdown | embedded_text | none",
+  "transform": "Description of normalizer logic (e.g. 'split CSV, preserve quoted commas')",
+  "status": "contract-backed | partially-backed | ...",
+  "null_handling": "What the normalizer returns when source is NULL",
+  "type": "Python type annotation for DisplayDTO",
+  "fe_type": "TypeScript type for frontend",
+  "notes": "Any caveats, edge cases, or open questions"
+}
+```
+
+## Output format
+
+1. JSON array of mapping entries (one per DisplayDTO field).
+2. `## Gaps` — fields that are `required-unplaced` or `blocked`, with recommended next action.
+3. `## Orphans` — source columns that are `source-capability-only`, listed for product review.
+
+---
+
+SOURCE_COLUMNS:
+
+REQUIREMENTS:
+
+UI_DEMAND:
+
+EXISTING_CONTRACT:

package/bundles/antigravity-plugin-extras/skills/data-contract-pipeline/assets/query-catalog.md.template

@@ -0,0 +1,62 @@
+# Query Catalog — {FeatureName}
+# Generated: {date}
+# Target: {DB server / environment}
+# Schema: {schema} (default: dbo)
+#
+# PURPOSE: Every SQL query that populates a DB-backed UI field is listed here.
+# Queries use named bind parameters and are bounded — safe to run during data analysis.
+# Substitute {schema} → dbo (or your override) before executing.
+
+---
+
+## {EntityName} — {repository_module}.fetch_{entity}_list
+
+**UI fields populated:** `fieldA`, `fieldB`, `fieldC`, `embeddedStatus`
+**Called by:** `{EntityService}.get_list` → `normalize_{entity}` → `{EntityResponse}`
+**Paging:** OFFSET + FETCH, max {MAX_PAGE} rows
+
+```sql
+-- List query — paged, optional filter
+-- Parameters:
+--   :filter_value  VARCHAR  optional WHERE filter (pass NULL to skip)
+--   :offset        INT      OFFSET rows (0-based)
+--   :page_size     INT      rows per page (max {MAX_PAGE})
+SELECT
+    t.primary_key_col     AS primary_key,
+    t.display_col_a       AS field_a,
+    t.display_col_b       AS field_b,
+    t.embedded_col        AS embedded_col   -- NVARCHAR(MAX): see derived fields below
+FROM {schema}.{SourceTable} AS t
+WHERE (:filter_value IS NULL OR t.filter_col = :filter_value)
+ORDER BY t.created_at DESC
+OFFSET :offset ROWS FETCH NEXT :page_size ROWS ONLY;
+```
+
+**Derived fields from `embedded_col` (extracted by normalizer, not in SQL):**
+
+| UI field | Payload path | Adapter | Null behaviour |
+|---|---|---|---|
+| `embeddedStatus` | NULL check on `embedded_col` | n/a | `blocked` |
+| `derivedField1` | `embedded_col → json_key` | `embedded_json_adapter` | `[]` |
+| `derivedField2` | `embedded_col → ## Section Header` | `embedded_markdown_adapter` | `None` |
+
+---
+
+## {EntityName} — {repository_module}.fetch_{entity}_by_id
+
+**UI fields populated:** same as list + detail-only fields
+**Called by:** `{EntityService}.get_by_id` → `normalize_{entity}` → `{EntityResponse}`
+
+```sql
+-- Single-row lookup by primary key
+-- Parameters:
+--   :record_id  VARCHAR  primary key value
+SELECT
+    t.primary_key_col     AS primary_key,
+    t.display_col_a       AS field_a,
+    t.display_col_b       AS field_b,
+    t.detail_only_col     AS detail_field,
+    t.embedded_col        AS embedded_col
+FROM {schema}.{SourceTable} AS t
+WHERE t.primary_key_col = :record_id;
+```

package/bundles/antigravity-plugin-extras/skills/data-contract-pipeline/references/consumer-driven-contracts.md

@@ -0,0 +1,131 @@
+# Consumer-Driven Contracts
+
+Reference summary of the Consumer-Driven Contracts pattern for data pipeline alignment.
+
+---
+
+## Origin
+
+**Consumer-Driven Contracts** (Ian Robinson, Martin Fowler — 2006) is a pattern for managing service integration where the **consumer** declares what it needs from a provider, and the provider validates that it satisfies all consumer expectations.
+
+---
+
+## Core Principles
+
+### 1. Contracts Are Code, Not Docs
+
+Documentation drifts. Contract **tests** are the source of truth.
+
+```
+❌ "The API returns these fields" (in a wiki page)
+✅ test_api_returns_expected_fields() (in the test suite)
+```
+
+### 2. Consumer Declares Needs
+
+The frontend (consumer) declares which fields it needs. The backend (provider) validates it serves them. This inverts the usual direction — instead of the backend dictating "here's what I send", the frontend says "here's what I need."
+
+In practice: TypeScript interfaces define the contract. Python DTOs validate they provide those fields.
+
+### 3. Additive Evolution
+
+Schema changes are additive — add optional fields with defaults. Never remove or rename a field without coordinating with all consumers.
+
+```python
+# ✅ Safe evolution: new optional field with default
+class MetricsResponse(BaseModel):
+    nps: float
+    responses: int
+    # New field — existing consumers ignore it, new consumers use it
+    trend_direction: str = Field(default="stable", alias="trendDirection")
+```
+
+### 4. Just Enough Validation
+
+Validate what matters at each layer:
+
+| Layer | Validates |
+|-------|-----------|
+| Ingestion Model | "Does the raw data have the expected shape?" |
+| Display DTO | "Does the API output have the exact contracted shape?" |
+| Contract Test | "Do all layers agree on the same shape?" |
+
+Don't over-validate at every layer. The ingestion model can be lenient (`extra="allow"`). The display DTO must be strict (`extra="forbid"`).
+
+### 5. Fail Fast, Fail Loud
+
+When a contract violation is detected, it should be an error — not a warning, not a log line.
+
+```python
+# ❌ Silent handling
+try:
+    dto = MyDTO.model_validate(data)
+except ValidationError:
+    return {}  # Silent failure — consumer gets empty data
+
+# ✅ Fail loud
+dto = MyDTO.model_validate(data)  # Raises ValidationError with details
+```
+
+---
+
+## How This Maps to Our Pipeline
+
+| CDC Concept | Our Implementation |
+|-------------|-------------------|
+| Consumer | Frontend TypeScript types + service functions |
+| Provider | Backend display DTO + API router |
+| Contract | Contract tests in `tests/unit/test_contract_*.py` |
+| Contract verification | `pytest` (local) → CI pipeline (when available) |
+| Golden sample | `tests/fixtures/payloads/{source}/sample.json` |
+| Schema evolution | Additive fields with defaults, never `extra="ignore"` |
+
+---
+
+## Golden Sample Pattern
+
+A **golden sample** is a curated, representative payload from the source that serves as the reference for all contract tests.
+
+### Properties of a Good Golden Sample
+
+1. **Complete**: Contains all known keys, including optional ones
+2. **Representative**: Values are realistic (not all null or placeholder)
+3. **Sanitized**: No PII, credentials, or sensitive data
+4. **Versioned**: Old samples renamed (not deleted) when schema changes
+5. **Committed**: Checked into version control, not generated at test time
+
+### How to Create One
+
+1. Copy a real payload from the source (API response, JSON file, DB query result)
+2. Sanitize: replace real names, emails, IDs with realistic fakes
+3. Save to `tests/fixtures/payloads/{source_name}/sample.json`
+4. Commit with a descriptive message: `test: add golden sample for {source_name}`
+
+---
+
+## Postel's Law and Its Limits
+
+> "Be conservative in what you send, liberal in what you accept."
+> — *Jon Postel, RFC 761 (1980)*
+
+This works for **ingestion models** (accept broad input to avoid brittle failures on new keys).
+
+It does NOT work for **display DTOs** at the API boundary. Display DTOs must be explicit:
+
+- **Conservative in what you send**: Only send fields the consumer declared it needs
+- **Explicit about what you accept**: `extra="forbid"` — don't silently drop or add fields
+
+The failure mode of Postel's Law at API boundaries: the backend silently drops a field (`extra="ignore"`), the frontend silently renders `undefined`, and nobody notices until a stakeholder asks "why is this blank?"
+
+---
+
+## When to Use CDC vs. Other Patterns
+
+| Pattern | Best For |
+|---------|----------|
+| Consumer-Driven Contracts | Frontend ↔ Backend in same team/repo |
+| Provider Contracts (Pact) | Microservices across teams |
+| Schema Registry (Avro/Protobuf) | Event streaming (Kafka, EventBridge) |
+| OpenAPI / Swagger | Public APIs with external consumers |
+
+For most internal dashboards and applications, Consumer-Driven Contracts with golden sample tests is the right level of formality.

package/bundles/antigravity-plugin-extras/skills/data-contract-pipeline/references/drift-check-catalog.md

@@ -0,0 +1,305 @@
+# Drift Check Catalog
+
+Complete reference of contract drift types, detection methods, and fix patterns.
+
+---
+
+## Drift Types
+
+### D1 — Payload Key Not in Ingestion Model
+
+**What**: A key exists in the source payload but has no corresponding field in the ingestion model — at **any nesting depth**, not just top-level.
+
+**Risk**: New data from the source is silently ignored. If the ingestion model uses `extra="ignore"`, this is invisible.
+
+**Detection — flat (top-level only)**:
+```python
+payload_keys = set(payload.keys())
+model_aliases = {
+    info.alias or name for name, info in IngestionModel.model_fields.items()
+}
+uncovered = payload_keys - model_aliases
+```
+
+**Detection — recursive (recommended)**:
+```python
+from contract_test_helpers import _collect_dropped_keys
+
+dropped = _collect_dropped_keys(payload, IngestionModel)
+# Returns dot-path list: ["surveys.broadband.newField", "metadata.x"]
+```
+
+The recursive variant (`_collect_dropped_keys` in the contract-test template) walks nested dicts and lists-of-dicts, comparing keys against model field aliases at every depth. It also respects the `KNOWN_EXCLUDED_KEYS` allow-list (see D12).
+
+**Fix**: Add the missing field to the ingestion model with the correct type.
+
+---
+
+### D2 — Ingestion Field Not in Display DTO
+
+**What**: The ingestion model has a field that doesn't map to any display DTO field. The normalizer silently drops it.
+
+**Risk**: Data is fetched and validated but never reaches the API consumer. If intentional, it should be documented.
+
+**Detection — flat (top-level only)**:
+```python
+ingestion_fields = set(IngestionModel.model_fields.keys())
+dto_fields = set(DisplayDTO.model_fields.keys())
+dropped = ingestion_fields - dto_fields  # Approximate — normalizer may rename
+```
+
+**Detection — recursive (recommended)**:
+```python
+from contract_test_helpers import _collect_dropped_keys
+
+dropped = _collect_dropped_keys(payload, DisplayDTO)
+# Returns dot-path list at every nesting level
+```
+
+**Fix**: Either add the field to the display DTO, or add the key to `KNOWN_EXCLUDED_KEYS` (see D12) with a ticket reference explaining why it's excluded.
+
+---
+
+### D3 — Display DTO Uses `extra="ignore"`
+
+**What**: The display DTO silently drops unknown fields during validation.
+
+**Risk**: When the source adds new keys, they pass through ingestion but vanish at the DTO layer. This causes data loss bugs that are nearly impossible to debug because no error is raised.
+
+**Detection**:
+```python
+config = DisplayDTO.model_config
+assert config.get("extra") != "ignore", "Display DTO must not use extra='ignore'"
+```
+
+**Fix**: Change to `extra="forbid"` (strict) or `extra="allow"` (migration period).
+
+---
+
+### D4 — DTO Alias Mismatch with TypeScript
+
+**What**: A DTO field alias (the camelCase name sent over the API) doesn't match the corresponding TypeScript interface property name.
+
+**Risk**: Frontend receives the data but accesses it with the wrong key → `undefined` values, silent display bugs.
+
+**Detection**:
+```python
+# Extract DTO aliases
+dto_aliases = {
+    info.alias or name for name, info in DisplayDTO.model_fields.items()
+}
+
+# Extract TypeScript property names (parse the .ts file)
+ts_props = extract_ts_interface_properties("frontend/src/types/source.ts")
+
+# Compare
+missing_in_ts = dto_aliases - ts_props
+extra_in_ts = ts_props - dto_aliases
+```
+
+**Fix**: Update the TypeScript interface to match DTO aliases exactly.
+
+---
+
+### D5 — DTO Type Mismatch with TypeScript
+
+**What**: A DTO field type doesn't correspond to the TypeScript type.
+
+**Risk**: Runtime type errors, NaN from treating strings as numbers, etc.
+
+**Detection**: Compare using the type mapping table:
+
+| Python | TypeScript |
+|--------|-----------|
+| `str` | `string` |
+| `int`, `float` | `number` |
+| `bool` | `boolean` |
+| `list[T]` | `T[]` |
+| `dict[str, T]` | `Record<string, T>` |
+| `Optional[T]` | `T \| null` |
+
+**Fix**: Update the TypeScript type to match.
+
+---
+
+### D6 — API Router Missing `response_model`
+
+**What**: FastAPI endpoint doesn't specify `response_model=DtoName`.
+
+**Risk**: FastAPI won't validate the response shape. If the service returns a dict with wrong keys, the API sends it as-is.
+
+**Detection**:
+```python
+# Check router decorators
+# @router.get("/", response_model=DtoName) ← must be present
+```
+
+**Fix**: Add `response_model=DtoName` to the endpoint decorator.
+
+---
+
+### D7 — Mapping Doc Stale
+
+**What**: A data mapping document (e.g., `ui-json-data-mapping.md`) claims a UI element reads from `data.foo.bar`, but the code actually reads from `data.baz.qux`.
+
+**Risk**: Document misleads developers, causes incorrect debugging assumptions.
+
+**Detection**: Compare doc field paths against actual code references. This is a manual review item — scripts can't reliably parse prose mapping docs.
+
+**Fix**: Regenerate the mapping doc from the contract tests and DTO definitions.
+
+---
+
+### D8 — Golden Sample Missing or Stale
+
+**What**: No `tests/fixtures/payloads/source/sample.json` exists, or it's from an old schema version.
+
+**Risk**: Contract tests run against outdated data, pass despite real drift.
+
+**Detection**:
+```python
+sample_path = Path("tests/fixtures/payloads/source/sample.json")
+assert sample_path.exists(), "Golden sample missing"
+
+# Optionally check freshness
+payload = json.loads(sample_path.read_text())
+# Compare key count against actual source
+```
+
+**Fix**: Copy a real payload from the source, sanitize PII, save as golden sample.
+
+---
+
+### D9 — Frontend Service Return Type Wrong
+
+**What**: The frontend service function's return type doesn't match the TypeScript interface.
+
+**Risk**: TypeScript compiler may not catch it if the return type is `any` or a superset.
+
+**Detection**:
+```typescript
+// Check that the service function's return type matches the interface
+export async function getData(): Promise<CorrectType> { ... }
+//                                       ^^^^^^^^^^^^ must match
+```
+
+**Fix**: Update the return type annotation.
+
+---
+
+### D10 — Normalizer Produces Wrong Shape
+
+**What**: The normalizer function's output doesn't match the display DTO fields.
+
+**Risk**: Runtime `ValidationError` when the DTO tries to validate the normalizer output.
+
+**Detection**: Unit test — pass a valid ingestion model through the normalizer and validate the result against the DTO.
+
+**Fix**: Update the normalizer to produce all required DTO fields.
+
+---
+
+### D11 — Drift Test Mode Mismatch
+
+**What**: The drift test uses hard assertions (`assert dropped == []`) while the upstream schema is still evolving, OR uses soft warnings (`warnings.warn`) after the schema has been frozen.
+
+**Risk**:
+- Hard asserts during evolution: CI breaks on every upstream change → drift tests get skipped or deleted.
+- Soft warnings after freeze: Regressions slip through because CI never fails.
+
+**Detection**:
+```python
+# Check freeze resolution precedence in the contract test file:
+# 1) CONTRACT_SCHEMA_FROZEN env var
+# 2) _contract_state.json marker
+# 3) CI fallback (CI=true + sample exists)
+import os
+schema_frozen = resolve_schema_frozen()
+
+# If the upstream schema is frozen, SCHEMA_FROZEN should be True
+# If the upstream schema is still evolving, SCHEMA_FROZEN should be False
+```
+
+**Modes**:
+
+| Schema state | `SCHEMA_FROZEN` | Drift detection behaviour |
+|---|---|---|
+| Evolving daily | `False` | `warnings.warn()` — CI passes, drift is visible in test output |
+| Stabilised / frozen | `True` | `assert` — CI blocks, any untyped key is a failure |
+
+**Fix**: Prefer explicit CI override (`CONTRACT_SCHEMA_FROZEN=1`) once the upstream schema is stable. If explicit env is missing, maintain `_contract_state.json` (`{"schema_frozen": true}`) to avoid ambiguous mode selection.
+
+---
+
+### D12 — Missing Exclusion Allow-List Entry
+
+**What**: A JSON key is intentionally NOT typed in the DTO (deprecated, experimental, frontend-only metadata) but has no entry in `KNOWN_EXCLUDED_KEYS`, so it triggers noisy drift warnings.
+
+**Risk**: Drift reports become cluttered with known intentional gaps. Teams learn to ignore warnings, defeating the purpose of drift detection.
+
+**Detection**:
+```python
+from fnmatch import fnmatch
+
+# KNOWN_EXCLUDED_KEYS uses dot-path globs (fnmatch syntax)
+KNOWN_EXCLUDED_KEYS: dict[str, set[str]] = {
+    "root.section.*": {"experimental_field"},  # TICKET-123
+}
+
+# A key is excluded if its full dot-path matches a pattern AND
+# the leaf key name is in the exclusion set for that pattern.
+is_excluded = any(
+    fnmatch(full_path, pattern) and key in excl
+    for pattern, excl in KNOWN_EXCLUDED_KEYS.items()
+)
+```
+
+**Rules**:
+- Every entry MUST have a ticket reference or justification comment
+- Review the list periodically — remove entries once the key is typed or removed upstream
+- Use dot-path + fnmatch globs: `"surveys.*"` matches `surveys.broadband`, `surveys.mobile`, etc.
+- If a key appears in many paths, use a broad glob; if it's specific, use the exact path
+
+**Fix**: Add the intentionally excluded key to `KNOWN_EXCLUDED_KEYS` in the contract test file with a justification.
+
+---
+
+## Drift Check Execution Order
+
+Run checks in this order for the most useful output:
+
+1. **D8** — Golden sample exists?
+2. **D1** — Payload keys covered by ingestion model? (recursive)
+3. **D3** — DTO `extra` config safe?
+4. **D2** — Ingestion fields mapped to DTO? (recursive)
+5. **D10** — Normalizer produces valid DTO?
+6. **D11** — Schema frozen flag matches upstream state?
+7. **D12** — Exclusion allow-list entries all have justifications?
+8. **D4** — DTO aliases match TypeScript?
+9. **D5** — DTO types match TypeScript?
+10. **D6** — Router has `response_model`?
+11. **D9** — Frontend service return type correct?
+12. **D7** — Mapping doc current?
+
+---
+
+## Drift Report Format
+
+When reporting drift, use this format:
+
+```markdown
+## Drift Report — {{source_name}}
+
+| Check | Status | Details |
+|-------|--------|---------|
+| D1 — Payload Coverage | ✅ PASS | 42/42 keys covered |
+| D2 — DTO Coverage | ❌ FAIL | `byPlay` in ingestion, not in DTO |
+| D3 — DTO Extra Config | ✅ PASS | `extra="forbid"` |
+| D4 — TS Alias Match | ❌ FAIL | DTO alias `npsDeltas`, TS has `nps_deltas` |
+| ... | ... | ... |
+
+### Fixes Required
+
+1. **D2 — `byPlay` not in DTO**: Add `by_play: dict = Field(alias="byPlay")` to `NpsMonthlyResponse`
+2. **D4 — `npsDeltas` alias**: Rename TS property from `nps_deltas` to `npsDeltas`
+```