@revos/cli 0.1.2 → 0.1.4

package/dist/templates/skills/create-dbt-transformations/SKILL.md

@@ -0,0 +1,214 @@
+---
+name: create-dbt-transformations
+description: Create new dbt transformations (bronze/silver/gold models) in the RevOS dbt project. Use when asked to create a dbt model, build a transformation, add a new layer model, declare a raw source, or register a new Airbyte-ingested table. Covers dbt project conventions, sources, materialization, schema.yml, and validation commands.
+---
+
+# Create dbt Transformations
+
+Use this skill to generate SQL models, declare sources, update `schema.yml`, and validate models with `revos dbt run` / `revos dbt test`.
+
+For BigQuery exploration (listing datasets, inspecting raw tables, previewing rows, null rates), load the `explore-lakehouse` skill. If that skill is not installed, fall back to:
+
+```bash
+bq show --format=prettyjson $REVOS_BQ_DATASET.<table>
+```
+
+Warn the user: "The `explore-lakehouse` skill is not installed — using `bq show` as a fallback. Install it for richer schema exploration."
+
+---
+
+# Part 1: Knowledge Base
+
+## Layer Conventions
+
+- **gold** — business-ready models exposed for reporting or downstream consumption.
+- **silver** — cleaned, deduplicated, type-conformed intermediates.
+- **bronze** — thin views over raw source data. References sources via `{{ source() }}`.
+
+When layer is not obvious from context, ask (see Checkpoint 1).
+
+## Sources (bronze layer)
+
+Raw tables ingested by Airbyte are not dbt models. Declare them as dbt sources so bronze models can reference them with `{{ source() }}`.
+
+Sources are declared in `dbt/models/bronze/schema.yml` under a `sources:` block using `schema` (the BigQuery dataset):
+
+```yaml
+sources:
+  - name: raw
+    schema: "{{ env_var('REVOS_BQ_DATASET') }}"
+    tables:
+      - name: hubspot_contacts
+```
+
+Reference in bronze SQL:
+
+```sql
+SELECT * FROM {{ source('raw', 'hubspot_contacts') }}
+```
+
+See [schema-conventions.md](references/schema-conventions.md) for the full declaration pattern alongside `models:`.
+
+## Materialization
+
+Inherited globally from `dbt_project.yml` — do not add `{{ config(materialized=...) }}` unless the user explicitly asks to override.
+
+## `schema.yml` Convention
+
+One shared file per layer at `dbt/models/<layer>/schema.yml`. Append new models; never create per-model YAML files. See [schema-conventions.md](references/schema-conventions.md) for full examples and composite-PK / dbt-utils patterns.
+
+## Resolving Physical BigQuery Tables
+
+Materialized table lives at: `$REVOS_BQ_DATASET.<model_name>`
+
+**When to use `{{ ref() }}` vs. `{{ source() }}`:**
+
+| Context                             | Use                              |
+| ----------------------------------- | -------------------------------- |
+| dbt SQL → other dbt model           | `{{ ref('<model>') }}`           |
+| dbt SQL → raw source table (bronze) | `{{ source('raw', '<table>') }}` |
+
+Always declare raw tables as sources before referencing them. Do not use bare fully qualified names — that bypasses dbt's dependency graph and source freshness tracking.
+
+## Standard dbt Commands
+
+Always use the `revos` wrapper:
+
+```bash
+revos dbt parse                               # validate syntax (no warehouse)
+revos dbt compile --select <model>            # resolve refs, produce compiled SQL
+revos dbt run --select <model>                # execute against warehouse
+revos dbt test --select <model>               # run tests
+revos dbt build --select <model>              # run + test
+revos dbt build --select path:models/<layer>  # entire layer
+```
+
+---
+
+# Part 2: Workflow — Create a New dbt Transformation
+
+## Execution Order
+
+For each transformation (one at a time — do not batch):
+
+1. Determine the target layer (Checkpoint 1 if unclear).
+2. Determine the model name.
+3. Check if that model already exists (Checkpoint 2 if yes).
+4. Gather source data and transformation logic. For bridge models, apply the bridge template ([sql-templates.md](references/sql-templates.md)).
+5. For bronze models: check if required sources are declared in `dbt/models/bronze/schema.yml`; add them if missing.
+6. Generate `dbt/models/<layer>/<model_name>.sql`.
+7. Detect the primary key (Checkpoint 3 if ambiguous).
+8. Add model entry to `dbt/models/<layer>/schema.yml` with PK and FK tests. See [schema-conventions.md](references/schema-conventions.md).
+9. Run `revos dbt run --select <model_name>` and report result.
+10. Run `revos dbt test --select <model_name>` and report result.
+11. Summarize (see Final Response Format).
+
+For multiple transformations in one request: repeat steps 1–11 per model in order.
+
+---
+
+## Mandatory User Checkpoints
+
+### Checkpoint 1: Layer Selection
+
+Ask if the layer is not obvious:
+
+```text
+Which layer should this transformation live in?
+
+- gold: business-ready, exposed for reporting or downstream consumption
+- silver: cleaned/intermediate, shared across downstream uses
+- bronze: close-to-source view over raw data, references sources
+```
+
+Layer is obvious when the user explicitly names it.
+
+### Checkpoint 2: Existing Model Conflict
+
+If `dbt/models/<layer>/<model_name>.sql` exists:
+
+```text
+A model named <model_name> already exists at dbt/models/<layer>/<model_name>.sql.
+
+Options:
+- overwrite: replace with new transformation
+- edit:      modify existing (describe the change)
+- rename:    use a different name
+```
+
+If found in a different layer, mention it too.
+
+### Checkpoint 3: Ambiguous Primary Key
+
+If PK detection produces no clear result:
+
+```text
+I could not unambiguously detect the primary key. Candidates:
+- <candidate_1>
+- <candidate_2>
+
+Which column(s) should be the primary key?
+```
+
+---
+
+## Primary Key Detection
+
+Apply in order; stop at first clear result:
+
+1. `ROW_NUMBER() OVER (PARTITION BY <cols>) = 1` → partition columns are PK.
+2. `SELECT DISTINCT` over a small column set → all selected columns form composite PK.
+3. `GROUP BY <cols>` at outermost level → grouping columns are PK.
+4. Single column named `id` → PK.
+5. Single column named `<entity>_id` matching the model name stem → PK.
+6. Bridge naming `<entity_a>_<entity_b>` → `(<entity_a>_id, <entity_b>_id)` composite PK.
+
+If none produce a clear answer → Checkpoint 3.
+
+## Foreign Key Detection
+
+A column is a FK candidate if it matches `<entity>_id` where `<entity>` ≠ model's own entity, is not part of the PK, and is not nullable by design. Add `not_null` test only (no `relationships` tests by default).
+
+## SQL File Generation
+
+See [sql-templates.md](references/sql-templates.md) for:
+
+- Bronze model template using `{{ source() }}`
+- Standard silver/gold model template
+- Bridge model (JSON array) template with concrete example
+- Bridge model naming convention and SQL content rules
+
+## schema.yml Update
+
+See [schema-conventions.md](references/schema-conventions.md) for full examples including sources declaration, composite PK, and dbt-utils patterns.
+
+## Edge Cases
+
+See [edge-cases.md](references/edge-cases.md) for: missing SQL details, missing upstream model, undeclared source, run/test failure handling.
+
+---
+
+## Final Response Format
+
+```text
+Created dbt transformation: <model_name>
+
+Layer:           <bronze | silver | gold>
+File:            dbt/models/<layer>/<model_name>.sql
+Materialization: <inherited: table | overridden: <type>>
+Primary key:     <pk_column>  (or composite: <col_1>, <col_2>)
+Foreign keys:    <fk_1>, <fk_2>  (or "none detected")
+schema.yml:      dbt/models/<layer>/schema.yml (entry added)
+
+Tests:
+- not_null on <pk>: added
+- unique on <pk>: added | skipped: dbt-utils unavailable
+- not_null on <fk>: added
+
+Validation:
+- revos dbt run:  passed | failed
+- revos dbt test: passed | failed
+
+Physical table after run:
+`<resolved_dataset>.<model_name>`
+```

package/dist/templates/skills/create-dbt-transformations/references/edge-cases.md

@@ -0,0 +1,46 @@
+# Edge Cases
+
+## User asks for a model but does not provide the SQL
+
+```text
+I can scaffold the model file and schema entry, but I need to know what the
+transformation should produce.
+
+Could you tell me:
+1. What source model(s) or tables does it read from?
+2. What columns should it expose, and what is the primary key?
+3. Any filtering, aggregation, or join logic?
+
+Alternatively, if this is a bridge model from a JSON array, I can apply the
+standard bridge template — just tell me the source model and the JSON column.
+```
+
+## User asks for a "quick" or "simple" model without details
+
+Same response as above. Do not invent business logic.
+
+## Model depends on another model that does not exist yet
+
+```text
+The transformation you described references `<missing_model>`, which does not
+exist in dbt/models/. Should I create that model first?
+```
+
+## Source is a raw Airbyte table not yet declared as a dbt source
+
+Declare it as a source in `dbt/models/bronze/schema.yml` first (see [schema-conventions.md](schema-conventions.md)), then reference it with `{{ source('raw', '<table>') }}` in the bronze model SQL. Do not use fully qualified BigQuery names directly — that bypasses dbt's dependency graph and source freshness tracking.
+
+## run fails
+
+1. Show the error verbatim — do not paraphrase warehouse errors.
+2. Offer to fix the SQL based on the error message.
+3. Do not proceed to `revos dbt test` until run succeeds.
+
+## test fails
+
+Show which test failed and explain the likely cause:
+
+- `unique` on PK fails → real duplicates exist; PK detection was wrong or source has unexpected duplicates needing `DISTINCT` or `ROW_NUMBER` dedup.
+- `not_null` on a column fails → source has nulls; either the column is genuinely nullable (remove the test) or filter them out in SQL.
+
+Ask the user how to proceed.

package/dist/templates/skills/create-dbt-transformations/references/schema-conventions.md

@@ -0,0 +1,128 @@
+# schema.yml Conventions
+
+## Contents
+
+- [Declaring Sources (bronze layer)](#declaring-sources-bronze-layer)
+- [Standard Model Entry](#standard-model-entry)
+- [Composite Primary Keys (Bridge Models)](#composite-primary-keys-bridge-models)
+- [Description Guidelines](#description-guidelines)
+- [Foreign Key Tests](#foreign-key-tests)
+
+---
+
+Each layer has one shared `schema.yml` at `dbt/models/<layer>/schema.yml`. Append new models; do not create per-model files.
+
+If the file does not exist, create it with:
+
+```yaml
+version: 2
+
+models:
+```
+
+## Declaring Sources (bronze layer)
+
+Raw tables must be declared as dbt sources before they can be referenced with `{{ source() }}`. Sources live in `dbt/models/bronze/schema.yml` under a `sources:` block alongside the `models:` block.
+
+`schema` maps to the BigQuery dataset (`REVOS_BQ_DATASET`):
+
+```yaml
+version: 2
+
+sources:
+  - name: raw
+    schema: "{{ env_var('REVOS_BQ_DATASET') }}"
+    tables:
+      - name: hubspot_contacts
+      - name: hubspot_deals
+      - name: stripe_charges
+
+models:
+  - name: bronze_hubspot_contacts
+    ...
+```
+
+Rules:
+
+- Use `raw` as the source name for all Airbyte-ingested tables.
+- Each raw table referenced in bronze SQL needs a corresponding entry under `tables:`.
+- If the source block already exists, append to the `tables:` list only.
+
+## Standard Model Entry
+
+```yaml
+- name: <model_name>
+  description: |
+    <1–2 sentences: what entity/relationship, what source, any non-obvious logic>
+  columns:
+    - name: <pk_column>
+      description: "Primary key."
+      tests:
+        - not_null
+        - unique
+
+    - name: <fk_column>
+      description: "Foreign key to <target_entity>."
+      tests:
+        - not_null
+```
+
+## Composite Primary Keys (Bridge Models)
+
+If `dbt-utils` is available:
+
+```yaml
+- name: <model_name>
+  description: |
+    <description>
+  tests:
+    - dbt_utils.unique_combination_of_columns:
+        combination_of_columns:
+          - <pk_col_1>
+          - <pk_col_2>
+  columns:
+    - name: <pk_col_1>
+      description: "Composite key part."
+      tests:
+        - not_null
+    - name: <pk_col_2>
+      description: "Composite key part."
+      tests:
+        - not_null
+```
+
+If `dbt-utils` is not available, omit `unique_combination_of_columns` and note it in the description:
+
+```yaml
+description: |
+  <description>
+  Note: composite uniqueness on (<pk_col_1>, <pk_col_2>) is not enforced —
+  dbt-utils is not installed in this project.
+```
+
+Check availability:
+
+```bash
+grep -A2 "packages:" dbt/packages.yml 2>/dev/null | grep dbt-utils || echo "dbt-utils not found"
+```
+
+## Description Guidelines
+
+Answer in 1–2 sentences:
+
+1. What entity or relationship this model represents.
+2. What source(s) it reads from.
+3. Any non-obvious filtering or transformation.
+
+Example:
+
+```yaml
+description: |
+  Bridge table linking HubSpot deals to companies, unpacked from the
+  `companies` JSON array on `hubspot_deals`. Excludes deals with no
+  associated companies.
+```
+
+## Foreign Key Tests
+
+Add `not_null` only. Do not add `relationships` tests by default — they require knowing the target model and column, which needs explicit user input.

package/dist/templates/skills/create-dbt-transformations/references/sql-templates.md

@@ -0,0 +1,73 @@
+# SQL Templates
+
+## Bronze Model (source reference)
+
+Bronze models read raw Airbyte-ingested tables via `{{ source() }}`:
+
+```sql
+SELECT
+  <pk_column>,
+  <business_columns>,
+  _airbyte_extracted_at
+FROM {{ source('raw', '<raw_table_name>') }}
+```
+
+Ensure the source is declared in `dbt/models/bronze/schema.yml` before using it (see `references/schema-conventions.md`).
+
+## Standard Silver / Gold Model
+
+```sql
+SELECT
+  <pk_column>,
+  <business_columns>,
+  _airbyte_extracted_at
+FROM {{ ref('<source_model>') }}
+WHERE <filtering_conditions>
+```
+
+## Bridge Model (JSON Array)
+
+When unpacking a JSON array into a many-to-many bridge table:
+
+```sql
+SELECT DISTINCT
+  d.id              AS <entity_a>_id,
+  <entity_b>_id,
+  d._airbyte_extracted_at
+FROM {{ ref('<source_model>') }} d,
+UNNEST(JSON_VALUE_ARRAY(d.<json_array_column>)) AS <entity_b>_id
+WHERE d.<json_array_column> IS NOT NULL
+```
+
+Concrete example (`gold_deals_companies.sql`, unpacking `companies` array on `hubspot_deals`):
+
+```sql
+SELECT DISTINCT
+  d.id                      AS deal_id,
+  company_id,
+  d._airbyte_extracted_at
+FROM {{ ref('hubspot_deals') }} d,
+UNNEST(JSON_VALUE_ARRAY(d.companies)) AS company_id
+WHERE d.companies IS NOT NULL
+```
+
+Notes:
+
+1. `SELECT DISTINCT` — a single source row can produce duplicate combinations under some Airbyte sync patterns.
+2. `WHERE d.<json_array_column> IS NOT NULL` is required — `UNNEST(JSON_VALUE_ARRAY(NULL))` is unsafe.
+3. `_airbyte_extracted_at` is preserved for downstream freshness checks.
+4. Composite PK: `(<entity_a>_id, <entity_b>_id)`.
+
+## Bridge Model Naming
+
+Convention: `<entity_a>_<entity_b>` (no `to_`, no `bridge_` prefix). Alphabetical order unless one entity clearly owns the relationship.
+
+Examples: `gold_deals_companies`, `gold_deals_contacts`, `gold_companies_contacts`.
+
+## SQL Content Rules
+
+1. No `{{ config(materialized=...) }}` unless user asks to override the layer default.
+2. `{{ source('raw', '<table>') }}` for raw source tables in bronze models.
+3. `{{ ref('<model>') }}` for references to other dbt models.
+4. Named CTEs for non-trivial logic, explicit column lists where practical.
+5. Preserve `_airbyte_extracted_at` from Airbyte-ingested sources.