@altimateai/altimate-code 0.4.9 → 0.5.1

package/skills/dbt-develop/SKILL.md

@@ -0,0 +1,151 @@
+---
+name: dbt-develop
+description: Create and modify dbt models — staging, intermediate, marts, incremental, medallion architecture. Use when building new SQL models, extending existing ones, scaffolding YAML configs, or reorganizing project structure. Powered by altimate-dbt.
+---
+
+# dbt Model Development
+
+## Requirements
+**Agent:** builder or migrator (requires file write access)
+**Tools used:** bash (runs `altimate-dbt` commands), read, glob, write, edit, schema_search, dbt_profiles, sql_analyze, altimate_core_validate, altimate_core_column_lineage
+
+## When to Use This Skill
+
+**Use when the user wants to:**
+- Create a new dbt model (staging, intermediate, mart, OBT)
+- Add or modify SQL logic in an existing model
+- Generate sources.yml or schema.yml from warehouse metadata
+- Reorganize models into layers (staging/intermediate/mart or bronze/silver/gold)
+- Convert a model to incremental materialization
+- Scaffold a new dbt project structure
+
+**Do NOT use for:**
+- Adding tests to models → use `dbt-test`
+- Writing model/column descriptions → use `dbt-docs`
+- Debugging build failures → use `dbt-troubleshoot`
+- Analyzing change impact → use `dbt-analyze`
+
+## Core Workflow: Plan → Discover → Write → Validate
+
+### 1. Plan — Understand Before Writing
+
+Before writing any SQL:
+- Read the task requirements carefully
+- Identify which layer this model belongs to (staging, intermediate, mart)
+- Check existing models for naming conventions and patterns
+- **Check dependencies:** If `packages.yml` exists, check for `dbt_packages/` or `package-lock.yml`. Only run `dbt deps` if packages are declared but not yet installed.
+
+```bash
+altimate-dbt info                           # project name, adapter type
+altimate-dbt parents --model <upstream>     # understand what feeds this model
+altimate-dbt children --model <downstream>  # understand what consumes it
+```
+
+**Check warehouse connection:** Run `dbt_profiles` to discover available profiles and map them to warehouse connections. This tells you which adapter (Snowflake, BigQuery, Postgres, etc.) and target the project uses — essential for dialect-aware SQL.
+
+
+### 2. Discover — Understand the Data Before Writing
+
+**Never write SQL without deeply understanding your data first.** The #1 cause of wrong results is writing SQL blind — assuming grain, relationships, column names, or values without checking.
+
+**Step 2a: Search for relevant tables and columns**
+- Use `schema_search` with natural-language queries to find tables/columns in large warehouses (e.g., `schema_search(query: "customer orders")` returns matching tables and columns from the indexed schema cache)
+- Read `sources.yml`, `schema.yml`, and any YAML files that describe the source/parent models
+- These contain column descriptions, data types, tests, and business context
+- Pay special attention to: primary keys, unique constraints, relationships between tables, and what each column represents
+
+**Step 2b: Understand the grain of each parent model/source**
+- What does one row represent? (one customer? one event? one day per customer?)
+- What are the primary/unique keys?
+- This is critical for JOINs — joining on the wrong grain causes fan-out (too many rows) or missing rows
+
+```bash
+altimate-dbt columns --model <name>                         # existing model columns
+altimate-dbt columns-source --source <src> --table <tbl>    # source table columns
+altimate-dbt execute --query "SELECT count(*) FROM {{ ref('model') }}" --limit 1
+altimate-dbt execute --query "SELECT * FROM {{ ref('model') }}" --limit 5
+altimate-dbt column-values --model <name> --column <col>    # sample values for key columns
+```
+
+**Step 2c: Query the actual data to verify your understanding**
+- Check row counts, NULLs, date ranges, cardinality of key columns
+- Verify foreign key relationships actually hold (do all IDs in child exist in parent?)
+- Check for duplicates in what you think are unique keys
+
+**Step 2d: Read existing models that your new model will reference**
+- Read the actual SQL of parent models — understand their logic, filters, and transformations
+- Read 2-3 existing models in the same directory to match patterns and conventions
+
+```bash
+glob models/**/*.sql     # find all model files
+read <model_file>        # understand existing patterns and logic
+```
+
+### 3. Write — Follow Layer Patterns
+
+See [references/layer-patterns.md](references/layer-patterns.md) for staging/intermediate/mart templates.
+See [references/medallion-architecture.md](references/medallion-architecture.md) for bronze/silver/gold patterns.
+See [references/incremental-strategies.md](references/incremental-strategies.md) for incremental materialization.
+See [references/yaml-generation.md](references/yaml-generation.md) for sources.yml and schema.yml.
+
+### 4. Validate — Build, Verify, Check Impact
+
+Never stop at writing the SQL. Always validate:
+
+**Build it:**
+```bash
+altimate-dbt compile --model <name>                        # catch Jinja errors
+altimate-dbt build --model <name>                          # materialize + run tests
+```
+
+**Verify the output:**
+```bash
+altimate-dbt columns --model <name>                        # confirm expected columns exist
+altimate-dbt execute --query "SELECT count(*) FROM {{ ref('<name>') }}" --limit 1
+altimate-dbt execute --query "SELECT * FROM {{ ref('<name>') }}" --limit 10  # spot-check values
+```
+- Do the columns match what schema.yml or the task expects?
+- Does the row count make sense? (no fan-out from bad joins, no missing rows from wrong filters)
+- Are values correct? (spot-check NULLs, aggregations, date ranges)
+
+**Check SQL quality** (on the compiled SQL from `altimate-dbt compile`):
+- `sql_analyze` — catches anti-patterns (SELECT *, cartesian products, missing filters)
+- `altimate_core_validate` — validates syntax and schema references
+- `altimate_core_column_lineage` — traces how source columns flow to output columns. Use this to verify your SELECT is pulling the right columns from the right sources, especially for complex JOINs or multi-CTE models.
+
+**Check downstream impact** (when modifying an existing model):
+```bash
+altimate-dbt children --model <name>                       # who depends on this?
+altimate-dbt build --model <name> --downstream             # rebuild downstream to catch breakage
+```
+Use `altimate-dbt children` and `altimate-dbt parents` to verify the DAG is intact when changes could affect downstream models.
+
+## Iron Rules
+
+1. **Never write SQL without reading the source columns first.** Use `altimate-dbt columns` or `altimate-dbt columns-source`.
+2. **Never stop at compile.** Always `altimate-dbt build` to catch runtime errors.
+3. **Match existing patterns.** Read 2-3 existing models in the same directory before writing.
+4. **One model, one purpose.** A staging model should not contain business logic. An intermediate model should not be materialized as a table unless it has consumers.
+5. **Fix ALL errors, not just yours.** After creating/modifying models, run a full `dbt build`. If ANY model fails — even pre-existing ones you didn't touch — fix them. Your job is to leave the project in a fully working state.
+
+## Common Mistakes
+
+| Mistake | Fix |
+|---------|-----|
+| Writing SQL without checking column names | Run `altimate-dbt columns` or `altimate-dbt columns-source` first |
+| Stopping at `compile` — "it compiled, ship it" | Always `altimate-dbt build` to materialize and run tests |
+| Hardcoding table references instead of `{{ ref() }}` | Always use `{{ ref('model') }}` or `{{ source('src', 'table') }}` |
+| Creating a staging model with JOINs | Staging = 1:1 with source. JOINs belong in intermediate or mart |
+| Not checking existing naming conventions | Read existing models in the same directory first |
+| Using `SELECT *` in final models | Explicitly list columns for clarity and contract stability |
+
+## Reference Guides
+
+| Guide | Use When |
+|-------|----------|
+| [references/altimate-dbt-commands.md](references/altimate-dbt-commands.md) | Need the full CLI reference |
+| [references/layer-patterns.md](references/layer-patterns.md) | Creating staging, intermediate, or mart models |
+| [references/medallion-architecture.md](references/medallion-architecture.md) | Organizing into bronze/silver/gold layers |
+| [references/incremental-strategies.md](references/incremental-strategies.md) | Converting to incremental materialization |
+| [references/yaml-generation.md](references/yaml-generation.md) | Generating sources.yml or schema.yml |
+| [references/common-mistakes.md](references/common-mistakes.md) | Extended anti-patterns catalog |

package/skills/dbt-develop/references/altimate-dbt-commands.md

@@ -0,0 +1,66 @@
+# altimate-dbt Command Reference
+
+All dbt operations use the `altimate-dbt` CLI. Output is JSON to stdout; logs go to stderr.
+
+```bash
+altimate-dbt <command> [args...]
+altimate-dbt <command> [args...] --format text    # Human-readable output
+```
+
+## First-Time Setup
+
+```bash
+altimate-dbt init                          # Auto-detect project root
+altimate-dbt init --project-root /path     # Explicit root
+altimate-dbt init --python-path /path      # Override Python
+altimate-dbt doctor                        # Verify setup
+altimate-dbt info                          # Project name, adapter, root
+```
+
+## Build & Run
+
+```bash
+altimate-dbt build --model <name> [--downstream]   # compile + run + test
+altimate-dbt run --model <name> [--downstream]      # materialize only
+altimate-dbt test --model <name>                     # run tests only
+altimate-dbt build-project                           # full project build
+```
+
+## Compile
+
+```bash
+altimate-dbt compile --model <name>
+altimate-dbt compile-query --query "SELECT * FROM {{ ref('stg_orders') }}" [--model <context>]
+```
+
+## Execute SQL
+
+```bash
+altimate-dbt execute --query "SELECT count(*) FROM {{ ref('orders') }}" --limit 100
+```
+
+## Schema & DAG
+
+```bash
+altimate-dbt columns --model <name>                         # column names and types
+altimate-dbt columns-source --source <src> --table <tbl>    # source table columns
+altimate-dbt column-values --model <name> --column <col>    # sample values
+altimate-dbt children --model <name>                        # downstream models
+altimate-dbt parents --model <name>                         # upstream models
+```
+
+## Packages
+
+```bash
+altimate-dbt deps                                           # install packages.yml
+altimate-dbt add-packages --packages dbt-utils,dbt-expectations
+```
+
+## Error Handling
+
+All errors return JSON with `error` and `fix` fields:
+```json
+{ "error": "dbt-core is not installed", "fix": "Install it: python3 -m pip install dbt-core" }
+```
+
+Run `altimate-dbt doctor` as the first diagnostic step for any failure.

package/skills/dbt-develop/references/common-mistakes.md

@@ -0,0 +1,49 @@
+# Common dbt Development Mistakes
+
+## SQL Mistakes
+
+| Mistake | Why It's Wrong | Fix |
+|---------|---------------|-----|
+| `SELECT *` in mart/fact models | Breaks contracts when upstream adds columns | Explicitly list all columns |
+| Hardcoded table names | Breaks when schema/database changes | Use `{{ ref() }}` and `{{ source() }}` |
+| Business logic in staging models | Staging = 1:1 source mirror | Move logic to intermediate layer |
+| JOINs in staging models | Same as above | Move JOINs to intermediate layer |
+| `LEFT JOIN` when `INNER JOIN` is correct | Returns NULLs for unmatched rows | Think about what NULL means for your use case |
+| Missing `GROUP BY` columns | Query fails or returns wrong results | Every non-aggregate column must be in GROUP BY |
+| Window functions without `PARTITION BY` | Aggregates across entire table | Add appropriate partitioning |
+
+## Project Structure Mistakes
+
+| Mistake | Fix |
+|---------|-----|
+| Models in wrong layer directory | staging/ for source mirrors, intermediate/ for transforms, marts/ for business tables |
+| No schema.yml for new models | Always create companion YAML with at minimum `unique` + `not_null` on primary key |
+| Naming doesn't match convention | Check existing models: `stg_`, `int_`, `fct_`, `dim_` prefixes |
+| Missing `{{ config() }}` block | Every model should declare materialization explicitly or inherit from dbt_project.yml |
+
+## Incremental Mistakes
+
+| Mistake | Fix |
+|---------|-----|
+| No `unique_key` on merge strategy | Causes duplicates. Set `unique_key` to your primary key |
+| Using `created_at` for mutable records | Use `updated_at` — `created_at` misses updates |
+| No lookback window | Use `max(ts) - interval '1 hour'` to catch late-arriving data |
+| Forgetting `is_incremental()` returns false on first run | The `WHERE` clause only applies after first run |
+
+## Validation Mistakes
+
+| Mistake | Fix |
+|---------|-----|
+| "It compiled, ship it" | Compilation only checks Jinja syntax. Always `altimate-dbt build` |
+| Not spot-checking output data | Run `altimate-dbt execute --query "SELECT * FROM {{ ref('model') }}" --limit 10` |
+| Not checking row counts | Compare source vs output: `SELECT count(*) FROM {{ ref('model') }}` |
+| Skipping downstream builds | Use `altimate-dbt build --model <name> --downstream` |
+
+## Rationalizations to Resist
+
+| You're Thinking... | Reality |
+|--------------------|---------|
+| "I'll add tests later" | You won't. Add them now. |
+| "SELECT * is fine for now" | It will break when upstream changes. List columns explicitly. |
+| "This model is temporary" | Nothing is more permanent than a temporary solution. |
+| "The data looks right at a glance" | Run the build. Check the tests. Spot-check edge cases. |

package/skills/dbt-develop/references/incremental-strategies.md

@@ -0,0 +1,118 @@
+# Incremental Materialization Strategies
+
+## When to Use Incremental
+
+Use incremental when:
+- Table has millions+ rows
+- Source data is append-only or has reliable `updated_at` timestamps
+- Full refreshes take too long or cost too much
+
+Do NOT use incremental when:
+- Table is small (< 1M rows)
+- Source data doesn't have reliable timestamps
+- Logic requires full-table window functions
+
+## Strategy Decision Tree
+
+```
+Is the data append-only (events, logs)?
+  YES → Append strategy
+  NO → Can rows be updated?
+    YES → Does your warehouse support MERGE?
+      YES → Merge/Upsert strategy
+      NO → Delete+Insert strategy
+    NO → Is data date-partitioned?
+      YES → Insert Overwrite strategy
+      NO → Append with dedup
+```
+
+## Append (Event Logs)
+
+```sql
+{{ config(
+    materialized='incremental',
+    on_schema_change='append_new_columns'
+) }}
+
+select
+    event_id,
+    event_type,
+    created_at
+from {{ ref('stg_events') }}
+
+{% if is_incremental() %}
+where created_at > (select max(created_at) from {{ this }})
+{% endif %}
+```
+
+## Merge/Upsert (Mutable Records)
+
+```sql
+{{ config(
+    materialized='incremental',
+    unique_key='order_id',
+    merge_update_columns=['status', 'updated_at', 'amount'],
+    on_schema_change='sync_all_columns'
+) }}
+
+select
+    order_id,
+    status,
+    amount,
+    created_at,
+    updated_at
+from {{ ref('stg_orders') }}
+
+{% if is_incremental() %}
+where updated_at > (select max(updated_at) from {{ this }})
+{% endif %}
+```
+
+## Insert Overwrite (Partitioned)
+
+```sql
+{{ config(
+    materialized='incremental',
+    incremental_strategy='insert_overwrite',
+    partition_by={'field': 'event_date', 'data_type': 'date'},
+    on_schema_change='fail'
+) }}
+
+select
+    date_trunc('day', created_at) as event_date,
+    count(*) as event_count
+from {{ ref('stg_events') }}
+
+{% if is_incremental() %}
+where date_trunc('day', created_at) >= (select max(event_date) - interval '3 days' from {{ this }})
+{% endif %}
+
+group by 1
+```
+
+## Common Pitfalls
+
+| Issue | Problem | Fix |
+|-------|---------|-----|
+| Missing `unique_key` | Duplicates on re-run | Add `unique_key` matching the primary key |
+| Wrong timestamp column | Missed updates | Use `updated_at` not `created_at` for mutable data |
+| No lookback window | Late-arriving data missed | `max(ts) - interval '1 hour'` instead of strict `>` |
+| `on_schema_change='fail'` | Breaks on column additions | Use `'append_new_columns'` or `'sync_all_columns'` |
+| Full refresh needed | Schema drift accumulated | `altimate-dbt run --model <name>` with `--full-refresh` flag |
+
+## Official Documentation
+
+For the latest syntax and adapter-specific options, refer to:
+- **dbt incremental models**: https://docs.getdbt.com/docs/build/incremental-models
+- **Incremental strategies by adapter**: https://docs.getdbt.com/docs/build/incremental-strategy
+- **Configuring incremental models**: https://docs.getdbt.com/reference/resource-configs/materialized#incremental
+
+## Warehouse Support
+
+| Warehouse | Default Strategy | Merge | Partition | Notes |
+|-----------|-----------------|-------|-----------|-------|
+| Snowflake | `merge` | Yes | Cluster keys | Best incremental support |
+| BigQuery | `merge` | Yes | `partition_by` | Requires partition for insert_overwrite |
+| PostgreSQL | `append` | No | No | Use delete+insert pattern |
+| DuckDB | `append` | Partial | No | Limited incremental support |
+| Redshift | `append` | No | dist/sort keys | Use delete+insert pattern |

package/skills/dbt-develop/references/layer-patterns.md

@@ -0,0 +1,158 @@
+# dbt Layer Patterns
+
+## Staging (`stg_`)
+
+**Purpose**: 1:1 with source table. Rename, cast, no joins, no business logic.
+**Materialization**: `view`
+**Naming**: `stg_<source>__<table>.sql`
+**Location**: `models/staging/<source>/`
+
+```sql
+with source as (
+    select * from {{ source('source_name', 'table_name') }}
+),
+
+renamed as (
+    select
+        -- Primary key
+        column_id as table_id,
+
+        -- Dimensions
+        column_name,
+
+        -- Timestamps
+        created_at,
+        updated_at
+
+    from source
+)
+
+select * from renamed
+```
+
+**Rules**:
+- One CTE named `source`, one named `renamed` (or `cast`, `cleaned`)
+- Only type casting, renaming, deduplication
+- No joins, no filters (except dedup), no business logic
+
+## Intermediate (`int_`)
+
+**Purpose**: Business logic, joins, transformations between staging and marts.
+**Materialization**: `ephemeral` or `view`
+**Naming**: `int_<entity>__<verb>.sql` (e.g., `int_orders__joined`, `int_payments__pivoted`)
+**Location**: `models/intermediate/`
+
+```sql
+with orders as (
+    select * from {{ ref('stg_source__orders') }}
+),
+
+customers as (
+    select * from {{ ref('stg_source__customers') }}
+),
+
+joined as (
+    select
+        orders.order_id,
+        orders.customer_id,
+        customers.customer_name,
+        orders.order_date,
+        orders.amount
+    from orders
+    left join customers on orders.customer_id = customers.customer_id
+)
+
+select * from joined
+```
+
+**Rules**:
+- Cross-source joins allowed
+- Business logic transformations
+- Not exposed to end users directly
+- Name the verb: `__joined`, `__pivoted`, `__filtered`, `__aggregated`
+
+## Mart: Facts (`fct_`)
+
+**Purpose**: Business events. Immutable, timestamped, narrow.
+**Materialization**: `table` or `incremental`
+**Naming**: `fct_<entity>.sql`
+**Location**: `models/marts/<domain>/`
+
+```sql
+with final as (
+    select
+        order_id,
+        customer_id,
+        order_date,
+        amount,
+        discount_amount,
+        amount - discount_amount as net_amount
+    from {{ ref('int_orders__joined') }}
+)
+
+select * from final
+```
+
+## Mart: Dimensions (`dim_`)
+
+**Purpose**: Descriptive attributes. Slowly changing, wide.
+**Materialization**: `table`
+**Naming**: `dim_<entity>.sql`
+
+```sql
+with final as (
+    select
+        customer_id,
+        customer_name,
+        email,
+        segment,
+        first_order_date,
+        most_recent_order_date,
+        lifetime_order_count
+    from {{ ref('int_customers__aggregated') }}
+)
+
+select * from final
+```
+
+## One Big Table (`obt_`)
+
+**Purpose**: Denormalized wide table combining fact + dimensions for BI consumption.
+**Materialization**: `table`
+**Naming**: `obt_<entity>.sql`
+
+```sql
+with facts as (
+    select * from {{ ref('fct_orders') }}
+),
+
+customers as (
+    select * from {{ ref('dim_customers') }}
+),
+
+dates as (
+    select * from {{ ref('dim_dates') }}
+),
+
+final as (
+    select
+        facts.*,
+        customers.customer_name,
+        customers.segment,
+        dates.day_of_week,
+        dates.month_name,
+        dates.is_weekend
+    from facts
+    left join customers on facts.customer_id = customers.customer_id
+    left join dates on facts.order_date = dates.date_day
+)
+
+select * from final
+```
+
+## CTE Style Guide
+
+- Name CTEs after what they contain, not what they do: `orders` not `get_orders`
+- Use `final` as the last CTE name
+- One CTE per source model (via `ref()` or `source()`)
+- End every model with `select * from final`

package/skills/dbt-develop/references/medallion-architecture.md

@@ -0,0 +1,125 @@
+# Medallion Architecture (Bronze / Silver / Gold)
+
+An alternative to staging/intermediate/mart layering. Common in Databricks and lakehouse environments.
+
+## Layer Mapping
+
+| Medallion | Traditional dbt | Purpose |
+|-----------|----------------|---------|
+| Bronze | Staging (`stg_`) | Raw ingestion, minimal transform |
+| Silver | Intermediate (`int_`) | Cleaned, conformed, joined |
+| Gold | Marts (`fct_`/`dim_`) | Business-ready aggregations |
+
+## Directory Structure
+
+```
+models/
+  bronze/
+    source_system/
+      _source_system__sources.yml
+      brz_source_system__table.sql
+  silver/
+    domain/
+      slv_domain__entity.sql
+  gold/
+    domain/
+      fct_metric.sql
+      dim_entity.sql
+```
+
+## Bronze (Raw)
+
+```sql
+-- brz_stripe__payments.sql
+{{ config(materialized='view') }}
+
+with source as (
+    select * from {{ source('stripe', 'payments') }}
+),
+
+cast as (
+    select
+        cast(id as varchar) as payment_id,
+        cast(amount as integer) as amount_cents,
+        cast(created as timestamp) as created_at,
+        _loaded_at
+    from source
+)
+
+select * from cast
+```
+
+**Rules**: 1:1 with source, only cast/rename, no joins, `view` materialization.
+
+## Silver (Cleaned)
+
+```sql
+-- slv_finance__orders_enriched.sql
+{{ config(materialized='table') }}
+
+with orders as (
+    select * from {{ ref('brz_stripe__payments') }}
+),
+
+customers as (
+    select * from {{ ref('brz_crm__customers') }}
+),
+
+enriched as (
+    select
+        o.payment_id,
+        o.amount_cents / 100.0 as amount_dollars,
+        c.customer_name,
+        c.segment,
+        o.created_at
+    from orders o
+    left join customers c on o.customer_id = c.customer_id
+    where o.created_at is not null
+)
+
+select * from enriched
+```
+
+**Rules**: Cross-source joins, business logic, quality filters, `table` materialization.
+
+## Gold (Business-Ready)
+
+```sql
+-- fct_daily_revenue.sql
+{{ config(materialized='table') }}
+
+with daily as (
+    select
+        date_trunc('day', created_at) as revenue_date,
+        segment,
+        count(*) as order_count,
+        sum(amount_dollars) as gross_revenue
+    from {{ ref('slv_finance__orders_enriched') }}
+    group by 1, 2
+)
+
+select * from daily
+```
+
+**Rules**: Aggregations, metrics, KPIs, `table` or `incremental` materialization.
+
+## dbt_project.yml Config
+
+```yaml
+models:
+  my_project:
+    bronze:
+      +materialized: view
+    silver:
+      +materialized: table
+    gold:
+      +materialized: table
+```
+
+## When to Use Medallion vs Traditional
+
+| Use Medallion When | Use Traditional When |
+|-------------------|---------------------|
+| Databricks/lakehouse environment | dbt Cloud or Snowflake-centric |
+| Team already uses bronze/silver/gold terminology | Team uses staging/intermediate/mart |
+| Data platform team maintains bronze layer | Analytics engineers own the full stack |