altimate-code 0.4.9 → 0.5.1

package/skills/dbt-test/references/unit-test-guide.md

@@ -0,0 +1,121 @@
+# dbt Unit Tests
+
+Unit tests validate model logic by mocking inputs and asserting expected outputs. Available in dbt-core 1.8+.
+
+## When to Unit Test
+
+**DO unit test:**
+- Complex calculations (revenue attribution, MRR changes, scoring)
+- Business logic with edge cases (null handling, date boundaries, status transitions)
+- Models with conditional logic (`CASE WHEN`, `IFF`, `COALESCE`)
+- Aggregations where correctness is critical
+
+**Do NOT unit test:**
+- Simple staging models (just rename/cast)
+- Pass-through models with no logic
+- Built-in dbt functions
+
+## Basic Structure
+
+Unit tests live in schema.yml (or a dedicated `_unit_tests.yml` file):
+
+```yaml
+unit_tests:
+  - name: test_net_revenue_calculation
+    description: Verify net_revenue = gross - refunds - discounts
+    model: fct_daily_revenue
+    given:
+      - input: ref('stg_orders')
+        rows:
+          - { order_id: 1, gross_amount: 100.00, refund_amount: 10.00, discount_amount: 5.00 }
+          - { order_id: 2, gross_amount: 50.00, refund_amount: 0.00, discount_amount: 0.00 }
+    expect:
+      rows:
+        - { order_id: 1, net_revenue: 85.00 }
+        - { order_id: 2, net_revenue: 50.00 }
+```
+
+## Mocking Multiple Inputs
+
+```yaml
+unit_tests:
+  - name: test_customer_lifetime_value
+    model: dim_customers
+    given:
+      - input: ref('stg_customers')
+        rows:
+          - { customer_id: 1, name: "Alice" }
+      - input: ref('stg_orders')
+        rows:
+          - { order_id: 1, customer_id: 1, amount: 50.00 }
+          - { order_id: 2, customer_id: 1, amount: 75.00 }
+    expect:
+      rows:
+        - { customer_id: 1, lifetime_value: 125.00 }
+```
+
+## Testing Edge Cases
+
+```yaml
+unit_tests:
+  - name: test_handles_null_discounts
+    model: fct_orders
+    given:
+      - input: ref('stg_orders')
+        rows:
+          - { order_id: 1, amount: 100.00, discount: null }
+    expect:
+      rows:
+        - { order_id: 1, net_amount: 100.00 }
+
+  - name: test_handles_zero_quantity
+    model: fct_orders
+    given:
+      - input: ref('stg_orders')
+        rows:
+          - { order_id: 1, quantity: 0, unit_price: 10.00 }
+    expect:
+      rows:
+        - { order_id: 1, order_total: 0.00 }
+```
+
+## Overriding Macros and Vars
+
+```yaml
+unit_tests:
+  - name: test_with_specific_date
+    model: fct_daily_metrics
+    overrides:
+      vars:
+        run_date: "2024-01-15"
+      macros:
+        - name: current_timestamp
+          result: "2024-01-15 00:00:00"
+    given:
+      - input: ref('stg_events')
+        rows:
+          - { event_id: 1, event_date: "2024-01-15" }
+    expect:
+      rows:
+        - { event_date: "2024-01-15", event_count: 1 }
+```
+
+## Running Unit Tests
+
+```bash
+altimate-dbt test --model <name>          # runs all tests including unit tests
+altimate-dbt build --model <name>         # build + test
+```
+
+## Test-Driven Development Pattern
+
+1. Write the unit test YAML first (expected inputs → outputs)
+2. Run it — it should fail (model doesn't exist yet or logic is missing)
+3. Write/fix the model SQL
+4. Run again — it should pass
+5. Add schema tests for data quality
+
+## Official Documentation
+
+- **dbt unit tests**: https://docs.getdbt.com/docs/build/unit-tests
+- **Unit test YAML spec**: https://docs.getdbt.com/reference/resource-properties/unit-tests

package/skills/dbt-troubleshoot/SKILL.md

@@ -0,0 +1,187 @@
+---
+name: dbt-troubleshoot
+description: Debug dbt errors — compilation failures, runtime database errors, test failures, wrong data, and performance issues. Use when something is broken, producing wrong results, or failing to build. Powered by altimate-dbt.
+---
+
+# dbt Troubleshooting
+
+## Requirements
+**Agent:** any (read-only diagnosis), builder (if applying fixes)
+**Tools used:** bash (runs `altimate-dbt` commands), read, glob, edit, altimate_core_semantics, altimate_core_column_lineage, altimate_core_correct, altimate_core_fix, sql_fix
+
+## When to Use This Skill
+
+**Use when:**
+- A dbt model fails to compile or build
+- Tests are failing
+- Model produces wrong or unexpected data
+- Builds are slow or timing out
+- User shares an error message from dbt
+
+**Do NOT use for:**
+- Creating new models → use `dbt-develop`
+- Adding tests → use `dbt-test`
+- Analyzing change impact → use `dbt-analyze`
+
+## Iron Rules
+
+1. **Never modify a test to make it pass without understanding why it's failing.**
+2. **Fix ALL errors, not just the reported one.** After fixing the specific issue, run a full `dbt build`. If other models fail — even ones not mentioned in the error report — fix them too. Your job is to leave the project in a fully working state. Never dismiss errors as "pre-existing" or "out of scope".
+
+## Diagnostic Workflow
+
+### Step 1: Health Check
+
+```bash
+altimate-dbt doctor
+altimate-dbt info
+```
+
+If `doctor` fails, fix the environment first. Common issues:
+- Python not found → reinstall or set `--python-path`
+- dbt-core not installed → `pip install dbt-core`
+- No `dbt_project.yml` → wrong directory
+- Missing packages → if `packages.yml` exists but `dbt_packages/` doesn't, run `dbt deps`
+
+### Step 2: Classify the Error
+
+| Error Type | Symptom | Jump To |
+|-----------|---------|---------|
+| Compilation Error | Jinja/YAML parse failure | [references/compilation-errors.md](references/compilation-errors.md) |
+| Runtime/Database Error | SQL execution failure | [references/runtime-errors.md](references/runtime-errors.md) |
+| Test Failure | Tests return failing rows | [references/test-failures.md](references/test-failures.md) |
+| Wrong Data | Model builds but data is incorrect | Step 3 below |
+
+### Step 3: Isolate the Problem
+
+```bash
+# Compile only — catches Jinja errors without hitting the database
+altimate-dbt compile --model <name>
+
+# If compile succeeds, try building
+altimate-dbt build --model <name>
+
+# Probe the data directly
+altimate-dbt execute --query "SELECT count(*) FROM {{ ref('<name>') }}" --limit 1
+altimate-dbt execute --query "SELECT * FROM {{ ref('<name>') }}" --limit 5
+```
+
+### Step 3b: Offline SQL Analysis
+
+Before hitting the database, analyze the compiled SQL offline:
+
+```bash
+# Check for semantic issues (wrong joins, cartesian products, NULL comparisons)
+altimate_core_semantics --sql <compiled_sql>
+
+# Trace column lineage to find where wrong data originates
+altimate_core_column_lineage --sql <compiled_sql>
+
+# Auto-suggest fixes for SQL errors
+altimate_core_correct --sql <compiled_sql>
+```
+
+**Quick-fix tools** — use these when the error type is clear:
+
+```
+# Schema-based fix: fuzzy-matches table/column names against schema to fix typos and wrong references
+altimate_core_fix(sql: <compiled_sql>, schema_context: <schema_object>)
+
+# Error-message fix: given a failing query + database error, analyzes root cause and proposes corrections
+sql_fix(sql: <compiled_sql>, error_message: <error_message>, dialect: <dialect>)
+```
+
+`altimate_core_fix` is best for compilation errors (wrong names, missing objects). `sql_fix` is best for runtime errors (the database told you what's wrong). Use `altimate_core_correct` for iterative multi-round correction when the first fix doesn't resolve the issue.
+
+
+Common findings:
+- **Wrong join type**: `INNER JOIN` dropping rows that should appear → switch to `LEFT JOIN`
+- **Fan-out**: One-to-many join inflating row counts → add deduplication or aggregate
+- **Column mismatch**: Output columns don't match schema.yml definition → reorder SELECT
+- **NULL comparison**: Using `= NULL` instead of `IS NULL` → silent data loss
+
+### Step 3c: Wrong Data Diagnosis — Deep Data Exploration
+
+When a model builds but produces wrong results, the bug is almost always in the data assumptions, not the SQL syntax. **You must explore the actual data to find it.**
+
+```bash
+# 1. Check the output for unexpected NULLs
+altimate-dbt execute --query "SELECT count(*) as total, count(<col>) as non_null, count(*) - count(<col>) as nulls FROM {{ ref('<name>') }}" --limit 1
+
+# 2. Check value ranges — are metrics within expected bounds?
+altimate-dbt execute --query "SELECT min(<metric>), max(<metric>), avg(<metric>) FROM {{ ref('<name>') }}" --limit 1
+
+# 3. Check distinct values for key columns — do they look right?
+altimate-dbt execute --query "SELECT <col>, count(*) FROM {{ ref('<name>') }} GROUP BY 1 ORDER BY 2 DESC" --limit 20
+
+# 4. Compare row counts between model output and parent tables
+altimate-dbt execute --query "SELECT count(*) FROM {{ ref('<parent>') }}" --limit 1
+```
+
+**Common wrong-data root causes:**
+- **Fan-out from joins**: If row count is higher than expected, a join key isn't unique — check with `SELECT key, count(*) ... GROUP BY 1 HAVING count(*) > 1`
+- **Missing rows from INNER JOIN**: If row count is lower than expected, switch to LEFT JOIN and check for NULL join keys
+- **Date spine issues**: If using `current_date` or `dbt_utils.date_spine`, output changes daily — check min/max dates
+
+### Step 4: Check Upstream
+
+Most errors cascade from upstream models:
+
+```bash
+altimate-dbt parents --model <name>
+```
+
+Read the parent models. Build them individually. **Query the parent data** — don't assume it's correct:
+```bash
+altimate-dbt execute --query "SELECT count(*), count(DISTINCT <pk>) FROM {{ ref('<parent>') }}" --limit 1
+altimate-dbt execute --query "SELECT * FROM {{ ref('<parent>') }}" --limit 5
+```
+
+### Step 5: Fix and Verify
+
+After applying a fix:
+
+```bash
+altimate-dbt build --model <name> --downstream
+```
+
+Always build with `--downstream` to catch cascading impacts.
+
+**Then verify the fix with data queries** — don't just trust the build:
+```bash
+altimate-dbt execute --query "SELECT count(*) FROM {{ ref('<name>') }}" --limit 1
+altimate-dbt execute --query "SELECT * FROM {{ ref('<name>') }}" --limit 10
+# Check the specific metric/column that was wrong:
+altimate-dbt execute --query "SELECT min(<col>), max(<col>), count(*) - count(<col>) as nulls FROM {{ ref('<name>') }}" --limit 1
+```
+
+## Rationalizations to Resist
+
+| You're Thinking... | Reality |
+|--------------------|---------|
+| "Just make the test pass" | The test is telling you something. Investigate first. |
+| "Let me delete this test" | Ask WHY it exists before removing it. |
+| "It works on my machine" | Check the adapter, Python version, and profile config. |
+| "I'll fix it later" | Later never comes. Fix it now. |
+
+## Common Mistakes
+
+| Mistake | Fix |
+|---------|-----|
+| Changing tests before understanding failures | Read the error. Query the data. Understand the root cause. |
+| Fixing symptoms instead of root cause | Trace the problem upstream. The bug is often 2 models back. |
+| Not checking upstream models | Run `altimate-dbt parents` and build parents individually |
+| Ignoring warnings | Warnings often become errors. Fix them proactively. |
+| Not running offline SQL analysis | Use `altimate_core_semantics` before building to catch join issues |
+| Column names/order don't match schema | Use `altimate_core_column_lineage` to verify output columns match schema.yml |
+| Not querying the actual data when debugging wrong results | Always run data exploration queries — check NULLs, value ranges, distinct values |
+| Trusting build success as proof of correctness | Build only checks syntax and constraints — wrong values pass silently |
+
+## Reference Guides
+
+| Guide | Use When |
+|-------|----------|
+| [references/altimate-dbt-commands.md](references/altimate-dbt-commands.md) | Need the full CLI reference |
+| [references/compilation-errors.md](references/compilation-errors.md) | Jinja, YAML, or parse errors |
+| [references/runtime-errors.md](references/runtime-errors.md) | Database execution errors |
+| [references/test-failures.md](references/test-failures.md) | Understanding and fixing test failures |

package/skills/dbt-troubleshoot/references/altimate-dbt-commands.md

@@ -0,0 +1,66 @@
+# altimate-dbt Command Reference
+
+All dbt operations use the `altimate-dbt` CLI. Output is JSON to stdout; logs go to stderr.
+
+```bash
+altimate-dbt <command> [args...]
+altimate-dbt <command> [args...] --format text    # Human-readable output
+```
+
+## First-Time Setup
+
+```bash
+altimate-dbt init                          # Auto-detect project root
+altimate-dbt init --project-root /path     # Explicit root
+altimate-dbt init --python-path /path      # Override Python
+altimate-dbt doctor                        # Verify setup
+altimate-dbt info                          # Project name, adapter, root
+```
+
+## Build & Run
+
+```bash
+altimate-dbt build --model <name> [--downstream]   # compile + run + test
+altimate-dbt run --model <name> [--downstream]      # materialize only
+altimate-dbt test --model <name>                     # run tests only
+altimate-dbt build-project                           # full project build
+```
+
+## Compile
+
+```bash
+altimate-dbt compile --model <name>
+altimate-dbt compile-query --query "SELECT * FROM {{ ref('stg_orders') }}" [--model <context>]
+```
+
+## Execute SQL
+
+```bash
+altimate-dbt execute --query "SELECT count(*) FROM {{ ref('orders') }}" --limit 100
+```
+
+## Schema & DAG
+
+```bash
+altimate-dbt columns --model <name>                         # column names and types
+altimate-dbt columns-source --source <src> --table <tbl>    # source table columns
+altimate-dbt column-values --model <name> --column <col>    # sample values
+altimate-dbt children --model <name>                        # downstream models
+altimate-dbt parents --model <name>                         # upstream models
+```
+
+## Packages
+
+```bash
+altimate-dbt deps                                           # install packages.yml
+altimate-dbt add-packages --packages dbt-utils,dbt-expectations
+```
+
+## Error Handling
+
+All errors return JSON with `error` and `fix` fields:
+```json
+{ "error": "dbt-core is not installed", "fix": "Install it: python3 -m pip install dbt-core" }
+```
+
+Run `altimate-dbt doctor` as the first diagnostic step for any failure.

package/skills/dbt-troubleshoot/references/compilation-errors.md

@@ -0,0 +1,57 @@
+# Compilation Errors
+
+Compilation errors happen before SQL hits the database. They're Jinja, YAML, or reference problems.
+
+## Diagnosis
+
+```bash
+altimate-dbt compile --model <name>
+```
+
+## Common Compilation Errors
+
+### `Compilation Error: Model 'model.project.name' depends on a node named 'missing_model'`
+
+**Cause**: `{{ ref('missing_model') }}` references a model that doesn't exist.
+
+**Fix**:
+1. Check the spelling: `glob models/**/*missing_model*`
+2. Check if it's in a package: `glob dbt_packages/**/*missing_model*`
+3. If it should be a source: use `{{ source('src', 'table') }}` instead
+
+### `Compilation Error: 'source_name' is undefined`
+
+**Cause**: Source not defined in any `sources.yml`.
+
+**Fix**: Create or update `sources.yml` with the source definition.
+
+### `Parsing Error in YAML`
+
+**Cause**: Invalid YAML syntax (bad indentation, missing colons, unquoted special characters).
+
+**Fix**: Check indentation (must be spaces, not tabs). Ensure strings with special characters are quoted.
+
+### `Compilation Error: Jinja template not found`
+
+**Cause**: Missing macro or wrong macro path.
+
+**Fix**:
+1. Check `macros/` directory
+2. Check `dbt_packages/` for package macros
+3. Verify `packages.yml` is installed: `altimate-dbt deps`
+
+### `dbt_utils is undefined`
+
+**Cause**: Package not installed.
+
+**Fix**:
+```bash
+altimate-dbt deps
+```
+
+## General Approach
+
+1. Read the full error message — it usually tells you exactly which file and line
+2. Open that file and read the surrounding context
+3. Check for typos in `ref()` and `source()` calls
+4. Verify all packages are installed with `altimate-dbt deps`

package/skills/dbt-troubleshoot/references/runtime-errors.md

@@ -0,0 +1,71 @@
+# Runtime / Database Errors
+
+Runtime errors happen when compiled SQL fails to execute against the database.
+
+## Diagnosis
+
+```bash
+# First compile to rule out Jinja issues
+altimate-dbt compile --model <name>
+
+# Then try to build
+altimate-dbt build --model <name>
+
+# Probe the data directly
+altimate-dbt execute --query "<diagnostic_sql>" --limit 10
+```
+
+## Common Runtime Errors
+
+### `Database Error: column "x" does not exist`
+
+**Cause**: Model references a column that doesn't exist in the source/upstream model.
+
+**Fix**:
+```bash
+altimate-dbt columns --model <upstream_model>    # check what columns actually exist
+```
+Update the column name in the SQL.
+
+### `Database Error: relation "schema.table" does not exist`
+
+**Cause**: The upstream model hasn't been built yet, or the schema doesn't exist.
+
+**Fix**:
+```bash
+altimate-dbt build --model <upstream_model>      # build the dependency first
+```
+
+### `Database Error: division by zero`
+
+**Cause**: Dividing by a column that contains zeros.
+
+**Fix**: Add a `NULLIF(denominator, 0)` or `CASE WHEN denominator = 0 THEN NULL ELSE ...` guard.
+
+### `Database Error: ambiguous column reference`
+
+**Cause**: Column name exists in multiple tables in a JOIN.
+
+**Fix**: Qualify with table alias: `orders.customer_id` instead of `customer_id`.
+
+### `Database Error: type mismatch`
+
+**Cause**: Comparing or operating on incompatible types (string vs integer, date vs timestamp).
+
+**Fix**: Add explicit `CAST()` to align types.
+
+### `Timeout` or `Memory Exceeded`
+
+**Cause**: Query is too expensive — full table scan, massive JOIN, or no partition pruning.
+
+**Fix**:
+1. Check if model should be incremental
+2. Add `WHERE` filters to limit data
+3. Check JOIN keys — are they indexed/clustered?
+
+## General Approach
+
+1. Read the compiled SQL: `altimate-dbt compile --model <name>`
+2. Try running a simplified version of the query directly
+3. Check upstream columns: `altimate-dbt columns --model <upstream>`
+4. Add diagnostic queries to understand the data shape

package/skills/dbt-troubleshoot/references/test-failures.md

@@ -0,0 +1,95 @@
+# Test Failures
+
+Test failures mean the data violates an expected constraint. The test is usually right — investigate before changing it.
+
+## Diagnosis
+
+```bash
+altimate-dbt test --model <name>
+```
+
+## Common Test Failures
+
+### `unique` test fails
+
+**Meaning**: Duplicate values exist in the column.
+
+**Investigate**:
+```bash
+altimate-dbt execute --query "
+  SELECT <column>, count(*) as cnt
+  FROM {{ ref('<model>') }}
+  GROUP BY 1
+  HAVING count(*) > 1
+  ORDER BY cnt DESC
+" --limit 10
+```
+
+**Common causes**:
+- Missing deduplication in staging model
+- Incorrect JOIN producing row multiplication (LEFT JOIN with 1:many relationship)
+- Incorrect `GROUP BY` (missing a dimension)
+
+### `not_null` test fails
+
+**Meaning**: NULL values exist where they shouldn't.
+
+**Investigate**:
+```bash
+altimate-dbt execute --query "
+  SELECT * FROM {{ ref('<model>') }}
+  WHERE <column> IS NULL
+" --limit 5
+```
+
+**Common causes**:
+- LEFT JOIN where INNER JOIN was intended (unmatched rows become NULL)
+- Source data has genuine NULLs — may need `COALESCE()` or filter
+- Wrong column referenced in the model SQL
+
+### `accepted_values` test fails
+
+**Meaning**: Values exist that weren't in the expected list.
+
+**Investigate**:
+```bash
+altimate-dbt column-values --model <name> --column <column>
+```
+
+**Common causes**:
+- New value appeared in source data (update the accepted list)
+- Data quality issue upstream (fix the source or add a filter)
+- Test list is incomplete (add the missing values)
+
+### `relationships` test fails
+
+**Meaning**: Foreign key references a value that doesn't exist in the parent table.
+
+**Investigate**:
+```bash
+altimate-dbt execute --query "
+  SELECT child.<fk_col>, count(*)
+  FROM {{ ref('<child>') }} child
+  LEFT JOIN {{ ref('<parent>') }} parent ON child.<fk_col> = parent.<pk_col>
+  WHERE parent.<pk_col> IS NULL
+  GROUP BY 1
+" --limit 10
+```
+
+**Common causes**:
+- Parent table hasn't been rebuilt with latest data
+- Orphan records in source data
+- Type mismatch between FK and PK (e.g., string vs integer)
+
+## The Decision Framework
+
+When a test fails:
+
+1. **Understand**: Query the failing rows. Why do they exist?
+2. **Classify**: Is it a data issue, a model logic bug, or a test definition problem?
+3. **Fix the right thing**:
+   - Data issue → fix upstream or add a filter/coalesce
+   - Logic bug → fix the model SQL
+   - Test is wrong → update the test (with explicit justification to the user)
+
+**Never silently weaken a test.** If you need to change a test, explain why to the user.

package/skills/lineage-diff/SKILL.md

@@ -0,0 +1,64 @@
+---
+name: lineage-diff
+description: Compare column-level lineage between two versions of a SQL query to show added, removed, and changed data flow edges.
+---
+
+# Lineage Diff
+
+## Requirements
+**Agent:** any (read-only analysis)
+**Tools used:** lineage_check, read, bash (for git operations), glob
+
+Compare column-level lineage between two versions of a SQL model to identify changes in data flow.
+
+## Workflow
+
+1. **Get the original SQL** — Either:
+   - Read the file from disk (current committed version)
+   - Use `git show HEAD:path/to/file.sql` via `bash` to get the last committed version
+   - Accept the "before" SQL directly from the user
+
+2. **Get the modified SQL** — Either:
+   - Read the current (modified) file from disk
+   - Accept the "after" SQL directly from the user
+
+3. **Run lineage on both versions**:
+   - Call `lineage_check` with the original SQL
+   - Call `lineage_check` with the modified SQL
+
+4. **Compute the diff**:
+   - **Added edges**: Edges in the new lineage that don't exist in the old
+   - **Removed edges**: Edges in the old lineage that don't exist in the new
+   - **Unchanged edges**: Edges present in both
+
+5. **Report the diff** in a clear format:
+
+```
+Lineage Diff: model_name
+═══════════════════════════
+
++ ADDED (new data flow):
+  + source_table.new_column → target_table.output_column
+
+- REMOVED (broken data flow):
+  - source_table.old_column → target_table.output_column
+
+  UNCHANGED: 5 edges
+
+Impact: 1 new edge, 1 removed edge
+```
+
+## Usage
+
+The user invokes this skill with a file path:
+- `/lineage-diff models/marts/dim_customers.sql` — Compare current file against last git commit
+- `/lineage-diff` — Compare staged changes in the current file
+
+## Edge Matching
+
+Two edges are considered the same if all four fields match:
+- `source_table` + `source_column` + `target_table` + `target_column`
+
+The `transform` field is informational and not used for matching.
+
+Use the tools: `lineage_check`, `read`, `bash` (for git operations), `glob`.