altimate-code 0.4.9 → 0.5.1

package/skills/dbt-develop/references/yaml-generation.md

@@ -0,0 +1,90 @@
+# YAML Config Generation
+
+## sources.yml — Define Raw Data Sources
+
+Generate from warehouse metadata using:
+```bash
+altimate-dbt columns-source --source <source_name> --table <table_name>
+```
+
+Template:
+```yaml
+version: 2
+
+sources:
+  - name: raw_stripe
+    description: Raw Stripe payment data
+    database: raw
+    schema: stripe
+    tables:
+      - name: payments
+        description: All payment transactions
+        columns:
+          - name: payment_id
+            description: Primary key
+            tests:
+              - unique
+              - not_null
+          - name: amount
+            description: Payment amount in cents
+          - name: created_at
+            description: Payment creation timestamp
+            tests:
+              - not_null
+```
+
+## schema.yml — Model Documentation and Tests
+
+Generate after building a model using:
+```bash
+altimate-dbt columns --model <model_name>
+```
+
+Template:
+```yaml
+version: 2
+
+models:
+  - name: stg_stripe__payments
+    description: Staged Stripe payments with renamed columns and type casts
+    columns:
+      - name: payment_id
+        description: Primary key from source
+        tests:
+          - unique
+          - not_null
+      - name: amount_dollars
+        description: Payment amount converted to dollars
+```
+
+## Column Pattern Heuristics
+
+When auto-generating YAML, infer descriptions and tests from column names:
+
+| Pattern | Description Template | Auto-Tests |
+|---------|---------------------|------------|
+| `*_id` | "Foreign key to {table}" or "Primary key" | `unique`, `not_null` |
+| `*_at`, `*_date`, `*_timestamp` | "Timestamp of {event}" | `not_null` |
+| `*_amount`, `*_price`, `*_cost` | "Monetary value" | `not_null` |
+| `is_*`, `has_*` | "Boolean flag for {condition}" | `accepted_values: [true, false]` |
+| `*_type`, `*_status`, `*_category` | "Categorical" | `accepted_values` (if inferable) |
+| `*_count`, `*_total`, `*_sum` | "Aggregated count/total" | — |
+| `*_name`, `*_title`, `*_label` | "Human-readable name" | — |
+
+## File Naming Conventions
+
+| Convention | Example |
+|-----------|---------|
+| Sources | `_<source>__sources.yml` |
+| Model schema | `_<model>__models.yml` or `schema.yml` |
+| Properties | `_<model>__models.yml` |
+
+Match whatever convention the project already uses. Check existing `.yml` files in the same directory.
+
+## Merging with Existing YAML
+
+When YAML files already exist:
+1. Read the existing file first
+2. Add new entries without duplicating existing ones
+3. Preserve human-written descriptions
+4. Use `edit` tool (not `write`) to merge

package/skills/dbt-docs/SKILL.md

@@ -0,0 +1,99 @@
+---
+name: dbt-docs
+description: Document dbt models and columns in schema.yml with business context — model descriptions, column definitions, and doc blocks. Use when adding or improving documentation for discoverability. Powered by altimate-dbt.
+---
+
+# dbt Documentation
+
+## Requirements
+**Agent:** builder or migrator (requires file write access)
+**Tools used:** bash (runs `altimate-dbt` commands), read, glob, write, edit
+
+## When to Use This Skill
+
+**Use when the user wants to:**
+- Add or improve model descriptions in schema.yml
+- Write column-level descriptions with business context
+- Create shared doc blocks for reusable definitions
+- Improve dbt docs site content
+
+**Do NOT use for:**
+- Adding tests → use `dbt-test`
+- Creating new models → use `dbt-develop`
+- Generating sources.yml from scratch → use `dbt-develop`
+
+## Workflow
+
+### 1. Understand the Model
+
+```bash
+altimate-dbt columns --model <name>                    # what columns exist
+altimate-dbt parents --model <name>                    # what feeds this model
+altimate-dbt children --model <name>                   # who consumes it
+altimate-dbt compile --model <name>                    # see the rendered SQL
+```
+
+Read the model SQL to understand the transformations:
+```bash
+glob models/**/<name>.sql
+read <model_file>
+```
+
+### 2. Read Existing Documentation
+
+Check what's already documented:
+```bash
+glob models/**/*schema*.yml models/**/*_models.yml
+read <yaml_file>
+```
+
+### 3. Write Documentation
+
+See [references/documentation-standards.md](references/documentation-standards.md) for quality guidelines.
+
+#### Model-Level Description
+Cover: **What** (business entity), **Why** (use case), **How** (key transforms), **When** (materialization).
+
+```yaml
+- name: fct_daily_revenue
+  description: >
+    Daily revenue aggregation by product category. Joins staged orders with
+    product dimensions and calculates gross/net revenue. Materialized as
+    incremental with unique key on (date_day, category_id). Used by the
+    finance team for daily P&L reporting.
+```
+
+#### Column-Level Description
+Describe business meaning, derivation formula, and caveats:
+
+```yaml
+columns:
+  - name: net_revenue
+    description: >
+      Total revenue minus refunds and discounts for the day.
+      Formula: gross_revenue - refund_amount - discount_amount.
+      Can be negative if refunds exceed sales.
+```
+
+### 4. Validate
+
+```bash
+altimate-dbt compile --model <name>    # ensure YAML is valid
+```
+
+## Common Mistakes
+
+| Mistake | Fix |
+|---------|-----|
+| Restating the column name as the description | `"order_id: The order ID"` → describe business meaning |
+| Empty descriptions | Every column should have a description. If unsure, describe the source. |
+| Not reading the SQL before documenting | Read the model to understand derivation logic |
+| Duplicating descriptions across models | Use doc blocks for shared definitions |
+| Writing implementation details instead of business context | Describe what it means to the business, not how it's computed |
+
+## Reference Guides
+
+| Guide | Use When |
+|-------|----------|
+| [references/altimate-dbt-commands.md](references/altimate-dbt-commands.md) | Need the full CLI reference |
+| [references/documentation-standards.md](references/documentation-standards.md) | Writing high-quality descriptions |

package/skills/dbt-docs/references/altimate-dbt-commands.md

@@ -0,0 +1,66 @@
+# altimate-dbt Command Reference
+
+All dbt operations use the `altimate-dbt` CLI. Output is JSON to stdout; logs go to stderr.
+
+```bash
+altimate-dbt <command> [args...]
+altimate-dbt <command> [args...] --format text    # Human-readable output
+```
+
+## First-Time Setup
+
+```bash
+altimate-dbt init                          # Auto-detect project root
+altimate-dbt init --project-root /path     # Explicit root
+altimate-dbt init --python-path /path      # Override Python
+altimate-dbt doctor                        # Verify setup
+altimate-dbt info                          # Project name, adapter, root
+```
+
+## Build & Run
+
+```bash
+altimate-dbt build --model <name> [--downstream]   # compile + run + test
+altimate-dbt run --model <name> [--downstream]      # materialize only
+altimate-dbt test --model <name>                     # run tests only
+altimate-dbt build-project                           # full project build
+```
+
+## Compile
+
+```bash
+altimate-dbt compile --model <name>
+altimate-dbt compile-query --query "SELECT * FROM {{ ref('stg_orders') }}" [--model <context>]
+```
+
+## Execute SQL
+
+```bash
+altimate-dbt execute --query "SELECT count(*) FROM {{ ref('orders') }}" --limit 100
+```
+
+## Schema & DAG
+
+```bash
+altimate-dbt columns --model <name>                         # column names and types
+altimate-dbt columns-source --source <src> --table <tbl>    # source table columns
+altimate-dbt column-values --model <name> --column <col>    # sample values
+altimate-dbt children --model <name>                        # downstream models
+altimate-dbt parents --model <name>                         # upstream models
+```
+
+## Packages
+
+```bash
+altimate-dbt deps                                           # install packages.yml
+altimate-dbt add-packages --packages dbt-utils,dbt-expectations
+```
+
+## Error Handling
+
+All errors return JSON with `error` and `fix` fields:
+```json
+{ "error": "dbt-core is not installed", "fix": "Install it: python3 -m pip install dbt-core" }
+```
+
+Run `altimate-dbt doctor` as the first diagnostic step for any failure.

package/skills/dbt-docs/references/documentation-standards.md

@@ -0,0 +1,94 @@
+# Documentation Standards
+
+## Model Descriptions
+
+A good model description answers four questions:
+
+1. **What**: What business entity or metric does this represent?
+2. **Why**: Who uses it and for what purpose?
+3. **How**: Key transformations (joins, filters, aggregations)
+4. **Grain**: What does one row represent?
+
+### Good Example
+```yaml
+description: >
+  Daily revenue by product category. One row per category per day.
+  Joins staged orders with product dimensions, calculates gross and net revenue.
+  Used by the finance team for P&L reporting and the marketing team for
+  campaign attribution.
+```
+
+### Bad Example
+```yaml
+description: "This model contains daily revenue data."
+```
+
+## Column Descriptions
+
+### Primary Keys
+State that it's a primary key and where it originates:
+```yaml
+- name: order_id
+  description: "Primary key. Unique identifier for each order, sourced from the orders table in Stripe."
+```
+
+### Foreign Keys
+State what it references:
+```yaml
+- name: customer_id
+  description: "Foreign key to dim_customers. The customer who placed this order."
+```
+
+### Calculated Columns
+Include the formula:
+```yaml
+- name: net_revenue
+  description: "Gross revenue minus refunds and discounts. Formula: gross_amount - refund_amount - discount_amount. Can be negative."
+```
+
+### Status/Category Columns
+List the possible values:
+```yaml
+- name: order_status
+  description: "Current order status. Values: pending, shipped, delivered, cancelled, refunded."
+```
+
+### Timestamps
+State what event they represent:
+```yaml
+- name: shipped_at
+  description: "Timestamp when the order was shipped. NULL if not yet shipped. UTC timezone."
+```
+
+## Doc Blocks (Shared Definitions)
+
+For definitions reused across multiple models, create doc blocks:
+
+```markdown
+-- docs/shared_definitions.md
+{% docs customer_id %}
+Unique identifier for a customer. Sourced from the customers table
+in the CRM system. Used as the primary join key across all
+customer-related models.
+{% enddocs %}
+```
+
+Reference in YAML:
+```yaml
+- name: customer_id
+  description: "{{ doc('customer_id') }}"
+```
+
+## Anti-Patterns
+
+| Anti-Pattern | Why It's Bad | Better Approach |
+|-------------|-------------|-----------------|
+| `"The order ID"` | Just restates the column name | Describe origin and business meaning |
+| `"See code"` | Defeats the purpose of docs | Summarize the logic |
+| `""` (empty) | Missing documentation | Write something, even if brief |
+| Technical jargon only | Business users can't understand | Lead with business meaning, add technical detail after |
+
+## Official Documentation
+
+- **dbt documentation**: https://docs.getdbt.com/docs/build/documentation
+- **Doc blocks**: https://docs.getdbt.com/docs/build/documentation#using-docs-blocks

package/skills/dbt-test/SKILL.md

@@ -0,0 +1,121 @@
+---
+name: dbt-test
+description: Add schema tests, unit tests, and data quality checks to dbt models. Use when validating data integrity, adding test definitions to schema.yml, writing unit tests, or practicing test-driven development in dbt. Powered by altimate-dbt.
+---
+
+# dbt Testing
+
+## Requirements
+**Agent:** builder or migrator (requires file write access)
+**Tools used:** bash (runs `altimate-dbt` commands), read, glob, write, edit, altimate_core_testgen, altimate_core_validate
+
+## When to Use This Skill
+
+**Use when the user wants to:**
+- Add tests to a model's schema.yml (unique, not_null, relationships, accepted_values)
+- Write dbt unit tests (mock inputs → expected outputs)
+- Create custom generic or singular tests
+- Debug why a test is failing
+- Practice test-driven development in dbt
+
+**Do NOT use for:**
+- Creating or modifying model SQL → use `dbt-develop`
+- Writing model descriptions → use `dbt-docs`
+- Debugging build/compilation errors → use `dbt-troubleshoot`
+
+## The Iron Rule
+
+**Never modify a test to make it pass without understanding why it's failing.**
+
+A failing test is information. It means either:
+1. The data has a real quality issue (fix the data or the model)
+2. The test expectation is wrong (update the test with justification)
+3. The model logic is wrong (fix the model)
+
+Option 2 requires explicit user confirmation. Do not silently weaken tests.
+
+## Schema Test Workflow
+
+### 1. Discover Columns
+
+```bash
+altimate-dbt columns --model <name>
+altimate-dbt column-values --model <name> --column <col>
+```
+
+### 2. Read Existing Tests
+
+```bash
+glob models/**/*schema*.yml models/**/*_models.yml
+read <yaml_file>
+```
+
+### 3. Generate Tests
+
+**Auto-generate with `altimate_core_testgen`:** Pass the compiled SQL and schema to generate boundary-value, NULL-handling, and edge-case test assertions automatically. This produces executable test SQL covering cases you might miss manually.
+
+```
+altimate_core_testgen(sql: <compiled_sql>, schema_context: <schema_object>)
+```
+
+Review the generated tests — keep what makes sense, discard trivial ones. Then apply test rules based on column patterns — see [references/schema-test-patterns.md](references/schema-test-patterns.md).
+
+### 4. Write YAML
+
+Merge into existing schema.yml (don't duplicate). Use `edit` for existing files, `write` for new ones.
+
+### 5. Validate SQL
+
+Before running, validate the compiled model SQL to catch syntax and schema errors early:
+
+```
+altimate_core_validate(sql: <compiled_sql>, schema_context: <schema_object>)
+```
+
+### 6. Run Tests
+
+```bash
+altimate-dbt test --model <name>          # run tests for this model
+altimate-dbt build --model <name>         # build + test together
+```
+
+## Unit Test Workflow
+
+See [references/unit-test-guide.md](references/unit-test-guide.md) for the full unit test framework.
+
+### Quick Pattern
+
+```yaml
+unit_tests:
+  - name: test_order_total_calculation
+    model: fct_orders
+    given:
+      - input: ref('stg_orders')
+        rows:
+          - { order_id: 1, quantity: 3, unit_price: 10.00 }
+          - { order_id: 2, quantity: 1, unit_price: 25.00 }
+    expect:
+      rows:
+        - { order_id: 1, order_total: 30.00 }
+        - { order_id: 2, order_total: 25.00 }
+```
+
+## Common Mistakes
+
+| Mistake | Fix |
+|---------|-----|
+| Testing every column with `not_null` | Only test columns that should never be null. Think about what NULL means. |
+| Missing `unique` test on primary keys | Every primary key needs `unique` + `not_null` |
+| `accepted_values` with incomplete list | Use `altimate-dbt column-values` to discover real values first |
+| Modifying a test to make it pass | Understand WHY it fails first. The test might be right. |
+| No `relationships` test on foreign keys | Add `relationships: {to: ref('parent'), field: parent_id}` |
+| Unit testing trivial logic | Don't unit test `SELECT a, b FROM source`. Test calculations and business logic. |
+
+## Reference Guides
+
+| Guide | Use When |
+|-------|----------|
+| [references/altimate-dbt-commands.md](references/altimate-dbt-commands.md) | Need the full CLI reference |
+| [references/schema-test-patterns.md](references/schema-test-patterns.md) | Adding schema.yml tests by column pattern |
+| [references/unit-test-guide.md](references/unit-test-guide.md) | Writing dbt unit tests |
+| [references/custom-tests.md](references/custom-tests.md) | Creating generic or singular tests |

package/skills/dbt-test/references/altimate-dbt-commands.md

@@ -0,0 +1,66 @@
+# altimate-dbt Command Reference
+
+All dbt operations use the `altimate-dbt` CLI. Output is JSON to stdout; logs go to stderr.
+
+```bash
+altimate-dbt <command> [args...]
+altimate-dbt <command> [args...] --format text    # Human-readable output
+```
+
+## First-Time Setup
+
+```bash
+altimate-dbt init                          # Auto-detect project root
+altimate-dbt init --project-root /path     # Explicit root
+altimate-dbt init --python-path /path      # Override Python
+altimate-dbt doctor                        # Verify setup
+altimate-dbt info                          # Project name, adapter, root
+```
+
+## Build & Run
+
+```bash
+altimate-dbt build --model <name> [--downstream]   # compile + run + test
+altimate-dbt run --model <name> [--downstream]      # materialize only
+altimate-dbt test --model <name>                     # run tests only
+altimate-dbt build-project                           # full project build
+```
+
+## Compile
+
+```bash
+altimate-dbt compile --model <name>
+altimate-dbt compile-query --query "SELECT * FROM {{ ref('stg_orders') }}" [--model <context>]
+```
+
+## Execute SQL
+
+```bash
+altimate-dbt execute --query "SELECT count(*) FROM {{ ref('orders') }}" --limit 100
+```
+
+## Schema & DAG
+
+```bash
+altimate-dbt columns --model <name>                         # column names and types
+altimate-dbt columns-source --source <src> --table <tbl>    # source table columns
+altimate-dbt column-values --model <name> --column <col>    # sample values
+altimate-dbt children --model <name>                        # downstream models
+altimate-dbt parents --model <name>                         # upstream models
+```
+
+## Packages
+
+```bash
+altimate-dbt deps                                           # install packages.yml
+altimate-dbt add-packages --packages dbt-utils,dbt-expectations
+```
+
+## Error Handling
+
+All errors return JSON with `error` and `fix` fields:
+```json
+{ "error": "dbt-core is not installed", "fix": "Install it: python3 -m pip install dbt-core" }
+```
+
+Run `altimate-dbt doctor` as the first diagnostic step for any failure.

package/skills/dbt-test/references/custom-tests.md

@@ -0,0 +1,59 @@
+# Custom dbt Tests
+
+## Generic Tests (Reusable)
+
+Generic tests are macros in `tests/generic/` or `macros/` that accept parameters:
+
+```sql
+-- tests/generic/test_positive_values.sql
+{% test positive_values(model, column_name) %}
+
+select *
+from {{ model }}
+where {{ column_name }} < 0
+
+{% endtest %}
+```
+
+Usage in schema.yml:
+```yaml
+columns:
+  - name: amount
+    tests:
+      - positive_values
+```
+
+## Singular Tests (One-Off)
+
+Singular tests are standalone SQL files in `tests/` that return failing rows:
+
+```sql
+-- tests/assert_no_orphan_orders.sql
+-- Orders must have a matching customer
+select o.order_id
+from {{ ref('fct_orders') }} o
+left join {{ ref('dim_customers') }} c on o.customer_id = c.customer_id
+where c.customer_id is null
+```
+
+A passing test returns zero rows.
+
+## When to Use Which
+
+| Type | Use When |
+|------|----------|
+| Schema tests (built-in) | `unique`, `not_null`, `accepted_values`, `relationships` |
+| dbt-utils tests | `expression_is_true`, `unique_combination_of_columns` |
+| Generic tests (custom) | Reusable validation logic across multiple models |
+| Singular tests | One-off business rules, cross-model assertions |
+| Unit tests | Testing calculation logic with mocked inputs |
+
+## Naming Conventions
+
+- Generic tests: `test_<what_it_checks>.sql` in `tests/generic/`
+- Singular tests: `assert_<business_rule>.sql` in `tests/`
+
+## Official Documentation
+
+- **Data tests**: https://docs.getdbt.com/docs/build/data-tests
+- **Custom generic tests**: https://docs.getdbt.com/docs/build/data-tests#generic-data-tests

package/skills/dbt-test/references/schema-test-patterns.md

@@ -0,0 +1,103 @@
+# Schema Test Patterns
+
+## Test Generation Rules
+
+Apply tests based on column name patterns:
+
+| Column Pattern | Tests |
+|---|---|
+| `*_id` (primary key) | `unique`, `not_null` |
+| `*_id` (foreign key) | `not_null`, `relationships: {to: ref('parent'), field: id}` |
+| `status`, `type`, `category` | `accepted_values` (discover values with `altimate-dbt column-values`) |
+| `*_at`, `*_date`, `*_timestamp` | `not_null` (if event timestamp that should always exist) |
+| `is_*`, `has_*` (booleans) | `accepted_values: [true, false]` |
+| Columns in JOIN conditions | `not_null` (nulls cause dropped rows in INNER JOIN) |
+
+## Test Priority Framework
+
+Not every column needs every test. Prioritize:
+
+### Tier 1: Always Test (Primary Keys)
+```yaml
+- name: order_id
+  tests:
+    - unique
+    - not_null
+```
+
+### Tier 2: Test Foreign Keys
+```yaml
+- name: customer_id
+  tests:
+    - not_null
+    - relationships:
+        to: ref('dim_customers')
+        field: customer_id
+```
+
+### Tier 3: Test Business-Critical Columns
+```yaml
+- name: order_status
+  tests:
+    - not_null
+    - accepted_values:
+        values: ['pending', 'shipped', 'delivered', 'cancelled']
+```
+
+### Tier 4: Test Derived Columns (When Logic is Complex)
+```yaml
+- name: net_revenue
+  tests:
+    - not_null
+    - dbt_utils.expression_is_true:
+        expression: ">= 0"
+```
+
+## Discovering Values for accepted_values
+
+Before adding `accepted_values`, discover what actually exists:
+
+```bash
+altimate-dbt column-values --model <name> --column status
+```
+
+This prevents false test failures from values you didn't know about.
+
+## Cost-Conscious Testing
+
+For large tables, add `where` clauses to expensive tests:
+
+```yaml
+- name: order_id
+  tests:
+    - unique:
+        config:
+          where: "order_date >= current_date - interval '30 days'"
+```
+
+## dbt-utils Test Extensions
+
+Common additional tests from `dbt-utils`:
+
+```yaml
+# Ensure expression evaluates to true for all rows
+- dbt_utils.expression_is_true:
+    expression: "amount >= 0"
+
+# Ensure no duplicate combinations
+- dbt_utils.unique_combination_of_columns:
+    combination_of_columns:
+      - date_day
+      - customer_id
+
+# Ensure referential integrity across multiple columns
+- dbt_utils.relationships_where:
+    to: ref('dim_products')
+    field: product_id
+    from_condition: "product_id is not null"
+```
+
+## Official Documentation
+
+- **dbt tests**: https://docs.getdbt.com/docs/build/data-tests
+- **dbt-utils tests**: https://github.com/dbt-labs/dbt-utils#generic-tests