@altimateai/altimate-code 0.5.20 → 0.6.0

package/CHANGELOG.md

@@ -5,6 +5,63 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.6.0] - 2026-04-21
+
+### Added
+
+- **Data parity (`data_diff` tool + skill)** — compare tables or SQL-query results row-by-row across Postgres, Snowflake, BigQuery, Databricks, ClickHouse, MySQL, Redshift, SQL Server, Microsoft Fabric, DuckDB, SQLite, and Oracle. Five algorithms: `auto` (JoinDiff same-dialect, HashDiff cross-dialect), `joindiff` (FULL OUTER JOIN), `hashdiff` (bisecting checksums — works at any scale without pulling data out), `profile` (column-level statistics, no row values leave the database), and `cascade` (profile first, then HashDiff on diverging columns). Partitioning supports date (`day`/`week`/`month`/`year`), numeric (`bucket_size`), and categorical (distinct values) modes so 100M+ row tables diff in independent batches. Auto-discovers comparable columns from `information_schema`, excludes audit/timestamp columns by name pattern AND by catalog default (`NOW()`, `CURRENT_TIMESTAMP`, `GETDATE()`, `SYSDATE`, `SYSTIMESTAMP`), and confirms exclusions with the user before diffing. (#493)
+- **MSSQL and Microsoft Fabric support in data-parity** — dialect-aware date truncation (`DATETRUNC`), locale-safe date literals (`CONVERT(DATE, ..., 23)`), and seven Azure AD / Entra ID authentication flows (password, access-token, service-principal-secret, MSI-VM, MSI-app-service, default credential chain, token-credential) delegated to `tedious`. Upgrades `mssql` v11 → v12 with explicit `ConnectionPool` isolation and correct handling of unnamed-column result sets. (#705)
+- **Databricks AI Gateway provider** — connect to Databricks serving endpoints (Foundation Model APIs) via PAT auth (`workspace-host::token`), with fallback to `DATABRICKS_HOST` / `DATABRICKS_TOKEN` environment variables. Registers 11 foundation models — Meta Llama 3.1 (405B / 70B / 8B), Claude Sonnet / Opus 4.6, GPT-5.4 / GPT-5 Mini, Gemini 3.1 Pro, DBRX Instruct, and Mixtral 8x7B. Host regex restricts credentials to `*.cloud.databricks.com`, `*.azuredatabricks.net`, and `*.gcp.databricks.com`. (#649, closes #602)
+- **Amazon Bedrock custom-endpoints guide** — dedicated docs page covering bearer-token auth, AWS credential chain, `baseURL` configuration, cross-region model-ID prefixing, and troubleshooting. Provider key corrected from `bedrock` to `amazon-bedrock` across quickstart, providers, and models pages. (#706)
+- **User-facing docs for the new features** — Databricks AI Gateway section in `configure/providers.md` and a full `data-engineering/guides/data-parity.md` covering supported warehouse pairs, algorithms, partition modes, Azure AD auth matrix for Fabric, and compliance guidance.
+
+### Changed
+
+- **`@altimateai/altimate-core` 0.2.6 → 0.3.0** — enables the cross-dialect data-parity engine, T-SQL dialect support for MSSQL/Fabric, and refined hashdiff bisection. Rebuilt native binaries published for all five supported platforms. (#717, closes #716)
+- **Altimate connect dialog polish** — `/connect` now accepts `instance-name::api-key` directly (default URL `https://api.myaltimate.com`). The three-part `api-url::instance-name::api-key` form still works for custom and self-hosted instances. Provider display name "Altimate" → "Altimate AI"; default model display "Altimate AI" → "Altimate LLM Gateway". The internal provider ID (`altimate-backend`) and model ID (`altimate-default`) are preserved — existing `model.json` favorites, recents, and pinned `model:` entries in `opencode.json` continue to work without migration. (#724)
+
+### Fixed
+
+- **Text contrast on light terminal backgrounds** — dark foreground (`#1a1a1a` or `palette[0]` when available) replaces the near-invisible `palette[7]` on light-mode system themes, and inline code blocks now render on an opaque background instead of transparent. Eager `COLORFGBG` env-var detection narrowed to `bg === 7 || bg === 15` skips the 1-second OSC 11 query altogether on light terminals that don't support it (urxvt, gnome-terminal). (#712, closes #704)
+- **Historical `tool_use` blocks after agent switches or MCP disconnects** — the LiteLLM-only `_noop` workaround is replaced by a general fix: tool names are extracted from both `tool-call` and `tool-result` blocks in message history (using `Object.hasOwn()` for prototype-pollution safety), validated against `/^[a-zA-Z0-9_-]{1,64}$/`, and registered as stubs that return "tool no longer available" if the model attempts to call them. Eliminates the Anthropic API 400 error "Requests with 'tool_use' and 'tool_result' blocks must include tool definition." (#703, closes [AI-678])
+- **`/docs` command and config links point to `docs.altimate.sh`** — the TUI "Open docs" action, Cloudflare AI Gateway help text, Anthropic system prompt, and `Config.command` / `Config.agent` schema descriptions previously linked to the wrong domain. Also updates paths to the `/configure/` prefix that matches the actual mkdocs site. (#715, closes #714)
+
+### Security
+
+- **Databricks host validation hardened** — new `isValidDatabricksHost` helper rejects CRLF/whitespace (JS regex `$` matches before `\n` by default), and the env-fallback path now validates host before constructing the `baseURL`.
+- **Tool-name validation** — stub registration ignores names with shell metacharacters, ANSI escapes, control characters, or lengths > 64, guarding against tampered session-file replays.
+- **Restricted `az` inherited env** — `az account get-access-token` is invoked with a whitelisted environment (`PATH`, `HOME`, `AZURE_*`, locale) so unrelated secrets (`DATABRICKS_TOKEN`, cloud provider keys) are not inherited by `az` or any `az` extension.
+
+### Testing
+
+- 38 new adversarial tests covering Databricks host validation (CRLF, anchoring, attacker suffixes), PAT parsing edge cases, body transform, tool-name tainted-input guards, and `data_diff` tool-description release contract.
+- 139-test consolidation across dbt helpers, file status, project-scan, session/llm, and MCP discovery — symlink cache round-trips, seed/test node exclusion, JSON array edge cases, sessionId sanitization, pagination boundary math, and connection-string masking. Two broken `${VAR}`-in-MCP-`command` tests removed (resolution was always restricted to `env` and `headers`). (#709)
+
+### Compliance note
+
+`data_diff` includes up to 5 sample diff rows in its tool output, which becomes part of the LLM conversation. For PII / PHI / PCI data, use `algorithm: "profile"` — column statistics compare without sending row values. The `data-parity` skill asks for explicit confirmation before running row-level diffs against tables whose names match common regulated patterns (`customers`, `patients`, `orders`, `payments`, `accounts`, `users`). A hard env-var opt-out for sample values is tracked in [#729](https://github.com/AltimateAI/altimate-code/issues/729).
+
+### Breaking
+
+- **`mssql` upgraded to v12**. Users with `mssql@^11` pinned at the application level will see "mssql.ConnectionPool is not available" on first SQL Server connection. Pin to `^12` or let altimate resolve.
+- **SQL Server result sets now expose `_`-prefixed columns**. The internal `startsWith("_")` column filter (introduced for partition-discovery noise suppression) was removed because it also stripped legitimate aliases like `_p` used by the partition engine. Queries that relied on this implicit filtering will see the extra columns in results.
+
+## [0.5.21] - 2026-04-13
+
+### Added
+
+- **Automated dbt unit test generation** — generate dbt unit tests (v1.8+) from your terminal with `/dbt-unit-tests` or the `dbt_unit_test_gen` tool. Detects testable SQL constructs (CASE/WHEN, JOINs, NULLs, window functions, division, incremental models) and assembles complete YAML with type-correct mock data across 7 dialects. Includes `input: this` mocks for incremental models, `format: sql` for ephemeral deps, and handles seeds/snapshots as first-class `ref()` deps. Five-phase skill workflow: Analyze → Generate → Refine → Validate → Write. Requires dbt-core 1.8+. (#673)
+- **Manifest parse cache** — `loadRawManifest()` caches by path+mtime so large manifests (100MB+) are parsed once per session, not once per tool call.
+- **Model/source descriptions in manifest** — `DbtModelInfo` and `DbtSourceInfo` now surface descriptions from `schema.yml`, giving downstream tools richer semantic context.
+- **`adapter_type` on `DbtManifestResult`** — exposes the dbt adapter type (snowflake, bigquery, etc.) from manifest metadata for dialect auto-detection.
+
+### Fixed
+
+- **MCP env-var `$${VAR}` escape and chain-injection vulnerability** — the two-layer env-var resolution design allowed `$${VAR}` escapes to be re-resolved (breaking literal `${VAR}` passthrough) and enabled variable-chain injection where `EVIL_VAR="${SECRET}"` could exfiltrate secrets the config never referenced. Collapsed to a single resolution pass scoped to `env` and `headers` fields only. (#697, relates to #656)
+- **MCP server environment variables passed as literals** — `${VAR}`, `${VAR:-default}`, and `{env:VAR}` patterns in MCP server `env` blocks were passed as literal strings to child processes, causing auth failures for tools like `gitlab-mcp-server`. (#666, closes #656)
+- **`sql_explain` and `altimate_core_validate` input hardening** — reject empty/placeholder SQL and warehouse names before hitting the warehouse. `sql_explain` now generates dialect-aware EXPLAIN statements for 12+ warehouse types. Driver errors are translated into actionable guidance (e.g., "No warehouses configured — run `warehouse_add`"). `altimate_core_validate` now runs even without a schema (previously hard-failed), with a `(no schema)` indicator and clear instructions for providing schema context. (#693, closes #691)
+- **`sql_explain` alternatives for unsupported warehouses** — BigQuery, Oracle, and SQL Server now return specific guidance (dry-run API, `DBMS_XPLAN`, `SET SHOWPLAN_TEXT ON`) instead of a generic "not supported" message.
+
 ## [0.5.20] - 2026-04-09
 
 ### Added

package/package.json

@@ -7,24 +7,24 @@
   "scripts": {
     "postinstall": "bun ./postinstall.mjs || node ./postinstall.mjs"
   },
-  "version": "0.5.20",
+  "version": "0.6.0",
   "license": "MIT",
   "dependencies": {
-    "@altimateai/altimate-core": "0.2.6"
+    "@altimateai/altimate-core": "0.3.0"
   },
   "optionalDependencies": {
-    "@altimateai/altimate-code-linux-arm64": "0.5.20",
-    "@altimateai/altimate-code-windows-x64": "0.5.20",
-    "@altimateai/altimate-code-windows-x64-baseline": "0.5.20",
-    "@altimateai/altimate-code-darwin-x64": "0.5.20",
-    "@altimateai/altimate-code-windows-arm64": "0.5.20",
-    "@altimateai/altimate-code-linux-x64-baseline": "0.5.20",
-    "@altimateai/altimate-code-linux-arm64-musl": "0.5.20",
-    "@altimateai/altimate-code-linux-x64-musl": "0.5.20",
-    "@altimateai/altimate-code-linux-x64": "0.5.20",
-    "@altimateai/altimate-code-darwin-x64-baseline": "0.5.20",
-    "@altimateai/altimate-code-linux-x64-baseline-musl": "0.5.20",
-    "@altimateai/altimate-code-darwin-arm64": "0.5.20"
+    "@altimateai/altimate-code-linux-x64-musl": "0.6.0",
+    "@altimateai/altimate-code-linux-x64-baseline": "0.6.0",
+    "@altimateai/altimate-code-windows-arm64": "0.6.0",
+    "@altimateai/altimate-code-darwin-arm64": "0.6.0",
+    "@altimateai/altimate-code-windows-x64-baseline": "0.6.0",
+    "@altimateai/altimate-code-linux-arm64": "0.6.0",
+    "@altimateai/altimate-code-linux-x64-baseline-musl": "0.6.0",
+    "@altimateai/altimate-code-darwin-x64-baseline": "0.6.0",
+    "@altimateai/altimate-code-windows-x64": "0.6.0",
+    "@altimateai/altimate-code-darwin-x64": "0.6.0",
+    "@altimateai/altimate-code-linux-x64": "0.6.0",
+    "@altimateai/altimate-code-linux-arm64-musl": "0.6.0"
   },
   "peerDependencies": {
     "pg": ">=8",

package/skills/data-parity/SKILL.md

@@ -0,0 +1,492 @@
+---
+name: data-parity
+description: Validate that two tables or query results are identical — or diagnose exactly how they differ. Discover schema, identify keys, profile cheaply, then diff. Use for migration validation, ETL regression, and query refactor verification.
+---
+
+# Data Parity (Table Diff)
+
+## CRITICAL: Regulated / Sensitive Data
+
+`data_diff` includes up to 5 **sample diff rows** in the tool output so you can see *which* values differ. Those rows are part of the conversation and are sent to the LLM provider you're using.
+
+Before running `data_diff` against a table that might contain PII, PHI, PCI, or other regulated data:
+
+1. **Ask the user** whether the target contains regulated columns.
+2. If yes, prefer `algorithm: "profile"` — it compares column-level statistics (count, nulls, min/max, distinct count) without any row values leaving the database.
+3. If a row-level diff is genuinely required, tell the user that up to 5 sample rows will be sent to the LLM and get explicit approval before calling the tool.
+4. Consider scoping with `where_clause` to exclude sensitive customers/accounts first.
+
+Default to profile mode whenever the table name suggests regulated data (`customers`, `patients`, `orders`, `payments`, `accounts`, `users`, etc.) unless the user explicitly requests row-level comparison.
+
+---
+
+## CRITICAL: Always Start With a Plan
+
+**Before doing anything else**, generate a numbered TODO list for the user:
+
+```
+Here's my plan:
+1. [ ] List available warehouse connections
+2. [ ] Inspect schema, discover primary key candidates, and detect auto-timestamp columns
+3. [ ] Confirm primary keys with you
+4. [ ] Confirm which auto-timestamp columns to exclude
+5. [ ] Check row counts on both sides
+6. [ ] Run column-level profile (cheap — no row scan)
+7. [ ] Ask whether to proceed with row-level diff (may be expensive for large tables)
+8. [ ] Run targeted row-level diff on diverging columns only
+9. [ ] Present findings with scope, filters, time period, columns compared/excluded, and assumptions
+```
+
+Update each item to `[x]` as you complete it. This plan should be visible before any tool is called.
+
+---
+
+## CRITICAL: Use `data_diff` Tool — Never Write Manual Diff SQL
+
+**NEVER** write SQL to diff tables manually (e.g., `EXCEPT`, `FULL OUTER JOIN`, `MINUS`).
+**ALWAYS** use the `data_diff` tool for any comparison operation.
+
+`sql_query` is only for:
+- Schema inspection (`information_schema`, `SHOW COLUMNS`, `DESCRIBE`)
+- Cardinality checks to identify keys
+- Row count estimates
+
+Everything else — profile, row diff, value comparison — goes through `data_diff`.
+
+---
+
+## Step 1: List Connections
+
+Use `warehouse_list` to show the user what connections are available and which warehouses map to source and target.
+
+---
+
+## Step 2: Inspect Schema, Discover Primary Keys, and Detect Auto-Timestamp Columns
+
+Use `sql_query` to get columns, defaults, and identify key candidates:
+
+```sql
+-- Postgres / Redshift / DuckDB
+SELECT column_name, data_type, is_nullable, column_default
+FROM information_schema.columns
+WHERE table_schema = 'public' AND table_name = 'orders'
+ORDER BY ordinal_position
+```
+
+```sql
+-- Snowflake
+SHOW COLUMNS IN TABLE orders
+```
+
+```sql
+-- MySQL / MariaDB  (also fetch EXTRA for ON UPDATE detection)
+SELECT column_name, data_type, is_nullable, column_default, extra
+FROM information_schema.columns
+WHERE table_schema = 'mydb' AND table_name = 'orders'
+ORDER BY ordinal_position
+```
+
+```sql
+-- SQL Server / Fabric
+SELECT c.name AS column_name, tp.name AS data_type, c.is_nullable,
+       dc.definition AS column_default
+FROM sys.columns c
+INNER JOIN sys.types tp ON c.user_type_id = tp.user_type_id
+INNER JOIN sys.objects o ON c.object_id = o.object_id
+INNER JOIN sys.schemas s ON o.schema_id = s.schema_id
+LEFT JOIN sys.default_constraints dc ON c.default_object_id = dc.object_id
+WHERE s.name = 'dbo' AND o.name = 'orders'
+ORDER BY c.column_id
+```
+
+```sql
+-- ClickHouse
+DESCRIBE TABLE source_db.events
+```
+
+**Look for:** columns named `id`, `*_id`, `*_key`, `uuid`, or with `NOT NULL` + unique index.
+
+**Also look for auto-timestamp columns** — any column whose `column_default` contains a time-generating function:
+- PostgreSQL/DuckDB/Redshift: `now()`, `CURRENT_TIMESTAMP`, `clock_timestamp()`
+- MySQL/MariaDB: `CURRENT_TIMESTAMP` (in default or EXTRA)
+- Snowflake: `CURRENT_TIMESTAMP()`, `SYSDATE()`
+- SQL Server: `getdate()`, `sysdatetime()`
+- Oracle: `SYSDATE`, `SYSTIMESTAMP`
+
+These columns auto-generate values on INSERT, so they inherently differ between source and target due to write timing — not because of actual data discrepancies. **Collect them for confirmation in Step 4.**
+
+If no obvious PK, run a cardinality check:
+
+```sql
+SELECT
+  COUNT(*) AS total_rows,
+  COUNT(DISTINCT order_id) AS distinct_order_id,
+  COUNT(DISTINCT customer_id) AS distinct_customer_id
+FROM orders
+```
+
+A valid key column: `distinct_count = total_rows`.
+
+For composite keys:
+```sql
+SELECT order_id, line_item_id, COUNT(*) AS cnt
+FROM order_lines
+GROUP BY order_id, line_item_id
+HAVING COUNT(*) > 1
+LIMIT 5
+```
+If this returns 0 rows, `(order_id, line_item_id)` is a valid composite key.
+
+## Step 3: Confirm Keys With the User
+
+**Always confirm** the identified key columns before proceeding:
+
+> "I identified `order_id` as the primary key (150,000 distinct values = 150,000 rows, no NULLs). Does that look right, or should I use a different column?"
+
+Do not proceed to diff until the user confirms or corrects.
+
+---
+
+## Step 4: Confirm Auto-Timestamp Column Exclusions
+
+If you detected any columns with auto-generating timestamp defaults in Step 2, **present them to the user and ask for confirmation** before excluding them.
+
+**Example prompt when auto-timestamp columns are found:**
+
+> "I found **3 columns** with auto-generating timestamp defaults that will inherently differ between source and target (due to when each row was written, not actual data differences):
+>
+> | Column | Default | Reason to exclude |
+> |--------|---------|-------------------|
+> | `created_at` | `DEFAULT now()` | Set on insert — reflects when this copy was written |
+> | `updated_at` | `DEFAULT now()` | Set on insert — reflects when this copy was written |
+> | `_loaded_at` | `DEFAULT CURRENT_TIMESTAMP` | ETL load timestamp |
+>
+> Should I **exclude** these from the comparison? Or do you want to include any of them (e.g., if you're verifying that `created_at` was preserved during migration)?"
+
+**If user confirms exclusion:** Omit those columns from `extra_columns` when calling `data_diff`.
+
+**If user wants to include some:** Add them explicitly to `extra_columns`.
+
+**If no auto-timestamp columns were detected:** Skip this step and proceed to Step 5.
+
+> **Why ask?** In migration validation, `created_at` should often be *identical* between source and target (it was migrated, not regenerated). But in ETL replication, `created_at` is freshly generated on each side and *should* differ. Only the user knows which case applies.
+
+---
+
+## Step 5: Check Row Counts
+
+```sql
+SELECT COUNT(*) FROM orders   -- run on both source and target
+```
+
+Use counts to:
+- Detect load completeness issues before row-level diff
+- Choose the algorithm and decide whether to ask about cost
+- If counts differ significantly (>5%), flag it immediately
+
+---
+
+## Step 6: Column-Level Profile (Always Run This First)
+
+Profile is cheap — it runs aggregates, not row scans. **Always run profile before row-level diff.**
+
+```
+data_diff(
+  source="orders",
+  target="orders",
+  key_columns=["order_id"],
+  source_warehouse="postgres_prod",
+  target_warehouse="snowflake_dw",
+  algorithm="profile"
+)
+```
+
+Profile tells you:
+- Row count on each side
+- Which columns have null count differences → NULL handling bug
+- Min/max divergence per column → value transformation bug
+- Which columns match exactly → safe to skip in row-level diff
+
+**Example output:**
+```
+Column Profile Comparison
+
+  ✓ order_id: match
+  ✓ customer_id: match
+  ✗ amount: DIFFER     ← source min=10.00, target min=10.01 — rounding?
+  ✗ status: DIFFER     ← source nulls=0, target nulls=47 — NULL mapping bug?
+  ✓ created_at: match
+```
+
+---
+
+## Step 7: Ask Before Running Row-Level Diff on Large Tables
+
+After profiling, check row count and **ask the user** before proceeding:
+
+**If table has < 100K rows:** proceed automatically.
+
+**If table has 100K–10M rows:**
+> "The table has 1.2M rows. Row-level diff will scan all rows on both sides — this may take 30–60 seconds and consume warehouse compute. Do you want to proceed? You can also provide a `where_clause` to limit the scope (e.g., `created_at >= '2024-01-01'`)."
+
+**If table has > 10M rows:**
+> "The table has 50M rows. Full row-level diff could be expensive. Options:
+> 1. Diff a recent window only (e.g., last 30 days)
+> 2. Partition by a date/key column — shows which partition has problems without scanning everything
+> 3. Proceed with full diff (may take several minutes)
+> Which would you prefer?"
+
+---
+
+## Step 8: Run Targeted Row-Level Diff
+
+Use only the columns that the profile said differ. This is faster and produces cleaner output.
+
+```
+data_diff(
+  source="orders",
+  target="orders",
+  key_columns=["order_id"],
+  extra_columns=["amount", "status"],    // only diverging columns from profile
+  source_warehouse="postgres_prod",
+  target_warehouse="snowflake_dw",
+  algorithm="hashdiff"
+)
+```
+
+### For large tables — use partition_column
+
+Split the table into groups and diff each independently. Three modes:
+
+```
+// Date column — partition by month
+data_diff(source="lineitem", target="lineitem",
+  key_columns=["l_orderkey", "l_linenumber"],
+  source_warehouse="pg_source", target_warehouse="pg_target",
+  partition_column="l_shipdate", partition_granularity="month",
+  algorithm="hashdiff")
+
+// Numeric column — partition by key ranges of 100K
+data_diff(source="orders", target="orders",
+  key_columns=["o_orderkey"],
+  source_warehouse="pg_source", target_warehouse="pg_target",
+  partition_column="o_orderkey", partition_bucket_size=100000,
+  algorithm="hashdiff")
+
+// Categorical column — partition by distinct values (string, enum, boolean)
+data_diff(source="orders", target="orders",
+  key_columns=["o_orderkey"],
+  source_warehouse="pg_source", target_warehouse="pg_target",
+  partition_column="o_orderstatus",
+  algorithm="hashdiff")
+```
+
+Output includes aggregate diff + per-partition breakdown showing which group has problems.
+
+---
+
+## Algorithm Selection
+
+| Algorithm | When to use |
+|-----------|-------------|
+| `profile` | **Always run first** — column stats (count, min, max, nulls). No row scan. |
+| `joindiff` | Same database — single FULL OUTER JOIN. Fast, exact. |
+| `hashdiff` | Cross-database or large tables — bisection with checksums. Scales to billions. |
+| `cascade` | Auto-escalate: profile → hashdiff on diverging columns. |
+| `auto` | JoinDiff if same warehouse, HashDiff if cross-database. |
+
+> **CRITICAL:** If `source_warehouse` ≠ `target_warehouse`, **never use `joindiff`** — it only sees one connection and always reports 0 differences. Use `hashdiff` or `auto`.
+
+---
+
+## Output Interpretation
+
+### IDENTICAL
+```
+✓ Tables are IDENTICAL
+  Rows checked: 1,000,000
+```
+
+### DIFFER
+```
+✗ Tables DIFFER
+
+  Source rows:      150,000
+  Target rows:      149,950
+  Only in source:   50       → rows deleted in target (ETL missed deletes)
+  Only in target:   0
+  Updated rows:     0
+  Identical rows:   149,950
+```
+
+| Pattern | Root cause |
+|---------|-----------|
+| `only_in_source > 0`, target = 0 | ETL dropped rows — check filters, incremental logic |
+| `only_in_target > 0`, source = 0 | Target has extra rows — dedup issue or wrong join |
+| `updated_rows > 0`, counts match | Silent value corruption — check type casts, rounding |
+| Row counts differ significantly | Load completeness — check ETL watermarks |
+
+---
+
+## CRITICAL: `extra_columns` Behavior
+
+The Rust engine **only compares columns listed in `extra_columns`**. If the list is empty, it compares key existence only — rows that match on key but differ in values will be silently reported as "identical". This is the most common source of false positives.
+
+**Auto-discovery (default for table names):** When `extra_columns` is omitted and the source is a plain table name, `data_diff` auto-discovers all non-key columns from the database catalog and excludes columns using two detection layers:
+
+1. **Name-pattern matching** — columns named like `updated_at`, `created_at`, `inserted_at`, `modified_at`, `publisher_last_updated_epoch_ms`, ETL metadata columns like `_fivetran_synced`, `_airbyte_extracted_at`, etc.
+2. **Schema-level default detection** — columns with auto-generating timestamp defaults (`DEFAULT NOW()`, `DEFAULT CURRENT_TIMESTAMP`, `GETDATE()`, `SYSDATE()`, `SYSTIMESTAMP`, etc.), detected directly from the database catalog. This catches columns that don't follow naming conventions but still auto-generate values on INSERT. Works across PostgreSQL, MySQL, Snowflake, SQL Server, Oracle, ClickHouse, DuckDB, SQLite, and Redshift.
+
+The output lists which columns were auto-excluded and why.
+
+**SQL queries:** When source is a SQL query (not a table name), auto-discovery cannot work. You **must** provide `extra_columns` explicitly. If you don't, only key-level matching occurs.
+
+**When to override auto-exclusion:** If the user specifically wants to compare audit columns (e.g., verifying that `created_at` was preserved during migration), pass those columns explicitly in `extra_columns`.
+
+---
+
+## Step 9: Present Findings — Always Surface Context
+
+When reporting diff results, **never present bare numbers**. Always frame the result with the full context that determines what the numbers actually mean.
+
+### Required elements in every result summary
+
+**1. Scope — what was compared**
+State exactly which tables/queries were diffed and on which warehouses:
+> "Compared `public.orders` on **postgres_prod** vs `public.orders` on **snowflake_dw**"
+
+**2. Filters and time period applied**
+If any `where_clause` or `partition_column` was used, state it explicitly:
+> "Scope limited to: `created_at >= '2024-01-01' AND created_at < '2024-04-01'` (Q1 2024 only)"
+> "Partitioned by `l_shipdate` (monthly buckets) — diff covers Jan 2023 through Mar 2024"
+
+If no filter was applied, say so:
+> "No row filter applied — full table compared"
+
+**3. Key columns used**
+> "Key: `order_id` (confirmed unique — 150,000 distinct values = 150,000 rows)"
+
+**4. Columns included and excluded**
+List what was compared and what was skipped, and why:
+> "Compared columns: `amount`, `status`, `customer_id`"
+> "Excluded (auto-timestamp defaults): `created_at`, `updated_at`, `_loaded_at`"
+> "Excluded (user request): `internal_score`"
+
+If the user confirmed exclusions in Step 4, reference that confirmation:
+> "Excluded per your confirmation: `created_at`, `updated_at`"
+
+**5. Algorithm used**
+> "Algorithm: `hashdiff` (cross-database)"
+
+### Example full result summary
+
+```
+## Data Parity Results
+
+**Compared:** `public.orders` (postgres_prod) → `public.orders` (snowflake_dw)
+**Scope:** `created_at >= '2024-01-01'` (Q1 2024 only — 42,301 rows in scope)
+**Key:** `order_id`
+**Columns compared:** `amount`, `status`, `customer_id`, `region`
+**Columns excluded:** `created_at`, `updated_at` (auto-timestamp defaults, per your confirmation)
+**Algorithm:** hashdiff
+
+### Result: ✗ DIFFER
+
+| Metric | Value |
+|--------|-------|
+| Source rows | 42,301 |
+| Target rows | 42,298 |
+| Only in source | 3 |
+| Only in target | 0 |
+| Updated rows | 47 |
+| Identical rows | 42,251 |
+
+**Findings:**
+- 3 rows exist in source but are missing in target → possible ETL delete propagation gap
+- 47 rows have value differences in `amount` or `status` → check rounding or status mapping
+```
+
+### When result is IDENTICAL — still surface the scope
+
+Even when tables match perfectly, state what was checked:
+> "✓ Tables are **identical** across 150,000 rows. Compared `amount`, `status`, `customer_id` (full table, no filter, key=`order_id`). Auto-timestamp columns `created_at`, `updated_at` were excluded."
+
+**Why this matters:** "Tables are identical" without context is meaningless — the user needs to know if you checked Q1 only, skipped 5 columns, or used a WHERE clause that covered just 1% of the data.
+
+---
+
+## Common Mistakes
+
+**Writing manual diff SQL instead of calling data_diff**
+→ Never use EXCEPT, MINUS, or FULL OUTER JOIN to diff tables. Use `data_diff`.
+
+**Calling data_diff without confirming the key**
+→ Confirm cardinality with the user first. A bad key gives meaningless results.
+
+**Using joindiff for cross-database tables**
+→ JoinDiff can't see the remote table. Always returns 0 diffs. Use `hashdiff` or `auto`.
+
+**Skipping the profile step and jumping to full row diff**
+→ Profile is free. It tells you which columns actually differ so you avoid scanning everything.
+
+**Running full diff on a billion-row table without asking**
+→ Always ask the user before expensive operations. Offer filtering and partition options.
+
+**Omitting extra_columns when source is a SQL query**
+→ Auto-discovery only works for table names. For SQL queries, always list the columns to compare explicitly.
+
+**Silently excluding auto-timestamp columns without asking the user**
+→ Always present detected auto-timestamp columns (Step 4) and get explicit confirmation. In migration scenarios, `created_at` should be *identical* — excluding it silently hides real bugs.
+
+---
+
+## SQL Server and Microsoft Fabric
+
+### Minimum Version Requirements
+
+| Component | Minimum Version | Why |
+|---|---|---|
+| **SQL Server** | 2022 (16.x) | `DATETRUNC()` used for date partitioning; `LEAST()`/`GREATEST()` used by Rust engine |
+| **Azure SQL Database** | Any current version | Always has `DATETRUNC()` and `LEAST()` |
+| **Microsoft Fabric** | Any current version | T-SQL surface includes all required functions |
+| **mssql** (npm) | 12.0.0 | `ConnectionPool` isolation for concurrent connections, tedious 19 |
+| **@azure/identity** (npm) | 4.0.0 | Required only for Azure AD authentication; tedious imports it internally |
+
+> **Note:** Date partitioning (`partition_column` + `partition_granularity`) uses `DATETRUNC()` which is **not available on SQL Server 2019 or earlier**. Basic diff operations (joindiff, hashdiff, profile) work on older versions. If you need partitioned diffs on SQL Server < 2022, use numeric or categorical partitioning instead.
+
+### Supported Configurations
+
+| Warehouse Type | Authentication | Notes |
+|---|---|---|
+| `sqlserver` / `mssql` | User/password or Azure AD | On-prem or Azure SQL. SQL Server 2022+ required for date partitioning. |
+| `fabric` | Azure AD only | Microsoft Fabric SQL endpoint. Always uses TLS encryption. |
+
+### Connecting to Microsoft Fabric
+
+Fabric uses the same TDS protocol as SQL Server — no separate driver needed. Configuration:
+
+```yaml
+type: "fabric"
+host: "<workspace-id>-<item-id>.datawarehouse.fabric.microsoft.com"
+database: "<warehouse-name>"
+authentication: "azure-active-directory-default"   # recommended
+```
+
+Auth shorthands (mapped to full tedious type names):
+- `CLI` or `default` → `azure-active-directory-default`
+- `password` → `azure-active-directory-password`
+- `service-principal` → `azure-active-directory-service-principal-secret`
+- `msi` or `managed-identity` → `azure-active-directory-msi-vm`
+
+Full Azure AD authentication types:
+- `azure-active-directory-default` — auto-discovers credentials via `DefaultAzureCredential` (recommended; works with `az login`)
+- `azure-active-directory-password` — username/password with `azure_client_id` and `azure_tenant_id`
+- `azure-active-directory-access-token` — pre-obtained token (does **not** auto-refresh)
+- `azure-active-directory-service-principal-secret` — service principal with `azure_client_id`, `azure_client_secret`, `azure_tenant_id`
+- `azure-active-directory-msi-vm` / `azure-active-directory-msi-app-service` — managed identity
+
+### Algorithm Behavior
+
+- **Same-warehouse** MSSQL or Fabric → `joindiff` (single FULL OUTER JOIN, most efficient)
+- **Cross-warehouse** MSSQL/Fabric ↔ other database → `hashdiff` (automatic when using `auto`)
+- The Rust engine maps `sqlserver`/`mssql` to `tsql` dialect and `fabric` to `fabric` dialect — both generate valid T-SQL syntax with bracket quoting (`[schema].[table]`).

package/skills/dbt-test/SKILL.md

@@ -81,6 +81,8 @@ altimate-dbt build --model <name>         # build + test together
 
 ## Unit Test Workflow
 
+**For automated unit test generation, use the `dbt-unit-tests` skill instead.** It analyzes model SQL, generates type-correct mock data, and assembles complete YAML automatically.
+
 See [references/unit-test-guide.md](references/unit-test-guide.md) for the full unit test framework.
 
 ### Quick Pattern

package/skills/dbt-unit-tests/SKILL.md

@@ -0,0 +1,209 @@
+---
+name: dbt-unit-tests
+description: Generate dbt unit tests automatically for any model. Analyzes SQL logic (CASE/WHEN, JOINs, window functions, NULLs), creates type-correct mock inputs from manifest schema, and assembles complete YAML. Use when a user says "generate tests", "add unit tests", "test this model", or "test coverage" for dbt models.
+---
+
+# dbt Unit Test Generation
+
+## Requirements
+**Agent:** builder or migrator (requires file write access)
+**Tools used:** dbt_unit_test_gen, dbt_manifest, dbt_lineage, altimate_core_validate, altimate_core_testgen, bash (runs `altimate-dbt` commands), read, glob, write, edit
+
+## When to Use This Skill
+
+**Use when the user wants to:**
+- Generate unit tests for a dbt model
+- Add test coverage to an existing model
+- Create mock data for testing
+- Test-driven development (TDD) for dbt
+- Verify CASE/WHEN logic, NULL handling, JOIN behavior, or aggregation correctness
+- Test incremental model logic
+
+**Do NOT use for:**
+- Adding schema tests (not_null, unique, accepted_values) -> use `dbt-test`
+- Creating or modifying model SQL -> use `dbt-develop`
+- Writing descriptions -> use `dbt-docs`
+- Debugging build failures -> use `dbt-troubleshoot`
+
+## The Iron Rules
+
+1. **Never guess expected outputs.** Compute them by running SQL against mock data when possible. If you cannot run SQL, clearly mark expected outputs as placeholders that need verification.
+2. **Never skip upstream dependencies.** Every ref() and source() the model touches MUST have a mock input. Miss one and the test won't compile.
+3. **Use sql format for ephemeral models.** Dict format fails silently for ephemeral upstreams.
+4. **Never weaken a test to make it pass.** If the test fails, the model logic may be wrong. Investigate before changing expected values.
+5. **Compile before committing.** Always run `altimate-dbt test --model <name>` to verify tests compile and execute.
+
+## Core Workflow: Analyze -> Generate -> Refine -> Validate -> Write
+
+### Phase 1: Analyze the Model
+
+Before generating any tests, deeply understand the model:
+
+```bash
+# 1. Ensure manifest is compiled
+altimate-dbt compile --model <name>
+
+# 2. Read the model SQL
+read <model_sql_file>
+
+# 3. Parse the manifest for dependencies
+dbt_unit_test_gen(manifest_path: "target/manifest.json", model: "<name>")
+```
+
+**What to look for:**
+- Which upstream refs/sources does this model depend on?
+- What SQL constructs need testing? (CASE/WHEN, JOINs, window functions, aggregations)
+- What edge cases exist? (NULLs, empty strings, zero values, boundary dates)
+- Is this an incremental model? (needs `is_incremental` override tests)
+- Are any upstream models ephemeral? (need sql format)
+
+### Phase 2: Generate Tests
+
+The `dbt_unit_test_gen` tool does the heavy lifting:
+
+```text
+dbt_unit_test_gen(
+  manifest_path: "target/manifest.json",
+  model: "fct_orders",
+  max_scenarios: 5
+)
+```
+
+This returns:
+- Complete YAML with mock inputs and expected outputs
+- Semantic context: model/column descriptions, column lineage, compiled SQL
+- List of anti-patterns that informed edge case generation
+- Warnings about ephemeral deps, missing columns, etc.
+
+**If the tool reports missing columns** (placeholder rows in the YAML), discover them:
+```bash
+altimate-dbt columns --model <upstream_model_name>
+altimate-dbt columns-source --source <source_name> --table <table_name>
+```
+Then update the generated YAML with real column names.
+
+### Phase 3: Refine Expected Outputs
+
+**This is the critical step that differentiates good tests from bad ones.**
+
+The tool generates placeholder expected outputs based on column types. You MUST refine them:
+
+**Option A: Compute by running SQL (preferred)**
+```bash
+# Run the model against mock data to get actual output
+altimate-dbt test --model <name>
+# If the test fails, the error shows actual vs expected — use actual as expected
+```
+
+**Option B: Manual computation**
+Read the model SQL carefully and mentally execute it against the mock inputs.
+For each test case:
+1. Look at the mock input rows
+2. Trace through the SQL logic (CASE/WHEN branches, JOINs, aggregations)
+3. Write the correct expected output
+
+**Option C: Use the warehouse (most accurate)**
+```bash
+# Build a CTE query with mock data and run the model SQL against it
+altimate-dbt execute --query "WITH mock_stg_orders AS (SELECT 1 AS order_id, 100.00 AS amount) SELECT * FROM (<model_sql>) sub"
+```
+
+### Phase 4: Validate
+
+```bash
+# 1. Run the unit tests
+altimate-dbt test --model <name>
+
+# 2. If tests fail, read the error carefully
+#    - Compilation error? Missing ref, wrong column name, type mismatch
+#    - Assertion error? Expected output doesn't match actual
+
+# 3. Fix and retry (max 3 iterations)
+```
+
+### Phase 5: Write to File
+
+Place unit tests in one of these locations (match project convention):
+- `models/<layer>/_unit_tests.yml` (dedicated file)
+- `models/<layer>/schema.yml` (append to existing)
+
+```bash
+# Check existing convention
+glob models/**/*unit_test*.yml models/**/*schema*.yml
+
+# Write or append
+edit <yaml_file>  # if file exists
+write <yaml_file> # if creating new
+```
+
+## Test Case Categories
+
+### Happy Path (always generate)
+Standard inputs that exercise the main logic path. 2 rows minimum.
+
+### NULL Handling
+Set nullable columns to NULL in the last row. Verify COALESCE/NVL/IFNULL behavior.
+
+### Boundary Values
+Zero amounts, empty strings, epoch dates, MAX values. Tests robustness.
+
+### Edge Cases
+- Division by zero (if model divides)
+- Non-matching JOINs (LEFT JOIN with no match)
+- Single-row aggregation
+- Duplicate key handling
+
+### Incremental
+For incremental models only. Use `overrides.macros.is_incremental: true` to test the incremental path.
+
+## Common Mistakes
+
+| Mistake | Fix |
+|---------|-----|
+| Missing a ref() in given | Parse manifest for ALL depends_on nodes |
+| Wrong column names in mock data | Use manifest columns, not guesses |
+| Wrong data types | Use schema catalog types |
+| Expected output is just mock input | Actually compute the transformation |
+| Dict format for ephemeral model | Use `format: sql` with raw SQL |
+| Not testing NULL path in COALESCE | Add null_handling test case |
+| Hardcoded dates with current_timestamp | Use overrides.macros to mock timestamps |
+| Testing trivial pass-through | Skip models with no logic |
+
+## YAML Format Reference
+
+```yaml
+unit_tests:
+  - name: test_<model>_<scenario>
+    description: "What this test verifies"
+    model: <model_name>
+    overrides:                    # optional
+      macros:
+        is_incremental: true      # for incremental models
+      vars:
+        run_date: "2024-01-15"    # for date-dependent logic
+    given:
+      - input: ref('upstream_model')
+        rows:
+          - { col1: value1, col2: value2 }
+      - input: source('source_name', 'table_name')
+        rows:
+          - { col1: value1 }
+      - input: ref('ephemeral_model')
+        format: sql
+        rows: |
+          SELECT 1 AS id, 'test' AS name
+          UNION ALL
+          SELECT 2 AS id, 'other' AS name
+    expect:
+      rows:
+        - { output_col1: expected1, output_col2: expected2 }
+```
+
+## Reference Guides
+
+| Guide | Use When |
+|-------|----------|
+| [references/unit-test-yaml-spec.md](references/unit-test-yaml-spec.md) | Full YAML specification and format details |
+| [references/edge-case-patterns.md](references/edge-case-patterns.md) | Catalog of edge cases by SQL construct |
+| [references/incremental-testing.md](references/incremental-testing.md) | Testing incremental models |
+| [references/altimate-dbt-commands.md](references/altimate-dbt-commands.md) | Full CLI reference |

package/skills/dbt-unit-tests/references/altimate-dbt-commands.md

@@ -0,0 +1,66 @@
+# altimate-dbt Command Reference
+
+All dbt operations use the `altimate-dbt` CLI. Output is JSON to stdout; logs go to stderr.
+
+```bash
+altimate-dbt <command> [args...]
+altimate-dbt <command> [args...] --format text    # Human-readable output
+```
+
+## First-Time Setup
+
+```bash
+altimate-dbt init                          # Auto-detect project root
+altimate-dbt init --project-root /path     # Explicit root
+altimate-dbt init --python-path /path      # Override Python
+altimate-dbt doctor                        # Verify setup
+altimate-dbt info                          # Project name, adapter, root
+```
+
+## Build & Run
+
+```bash
+altimate-dbt build                                  # full project build (compile + run + test)
+altimate-dbt build --model <name> [--downstream]   # build a single model
+altimate-dbt run --model <name> [--downstream]      # materialize only
+altimate-dbt test --model <name>                     # run tests only
+```
+
+## Compile
+
+```bash
+altimate-dbt compile --model <name>
+altimate-dbt compile-query --query "SELECT * FROM {{ ref('stg_orders') }}" [--model <context>]
+```
+
+## Execute SQL
+
+```bash
+altimate-dbt execute --query "SELECT count(*) FROM {{ ref('orders') }}" --limit 100
+```
+
+## Schema & DAG
+
+```bash
+altimate-dbt columns --model <name>                         # column names and types
+altimate-dbt columns-source --source <src> --table <tbl>    # source table columns
+altimate-dbt column-values --model <name> --column <col>    # sample values
+altimate-dbt children --model <name>                        # downstream models
+altimate-dbt parents --model <name>                         # upstream models
+```
+
+## Packages
+
+```bash
+altimate-dbt deps                                           # install packages.yml
+altimate-dbt add-packages --packages dbt-utils,dbt-expectations
+```
+
+## Error Handling
+
+All errors return JSON with `error` and `fix` fields:
+```json
+{ "error": "dbt-core is not installed", "fix": "Install it: python3 -m pip install dbt-core" }
+```
+
+Run `altimate-dbt doctor` as the first diagnostic step for any failure.

package/skills/dbt-unit-tests/references/edge-case-patterns.md

@@ -0,0 +1,189 @@
+# Edge Case Patterns by SQL Construct
+
+## CASE/WHEN
+
+**What to test:** Every branch, including ELSE/default.
+
+```yaml
+# Test the TRUE branch
+- { status: "completed", amount: 100 }
+# Expected: { category: "done" }
+
+# Test the FALSE/ELSE branch
+- { status: "unknown", amount: 100 }
+# Expected: { category: "other" }
+
+# Test NULL input
+- { status: null, amount: 100 }
+# Expected: depends on whether NULL matches any WHEN
+```
+
+**Common bugs:**
+- NULL doesn't match `WHEN status = 'active'` — it falls to ELSE
+- Multiple WHEN clauses: first match wins, test ordering
+
+## COALESCE / NVL / IFNULL
+
+**What to test:** NULL in each position.
+
+```yaml
+# COALESCE(a, b, c) — test a=NULL
+- { a: null, b: "fallback", c: "default" }
+# Expected: { result: "fallback" }
+
+# COALESCE(a, b, c) — test a=NULL, b=NULL
+- { a: null, b: null, c: "default" }
+# Expected: { result: "default" }
+
+# All non-null
+- { a: "primary", b: "fallback", c: "default" }
+# Expected: { result: "primary" }
+```
+
+## JOINs
+
+**What to test:** Matching rows, non-matching rows, NULL join keys.
+
+```yaml
+# LEFT JOIN — matching row
+orders: [{ order_id: 1, customer_id: 1 }]
+customers: [{ customer_id: 1, name: "Alice" }]
+# Expected: { order_id: 1, name: "Alice" }
+
+# LEFT JOIN — no match (customer missing)
+orders: [{ order_id: 2, customer_id: 99 }]
+customers: [{ customer_id: 1, name: "Alice" }]
+# Expected: { order_id: 2, name: null }
+
+# JOIN with NULL key
+orders: [{ order_id: 3, customer_id: null }]
+customers: [{ customer_id: 1, name: "Alice" }]
+# Expected: depends on join type
+```
+
+**Common bugs:**
+- INNER JOIN drops rows when key is NULL or missing
+- Fan-out: duplicate keys in right table multiply left rows
+
+## Window Functions
+
+**What to test:** Ordering, partitioning, boundary rows.
+
+```yaml
+# ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY order_date)
+- { customer_id: 1, order_date: "2024-01-01", amount: 50 }
+- { customer_id: 1, order_date: "2024-01-15", amount: 75 }
+- { customer_id: 2, order_date: "2024-01-10", amount: 30 }
+# Expected:
+# { customer_id: 1, order_date: "2024-01-01", row_num: 1 }
+# { customer_id: 1, order_date: "2024-01-15", row_num: 2 }
+# { customer_id: 2, order_date: "2024-01-10", row_num: 1 }
+```
+
+**What to test for LAG/LEAD:**
+- First/last row in partition (LAG returns NULL for first row)
+- Single-row partition
+
+## Aggregations (GROUP BY)
+
+**What to test:** Multiple groups, single group, empty group.
+
+```yaml
+# SUM(amount) GROUP BY customer_id
+- { customer_id: 1, amount: 50 }
+- { customer_id: 1, amount: 25 }
+- { customer_id: 2, amount: 100 }
+# Expected:
+# { customer_id: 1, total_amount: 75 }
+# { customer_id: 2, total_amount: 100 }
+
+# Single row group
+- { customer_id: 3, amount: 10 }
+# Expected: { customer_id: 3, total_amount: 10 }
+
+# NULL in aggregated column
+- { customer_id: 4, amount: null }
+# Expected: { customer_id: 4, total_amount: null }  # SUM of NULLs = NULL
+```
+
+## Division
+
+**What to test:** Normal, divide by zero, NULL.
+
+```yaml
+# amount / quantity
+- { amount: 100, quantity: 4 }
+# Expected: { unit_price: 25 }
+
+# Divide by zero
+- { amount: 100, quantity: 0 }
+# Expected: depends — NULL, error, or COALESCE fallback?
+
+# NULL divisor
+- { amount: 100, quantity: null }
+# Expected: { unit_price: null }
+```
+
+## Date/Timestamp Logic
+
+**What to test:** Boundaries, NULL dates, timezone edge cases.
+
+```yaml
+# DATEDIFF or date filtering
+- { event_date: "2024-01-01" }  # start of year
+- { event_date: "2024-12-31" }  # end of year
+- { event_date: "2024-02-29" }  # leap year
+- { event_date: null }           # NULL date
+```
+
+## Type Coercion
+
+**What to test:** Implicit casts that may fail.
+
+```yaml
+# String that looks like a number
+- { amount_str: "100.50" }  # CAST to DECIMAL
+- { amount_str: "not_a_number" }  # should this fail?
+- { amount_str: "" }  # empty string cast
+- { amount_str: null }  # NULL cast
+```
+
+## Incremental Models
+
+**What to test:** Full refresh vs incremental path.
+
+```yaml
+# Test 1: Full refresh (is_incremental = false, default)
+# All rows processed
+
+# Test 2: Incremental (is_incremental = true)
+unit_tests:
+  - name: test_incremental_new_rows_only
+    model: fct_orders
+    overrides:
+      macros:
+        is_incremental: true
+    given:
+      - input: this  # existing table state
+        rows:
+          - { order_id: 1, updated_at: "2024-01-14" }
+      - input: ref('stg_orders')
+        rows:
+          - { order_id: 1, updated_at: "2024-01-14" }  # old, should be skipped
+          - { order_id: 2, updated_at: "2024-01-15" }  # new, should be processed
+    expect:
+      rows:
+        - { order_id: 2, updated_at: "2024-01-15" }
+```
+
+## Empty Inputs
+
+**What to test:** Model behavior when upstream has zero rows.
+
+```yaml
+given:
+  - input: ref('stg_orders')
+    rows: []
+expect:
+  rows: []  # or specific default behavior
+```

package/skills/dbt-unit-tests/references/incremental-testing.md

@@ -0,0 +1,109 @@
+# Testing Incremental dbt Models
+
+Incremental models have two code paths controlled by `{% if is_incremental() %}`. Both paths must be tested.
+
+## The Two Paths
+
+```sql
+SELECT * FROM {{ ref('stg_orders') }}
+
+{% if is_incremental() %}
+  -- Incremental path: only process new rows
+  WHERE updated_at > (SELECT MAX(updated_at) FROM {{ this }})
+{% endif %}
+```
+
+## Test 1: Full Refresh
+
+```yaml
+unit_tests:
+  - name: test_fct_orders_full_refresh
+    description: "Full refresh processes all rows"
+    model: fct_orders
+    # No overrides needed — is_incremental defaults to false
+    given:
+      - input: ref('stg_orders')
+        rows:
+          - { order_id: 1, amount: 100, updated_at: "2024-01-10" }
+          - { order_id: 2, amount: 200, updated_at: "2024-01-15" }
+    expect:
+      rows:
+        - { order_id: 1, amount: 100, updated_at: "2024-01-10" }
+        - { order_id: 2, amount: 200, updated_at: "2024-01-15" }
+```
+
+## Test 2: Incremental — New Rows Only
+
+```yaml
+unit_tests:
+  - name: test_fct_orders_incremental_new_only
+    description: "Incremental run only processes rows newer than existing max"
+    model: fct_orders
+    overrides:
+      macros:
+        is_incremental: true
+    given:
+      - input: this   # mock the existing target table
+        rows:
+          - { order_id: 1, amount: 100, updated_at: "2024-01-10" }
+      - input: ref('stg_orders')
+        rows:
+          - { order_id: 1, amount: 100, updated_at: "2024-01-10" }  # old
+          - { order_id: 2, amount: 200, updated_at: "2024-01-15" }  # new
+    expect:
+      rows:
+        - { order_id: 2, amount: 200, updated_at: "2024-01-15" }
+```
+
+## Test 3: Incremental — Updated Rows
+
+If your model uses `unique_key` for merge/upsert:
+
+```yaml
+unit_tests:
+  - name: test_fct_orders_incremental_update
+    description: "Updated rows are captured in incremental run"
+    model: fct_orders
+    overrides:
+      macros:
+        is_incremental: true
+    given:
+      - input: this
+        rows:
+          - { order_id: 1, amount: 100, updated_at: "2024-01-10" }
+      - input: ref('stg_orders')
+        rows:
+          - { order_id: 1, amount: 150, updated_at: "2024-01-15" }  # updated
+    expect:
+      rows:
+        - { order_id: 1, amount: 150, updated_at: "2024-01-15" }
+```
+
+## Test 4: Incremental — Empty Source
+
+```yaml
+unit_tests:
+  - name: test_fct_orders_incremental_no_new_data
+    description: "No new rows when source has nothing newer"
+    model: fct_orders
+    overrides:
+      macros:
+        is_incremental: true
+    given:
+      - input: this
+        rows:
+          - { order_id: 1, amount: 100, updated_at: "2024-01-15" }
+      - input: ref('stg_orders')
+        rows:
+          - { order_id: 1, amount: 100, updated_at: "2024-01-10" }  # older
+    expect:
+      rows: []
+```
+
+## Key Points
+
+1. **Always mock `this`** when testing incremental path — it represents the existing target table
+2. **Set `is_incremental: true`** in overrides.macros to activate the incremental code path
+3. **Test both paths** — full refresh AND incremental
+4. **Include overlap rows** — rows that exist in both `this` and source to verify filtering
+5. **Test the merge key** — if `unique_key` is set, verify upsert behavior

package/skills/dbt-unit-tests/references/unit-test-yaml-spec.md

@@ -0,0 +1,174 @@
+# dbt Unit Test YAML Specification
+
+Available in dbt-core 1.8+ (released mid-2024).
+
+## Top-Level Structure
+
+Unit tests are defined under the `unit_tests:` key in any YAML file within your dbt project (typically `schema.yml` or `_unit_tests.yml`).
+
+```yaml
+unit_tests:
+  - name: <test_name>           # required, snake_case
+    description: <string>        # optional but recommended
+    model: <model_name>          # required, the model being tested
+    given: <list of inputs>      # required, mock input data
+    expect: <expected output>    # required, expected output rows
+    overrides: <overrides>       # optional, macro/var overrides
+    config: <config>             # optional, test configuration
+    tags: <list>                 # optional, for filtering
+```
+
+## Input Formats
+
+### Dict Format (default, preferred)
+
+```yaml
+given:
+  - input: ref('stg_orders')
+    rows:
+      - { order_id: 1, amount: 100.00, status: "completed" }
+      - { order_id: 2, amount: null, status: "pending" }
+```
+
+**Rules:**
+- Only include columns that the model actually uses
+- Column names must match the upstream model exactly
+- Use `null` for NULL values (not empty string)
+- Dates as strings: `"2024-01-15"`
+- Timestamps as strings: `"2024-01-15 10:30:00"`
+- Booleans: `true` / `false`
+- Numbers: no quotes (`100.00`, not `"100.00"`)
+
+### SQL Format (required for ephemeral models)
+
+```yaml
+given:
+  - input: ref('ephemeral_model')
+    format: sql
+    rows: |
+      SELECT 1 AS id, 'test' AS name
+      UNION ALL
+      SELECT 2 AS id, 'other' AS name
+```
+
+**When to use SQL format:**
+- Upstream model is materialized as `ephemeral`
+- Complex data types that dict can't represent
+- Need to use SQL functions in mock data
+
+### Empty Input
+
+```yaml
+given:
+  - input: ref('stg_orders')
+    rows: []
+```
+
+Tests behavior with no input rows (empty table).
+
+## Expected Output
+
+```yaml
+expect:
+  rows:
+    - { order_id: 1, net_revenue: 85.00 }
+    - { order_id: 2, net_revenue: 50.00 }
+```
+
+**Rules:**
+- Only include columns you want to assert on (subset is OK)
+- Row order matters — rows are compared positionally
+- Use exact values for numeric assertions
+- `null` to assert NULL output
+
+## Overrides
+
+### Macro Overrides
+
+```yaml
+overrides:
+  macros:
+    is_incremental: true          # boolean
+    current_timestamp: "2024-01-15 00:00:00"  # string
+```
+
+Common macros to override:
+- `is_incremental` — test incremental vs full-refresh path
+- `current_timestamp` / `current_date` — deterministic date testing
+
+### Variable Overrides
+
+```yaml
+overrides:
+  vars:
+    run_date: "2024-01-15"
+    lookback_days: 30
+```
+
+## Input Sources
+
+### ref() — Model references
+
+```yaml
+- input: ref('model_name')
+```
+
+### source() — Source table references
+
+```yaml
+- input: source('source_name', 'table_name')
+```
+
+### this — Self-reference for incremental models
+
+```yaml
+- input: this
+  rows:
+    - { order_id: 1, updated_at: "2024-01-14" }
+```
+
+Used with `overrides.macros.is_incremental: true` to mock the existing table state.
+
+## Configuration
+
+Tags can be set at the top level (sibling of `config`) or nested under `config`:
+
+```yaml
+unit_tests:
+  - name: test_example
+    model: fct_orders
+    tags: ["unit-test", "revenue"]
+    # ... rest of test
+```
+
+Or via config:
+
+```yaml
+unit_tests:
+  - name: test_example
+    model: fct_orders
+    config:
+      tags: ["unit-test", "revenue"]
+```
+
+## Naming Conventions
+
+- Test names: `test_<model>_<what_it_tests>`
+- Examples:
+  - `test_fct_orders_happy_path`
+  - `test_fct_orders_null_discount`
+  - `test_fct_orders_zero_quantity`
+  - `test_fct_orders_incremental_new_rows`
+
+## Running Unit Tests
+
+```bash
+dbt test --select test_type:unit                    # all unit tests
+dbt test --select test_type:unit,model_name:fct_orders  # unit tests for one model
+dbt build --select +fct_orders                      # build + all tests
+```
+
+## Official Documentation
+
+- https://docs.getdbt.com/docs/build/unit-tests
+- https://docs.getdbt.com/reference/resource-properties/unit-tests