@kaelio/ktx 0.0.0 → 0.1.0-rc.1

package/node_modules/@ktx/context/skills/metabase_ingest/SKILL.md

@@ -0,0 +1,253 @@
+---
+name: metabase_ingest
+description: Convert Metabase questions, models, and metrics into KTX Semantic Layer source definitions. Covers result-metadata to KSL column type mapping, FK/PK detection, near-duplicate deduplication, pre-aggregation decomposition, join-graph connectivity, and how to react to priorProvenance from earlier ingest syncs. Load when the WorkUnit contains `cards/<id>.json` files under a Metabase bundle.
+callers: [memory_agent]
+---
+
+# Metabase to KTX Semantic Layer
+
+Each WorkUnit represents one Metabase collection's cards for one Metabase database (mapped to exactly one KTX connection). Every `cards/<id>.json` file carries the resolved SQL, result_metadata, card type, collection path, and referenced-card ids. The WU's `sync-config.json` tells you which sync mode is active and which selections apply. `databases/<id>.json` tells you the target KTX connection.
+
+## Context format
+
+Each card JSON looks like:
+```json
+{
+  "metabaseId": 7,
+  "name": "Daily orders",
+  "description": "Orders by day",
+  "type": "model",
+  "databaseId": 42,
+  "collectionId": 5,
+  "resolvedSql": "SELECT ...",
+  "templateTags": [{"name": "ref", "type": "card", "cardReference": 10}],
+  "resultMetadata": [
+    {"name": "day", "base_type": "type/DateTime", "semantic_type": "type/CreationTimestamp"},
+    {"name": "order_count", "base_type": "type/Integer"}
+  ],
+  "collectionPath": ["Data", "Orders Team"],
+  "referencedCardIds": [10]
+}
+```
+
+Use `resultMetadata` to:
+- Map `base_type` to KSL column type: `type/Integer`, `type/Float`, `type/Decimal`, `type/BigInteger` → `number`; `type/Text`, `type/TextLike` → `string`; `type/DateTime`, `type/Date`, `type/DateTimeWithTZ` → `time`; `type/Boolean` → `boolean`.
+- Identify grain candidates: columns with `semantic_type: type/PK`.
+- Identify join candidates: columns with `semantic_type: type/FK` plus `fk_target_field_id`.
+- Identify time columns: `semantic_type: type/CreationTimestamp` or `type/UpdatedTimestamp` → set `role: time`.
+- Use `display_name` for measure descriptions when available.
+
+### Additional card metadata
+
+- `parameters`: list of card-level parameters with widget types and defaults. When SQL resolution fell back to unresolved SQL, use this to drive Step A of the SQL-translation workflow (drop optional clauses): knowing each `{{ var }}` is `type: "date/range"` vs `type: "category"` tells you what kind of clause it is.
+- `resultMetadata[i].field_ref`: Metabase's canonical reference to the source warehouse field. Shape `["field", <field_id>, <options>]`. When this is set, the column maps directly to a warehouse field, which is useful for declaring joins from FK metadata without re-parsing SQL.
+- `lastRunAt`: ISO timestamp of the card's last execution. If null or very old, the card may be dead; prefer skipping over creating a source.
+- `dashboardCount`: number of dashboards referencing the card. Cards with `dashboardCount: 0` and a stale `lastRunAt` are strong skip signals.
+
+Before writing a wiki page derived from a Metabase question SQL, verify each
+schema.table.column mentioned with entity_details.
+
+## Identifier Verification Protocol
+
+Before writing a wiki page or SL source on any topic:
+
+1. `discover_data({query: "<topic>"})` - see what wikis, SL sources, and raw
+   tables already exist. Prefer updating existing pages over creating new ones.
+
+Before emitting any `schema.table` or `schema.table.column` into a wiki body,
+SL source, `tables:` frontmatter, `sl_refs`, or `emit_unmapped_fallback`:
+
+2. `entity_details({connectionId, targets: [{display: "<identifier>"}]})` -
+   confirm the identifier resolves; inspect native types, FK/PK, and
+   sampleValues.
+3. For literal values from the source, such as status codes or plan tiers,
+   check whether they appear in `entity_details` sampleValues for the relevant
+   column. If sampleValues is short or the sample may have missed real values,
+   run a `sql_execution` probe with the same warehouse connection id:
+   `sql_execution({connectionId, sql: "SELECT DISTINCT <col> FROM <ref> LIMIT 50"})`.
+4. If the candidate identifier still does not resolve, do one of:
+   - Use `sql_execution({connectionId, sql: "SELECT 1 FROM <ref> LIMIT 0"})`.
+     If it errors, the identifier is fictional.
+   - Wrap the identifier in `[unverified - from <rawPath>]` in the wiki body,
+     citing the exact raw path that mentioned it.
+   - When recording `emit_unmapped_fallback` with `no_physical_table`, include
+     the failing probe error in `clarification`.
+5. Never copy `<schema>.<table>` placeholder strings from these instructions
+   into output.
+
+## Decision tree
+
+For each card:
+1. Analyze `resolvedSql` + `resultMetadata`: identify base tables, aggregations, joins, filters, column types.
+2. **REQUIRED before any write**: call `sl_discover` for every candidate target source name. The response tells you whether the name is manifest-backed (`Type: table` or `Type: sql`). For manifest-backed names you MUST use the overlay shape (`name:` plus overlay fields such as `measures:`, `segments:`, `descriptions:`, `joins:`, `disable_joins:`, `column_overrides:`, and computed-only `columns:` entries with `expr` + `type`; no `sql:`, `table:`, `grain:`, or base-table `columns:`); the tool will reject a standalone write and you'll have wasted the call. If `sl_discover` returns nothing for the name, you can write a standalone source. Also call `sl_read_source` on existing sources you intend to extend so you don't duplicate measures.
+3. Include `rawPaths: ["cards/<id>.json"]` on every `sl_write_source`, `sl_edit_source`, and `wiki_write` call. If one artifact generalizes multiple near-duplicate cards, include each contributing card path and no unrelated cards.
+4. Decide:
+   - Simple aggregation on a table that already has a source → `sl_edit_source` to add a measure.
+   - Join between tables that should be linked in the SL graph → `sl_edit_source` to add a join.
+   - Complex derived SQL (CTEs, multi-layer aggregation, scoring models) → `sl_write_source` with `source_type: sql`. When the SQL projects/filters from a single manifest-backed base table, set `inherits_columns_from: <manifest_key>` so columns inherit type and description from the manifest - see `sl_capture` skill for the slim form. Use `sl_discover` to discover the manifest key from the table reference in the SQL (it accepts `MARTS.CONSIGNMENTS`, `ANALYTICS.MARTS.CONSIGNMENTS`, or `CONSIGNMENTS`).
+   - New base table not yet in the semantic layer → `sl_write_source` with `source_type: table`.
+   - Trivial query (`SELECT *`, simple `COUNT(*)` with no business logic) → do nothing; the runner will record this card as `action_type='skipped'`.
+   - Duplicate of an existing measure → same as trivial; do nothing for this card.
+
+**Manifest-only names need an overlay first.** If `sl_discover` shows a source name with `Type: table` but `sl_read_source` returns "Source not found", the source lives only in the schema manifest (no standalone overlay yet). `sl_edit_source` cannot edit manifest-only names, and a full standalone `sl_write_source` for that name would shadow manifest columns and joins. Bootstrap an overlay with `sl_write_source` using the overlay shape:
+
+```yaml
+name: <SOURCE_NAME>
+measures:
+  - name: <measure_name>
+    expr: "<expression>"
+```
+
+Overlay shape: `name:` plus any of `measures:`, `segments:`, `descriptions:`, `joins:`, `disable_joins:`, `exclude_columns:`, `column_overrides:`, or computed-only `columns:` entries with `expr` + `type`. Never include `sql:`, `table:`, `grain:`, or base-table `columns:` on a manifest-backed name — those would shadow the manifest's schema and drop its joins. Use `column_overrides:` for inherited column descriptions. Overlay `joins:` are merged additively with the manifest's joins (deduped by `to` + `on`); use `disable_joins: ["<on-clause>"]` to suppress a specific manifest join. After the overlay exists, use `sl_edit_source` for further tweaks. See `sl_capture` skill for the canonical overlay rule.
+
+**Join discovery:** When your card's SQL references warehouse tables (e.g. in `FROM` or `JOIN` clauses), call `sl_discover({ query: '<table>' })` before writing. The matching manifest entry's `name` is the value you use in `joins: [- to: <name>]` only when the card output exposes a local key that matches the target source grain (for example `account_id = mart_account_segments.account_id`). Do not declare a KTX join just because the card SQL joins that table internally. If the output only exposes display fields such as `account_name`, keep the SQL source self-contained or project the key before adding the join. Use `many_to_one` for FK-to-dimension joins, `one_to_many` for the reverse.
+
+**Hard rule on join columns (prevents broken joins):** For every join you declare, the local column on the left of `on:` MUST be (a) present in your source's projected output and (b) a key/ID column, never a display value. If the natural FK isn't in your SELECT, add it to SELECT before declaring the join. Joining `account_name = mart_account_segments.account_id` is always wrong - names are not identifiers and the equality produces zero matches. The validator rejects this with a "display value to identifier" error; the tool will refuse to save it. Add `account_id` to your SELECT and join on `account_id = mart_account_segments.account_id`, or omit the join entirely.
+
+## priorProvenance
+
+If the WU prompt includes a `priorProvenance` section for a card, it tells you what happened on prior ingest syncs. Treat it as advisory:
+- `action_type: source_created` on source X → prefer editing X with `sl_edit_source` rather than writing a new source.
+- `action_type: measure_added` on source X → you already contributed to X; add only measures that aren't present.
+- `action_type: subsumed` or `merged` → this card was folded into another source last time; unless its SQL has changed structurally, keep it subsumed (no new write).
+- `action_type: skipped` → last time we decided not to ingest this card; re-read the SQL and confirm the decision still holds. If the card now has non-trivial business logic, ingest it.
+
+## Deduplication
+
+Before writing, scan all cards in this WU for near-duplicate groups - cards whose `resolvedSql` shares the same CTEs, base tables, joins, and aggregation structure but differs only in:
+- Trailing filters (e.g. `date_trunc(week, date)` vs `date_trunc(month, date)`).
+- Minor `WHERE` clause variations.
+- Column aliases or output column subsets.
+- Aggregation granularity (daily vs weekly vs monthly).
+
+When you find a group of near-duplicates:
+1. Create ONE generalized source from the most comprehensive card in the group.
+2. Strip card-specific trailing filters from the SQL so the source covers all variants (e.g. keep daily grain instead of filtering to week/month).
+3. If each card had a distinct measure or filter, add them as separate measures on the single source.
+4. For all cards except the canonical one, do nothing - they'll be recorded as `action_type='skipped'` automatically by the runner.
+
+Do NOT merge cards with fundamentally different business logic, even if they share CTEs.
+
+## Pre-aggregation decomposition
+
+When a card's `resolvedSql` contains `GROUP BY` with aggregation functions (`SUM`, `COUNT`, `AVG`, …):
+
+1. **Detect**: simple aggregation on base tables/joins - `SELECT` with `GROUP BY`, no complex CTEs or window functions.
+2. **Decompose**: strip the `GROUP BY` and aggregation functions. Keep `FROM`, `JOIN`, and `WHERE` intact.
+3. **Expose row-level columns**: include the grouped-by columns AND the raw columns being aggregated (e.g. `money_out` instead of `SUM(money_out) AS total_money_out`).
+4. **Define aggregations as measures**: convert each aggregation into a KSL measure (e.g. `sum(money_out)`).
+5. **Add joins**: with FK columns now exposed, declare joins to dimension sources.
+
+Exception: keep the pre-aggregated SQL when the query involves multi-CTE pipelines, window functions, or recursive logic where decomposition would lose business logic.
+
+## SQL translation from raw native to KSL
+
+Every card carries a `resolvedSql` field. Check the staged card's `resolutionStatus` first:
+
+- `resolutionStatus: "resolved"` - `{{#N}}` references are inlined and `[[ ... ]]` optional clauses have been dropped locally. If the resolved SQL contains no other parameters the SQL is executable as-is. If the card had **required** (non-bracketed) `{{ var }}` placeholders, the SQL is prefixed with a placeholder-warning comment block listing every dummy substitution Metabase made - see "Step A" below.
+- `resolutionStatus: "fallback"` - Metabase failed to resolve. The SQL still contains `{{#N}}`, `{{#N-name}} alias`, `{{ var }}`, and `[[ ... ]]` syntax. Do the translation steps below before writing a source.
+
+### Step A - Handle dummy-substituted placeholders (resolved cards only)
+
+When a card has a required `{{ var }}` outside any `[[ ]]` block, the resolver substitutes a **dummy value** purely so Metabase's parser will accept the query. The resulting SQL is prefixed with a comment like:
+
+```sql
+-- PLACEHOLDER_WARNING: this SQL was extracted from a Metabase card with
+-- unbound template parameters. The placeholders below were substituted with DUMMY
+-- values to satisfy Metabase's parser - they DO NOT represent intended filters.
+-- Drop the corresponding clauses (or expose them as runtime SL filters) before
+-- persisting this SQL as a semantic-layer source.
+--   {{ auction_end }} (type=dimension, widget=date/all-options) → '2020-01-01~2020-12-31'
+--   {{ status }} (type=text) → 'placeholder'
+SELECT ...
+WHERE start_date >= '2020-01-01' AND start_date < '2021-01-01' AND status = 'placeholder'
+```
+
+For each listed placeholder: locate the WHERE clause(s) in the SQL that reference the dummy literal and **drop them**, then strip the warning comment. SL chat-time filters compose narrowing predicates dynamically, so the source should represent the unfiltered dataset.
+
+For `fallback` cards, dropping is simpler - the SQL still has the `[[ ... ]]` brackets and `{{ var }}` placeholders intact:
+
+```sql
+-- before:
+WHERE 1=1
+  [[AND {{ auction_end }} ]]
+  [[AND status = {{ status }} ]]
+
+-- after:
+WHERE 1=1
+```
+
+### Step B - Inline `{{#N}}` references (fallback cards only)
+
+Resolved cards already have `{{#N}}` inlined for you. For `fallback` cards, each `{{#N}}` (or `{{#N-some-slug}}`) in the SQL refers to another card's `resolvedSql`. The referenced card is in the WU's `rawFiles` or `dependencyPaths`. Read it with `read_raw_file`, then inline its SQL.
+
+If the reference has an alias (`from {{#5996-listing-interactions}} tb`), the **outer** SQL probably uses that alias (`select tb.* ...`, `tb.column_name`, etc.). When you inline, you must EITHER:
+
+1. **Pick a single base table inside the inlined SQL and rename its alias to the outer alias.** Useful when the inlined card is `SELECT * FROM listings JOIN ...` - set the LISTINGS alias to `tb` and `tb.*` keeps working in the outer query.
+2. **Replace the outer alias references with explicit columns from the inlined SQL.** Useful when the inlined card has multiple JOINs and `tb.*` is ambiguous.
+
+Never leave the outer alias dangling: after inlining, **grep your SQL for the outer alias name and rewrite or remove every reference**. A leftover `tb.*` with no `tb` table is the most common failure mode here.
+
+### Step C - Inlining cleanup checklist
+
+After Steps A and B, your SQL must:
+
+- Contain no placeholder-warning comment, no `{{`, `}}`, `[[`, or `]]` characters anywhere.
+- Reference no aliases that aren't defined inside the SQL itself.
+- Be valid as a standalone subquery (the validator runs `SELECT * FROM (your_sql) LIMIT 1`).
+
+If `resolutionStatus: "fallback"` and the SQL is still complex enough that you can't confidently translate it, **skip the card** rather than writing broken SQL. Call `emit_unmapped_fallback` with the staged card path as `rawPath`, `reason: "parse_error"`, `clarification: "metabase_sql_untranslated"`, and `fallback: "flagged"`.
+
+## Join-graph connectivity
+
+For `source_type: table`:
+- Use FK columns (`semantic_type: type/FK`) to declare `many_to_one` joins to dimension sources.
+- Match column names ending in `_id` against existing sources' grain columns.
+
+For `source_type: sql`:
+- The validator parses your SQL and rejects the write when a referenced manifest table has a viable projected local key but no declared `joins:` entry. Add the join only after confirming the output key and target grain match.
+- If `sl_discover` resolves the table, it is not outside the manifest. Do not write an `unmapped-table-*` fallback for resolved `orbit_raw`, `mart`, or other manifest-backed sources just because they appear inside card SQL.
+- If `sl_discover` cannot resolve a referenced table at all, write a single-line `wiki_write` with key `unmapped-table-<table_name>` and `rawPaths: ["cards/<id>.json"]` so the gap is documented, then call `emit_unmapped_fallback` with the staged card path as `rawPath`, `reason: "missing_target_table"`, `tableRef: "<table_name>"`, and `fallback: "wiki_only"`. Do not use this fallback if `sl_discover` resolved the table/source.
+
+Joins on manifest-backed names compose: the manifest's joins are inherited automatically, and any overlay `joins:` are merged on top (deduped by `to` + `on`). Use `disable_joins: ["<on-clause>"]` in the overlay to suppress a specific manifest join. If `sl_discover` shows a manifest-backed source with `Joins: 0` and the warehouse FK metadata is genuinely absent, declaring application-level joins via the overlay is fair game - bootstrap with `sl_write_source` (overlay shape above), then refine via `sl_edit_source`.
+
+## Cross-card references (`{{#N}}`)
+
+Resolved cards (`resolutionStatus: "resolved"`) have these inlined for you. Unresolved cards (`resolutionStatus: "fallback"`) need manual handling - see "SQL translation from raw native to KSL" above.
+
+## Provenance markers
+
+Every SL source and wiki page you write carries HTML-comment provenance tags pointing to the `cards/<id>.json` files they derive from:
+
+```yaml
+# <!-- from: raw-sources/<connId>/metabase/<syncId>/cards/7.json -->
+name: orders
+...
+```
+
+If a source is derived from multiple cards (e.g. a generalized source for a near-duplicate group), emit one tag per contributing card.
+
+## Quality standards
+
+Source definitions must follow ktx-sl YAML conventions:
+- `source_type`: `"table"` (physical table/view) or `"sql"` (arbitrary SQL / derived view).
+- `table`: required when `source_type: "table"` (e.g. `"public.orders"`).
+- `sql`: required when `source_type: "sql"`.
+- `grain`: what one row represents (e.g. `[id]`, `[customer_id, product_id]`).
+- `columns`: all columns with correct types (`string`, `number`, `time`, `boolean`).
+- Time columns: mark with `role: time`.
+- `joins`: use correct `relationship` types (`many_to_one` for FK→PK, `one_to_many` for reverse).
+- `joins.on`: `local_column = TARGET_SOURCE.target_column` - the right side MUST include the target source name.
+- `measures.expr`: aggregation expression (e.g. `"sum(amount)"`); optional `filter` for business rules; required `description`.
+
+Measure naming: descriptive `snake_case` (e.g. `total_revenue`, `avg_order_value`).
+
+## Rules
+
+- Prefer adding measures to existing sources over creating new ones.
+- Before editing, always `sl_read_source` the source to check for existing measures.
+- Don't duplicate measures (same aggregation on the same column).
+- If two measures differ only by a filter (e.g. `revenue` vs `paid_revenue`), they are distinct.
+- Use the card's `name` + `description` to write meaningful measure descriptions.
+- When multiple cards in a WU are near-duplicates, create ONE generalized source; the runner will skip the rest automatically.
+- Process every card in the WU - don't stop early.

package/node_modules/@ktx/context/skills/metricflow_ingest/SKILL.md

@@ -0,0 +1,313 @@
+---
+name: metricflow_ingest
+description: Map a MetricFlow semantic_model or metric into KTX semantic layer sources. Covers the MetricFlow to KTX primitive table, `extends:` inheritance flattening, metric-type handling (simple / derived / ratio / cumulative / conversion), `model: ref('x')` resolution, and four worked examples. Load when the turn contains `.yml`/`.yaml` files with top-level `semantic_models:` or `metrics:`.
+callers: [memory_agent]
+---
+
+# MetricFlow to KTX Semantic Layer
+
+A MetricFlow `semantic_model` maps to an SL source; MetricFlow `measures` map to KTX measures; MetricFlow `entities` map to KTX `joins`; MetricFlow `metrics` (top-level) map to KTX measures OR to cross-model derived measures. Files in one WorkUnit are ALWAYS part of the same logical entity (a connected component, possibly spanning `extends:` + cross-model metric refs). Flatten inheritance and cross-file references at write time.
+
+## Mapping table
+
+| MetricFlow | KTX form | Notes |
+|---|---|---|
+| `semantic_model: X { model: ref('t') }` with measures + dimensions | **Overlay** at `<connId>/X.yaml` with `measures`, computed-only `columns`, `column_overrides`, `joins` | The `model:` ref resolves to a manifest table. |
+| `semantic_model: X { model: source('s','t') }` | **Overlay** at `<connId>/X.yaml` over table `t`. | Same shape; `source()` still resolves to a physical table. |
+| `semantic_model: X { model: <literal> }` with no manifest entry | **Standalone** with explicit `sql:`, `grain:`, `columns:` | Happens when the dbt manifest isn't available. |
+| `semantic_model: Y { extends: X }` | **Merge** Y's measures/dimensions/entities into X's overlay, or write a single overlay named for the most-derived child (Y) containing both X's and Y's primitives | Do not emit a second overlay for X - flatten. |
+| `measures: [{ name, agg, expr }]` | `measures: [{ name, expr: "<agg>(<expr>)" }]` | Aggregation inlined. `agg: count_distinct` → `count(distinct ...)`. |
+| `entities: [{ name, type: primary }]` | `grain: [<entity_name-or-expr>]` on the overlay/standalone | Primary/unique entities drive grain. |
+| `entities: [{ name, type: foreign }]` | `joins:` entry joining to the primary-entity's semantic_model | Only when a matching primary is discoverable. |
+| `metrics: [{ type: simple, type_params: { measure: X } }]` | If the base measure is labeled/described by the metric: in-place edit to the existing measure. Otherwise leave as-is. | Same-name metrics can absorb metadata. |
+| `metrics: [{ type: simple, filter: <jinja> }]` | **New measure** on the same source, with the filter translated to SQL and attached via `filter:` | Translate Jinja `{{ Dimension('x__y') }}` to the column name `y`. |
+| `metrics: [{ type: derived, type_params: { expr, metrics } }]` | **Derived measure** on whichever source owns the referenced measures, with `expr:` referencing measure names | If the metric spans models, still write it once on the source owning the "primary" measure (the one the agent judges most central). Mention the cross-model chain in the description. |
+| `metrics: [{ type: ratio, type_params: { numerator, denominator } }]` | Same as derived; `expr: "numerator / NULLIF(denominator, 0)"` if no explicit expr | Safe-division by default. |
+| `metrics: [{ type: cumulative, type_params: { window, grain_to_date } }]` | **Standalone** source with a window-function SQL; reference the resulting column as a normal measure | KTX SL has no first-class cumulative primitive (spec Non-goals). |
+| `metrics: [{ type: conversion }]` | **Flag for human** - do NOT write. Emit a wiki note describing the intended semantics. | No KTX equivalent in v1. |
+| Metric not mappable | Wiki page `<metric_name>-definition.md` with the full YAML body quoted | Capture the intent even if we can't emit SL. |
+
+Type map: MetricFlow `time` to KTX `time`; `categorical` to `string`; `number` to `number`; `boolean` to `boolean`. Follow `expr` over `name` when both differ - `expr` is the physical column.
+
+Verify each MetricFlow model source table with entity_details before producing
+the corresponding sl_write_source.
+
+## Identifier Verification Protocol
+
+Before writing a wiki page or SL source on any topic:
+
+1. `discover_data({query: "<topic>"})` - see what wikis, SL sources, and raw
+   tables already exist. Prefer updating existing pages over creating new ones.
+
+Before emitting any `schema.table` or `schema.table.column` into a wiki body,
+SL source, `tables:` frontmatter, `sl_refs`, or `emit_unmapped_fallback`:
+
+2. `entity_details({connectionId, targets: [{display: "<identifier>"}]})` -
+   confirm the identifier resolves; inspect native types, FK/PK, and
+   sampleValues.
+3. For literal values from the source, such as status codes or plan tiers,
+   check whether they appear in `entity_details` sampleValues for the relevant
+   column. If sampleValues is short or the sample may have missed real values,
+   run a `sql_execution` probe with the same warehouse connection id:
+   `sql_execution({connectionId, sql: "SELECT DISTINCT <col> FROM <ref> LIMIT 50"})`.
+4. If the candidate identifier still does not resolve, do one of:
+   - Use `sql_execution({connectionId, sql: "SELECT 1 FROM <ref> LIMIT 0"})`.
+     If it errors, the identifier is fictional.
+   - Wrap the identifier in `[unverified - from <rawPath>]` in the wiki body,
+     citing the exact raw path that mentioned it.
+   - When recording `emit_unmapped_fallback` with `no_physical_table`, include
+     the failing probe error in `clarification`.
+5. Never copy `<schema>.<table>` placeholder strings from these instructions
+   into output.
+
+## Flattening `extends:`
+
+Within one WorkUnit, multiple semantic_models linked by `extends:` are guaranteed to be present (the chunker groups them). Resolve inheritance **before** writing:
+
+1. Start with the most-derived child (the one that no other semantic_model extends).
+2. Walk the `extends:` chain upward, accumulating measures, dimensions, entities.
+3. Write ONE overlay/standalone, named for the most-derived child's SL-appropriate name (not the base).
+4. Parents that lack their own distinctive content should NOT get a separate overlay. If a parent has unique measures a child doesn't inherit, consider whether the base is used elsewhere - if yes, write both; if no, still one overlay.
+5. Measure/dimension name collisions: child wins, but note the overridden parent in the overlay's description or in a sibling wiki page.
+
+The spec's worked example has `orders`, `orders_ext` (extends orders), and `metrics/orders_final.yml` (defines `revenue` referencing both). The right output is ONE overlay named `orders_ext` (or `orders` if the team's naming favors the base) containing `order_count`, `gross_amount`, `refund_amount`, and a derived `revenue` measure. Provenance tags point to all three source files.
+
+## `model:` ref resolution
+
+The `model:` field on a semantic_model is a string like `ref('table_name')`, `source('src','table_name')`, or a literal. Resolve:
+
+- `ref('x')` → table name `x`. Verify via `sl_discover(x)`.
+- `source('s','t')` → table name `t`. Verify via `sl_discover(t)`.
+- Literal (no `ref(...)` / `source(...)`) → treat as the table name directly.
+
+If `sl_discover` errors because no such table exists, use `discover_data` and
+`entity_details` to find the warehouse target. If a SQL probe is still needed,
+call `sql_execution` with the same warehouse connection id, for example:
+`sql_execution({connectionId: "warehouse", sql: "SELECT 1 FROM analytics.orders LIMIT 0"})`.
+**Never invent column names** - every column in computed `columns:`, `column_overrides:`, `grain:`, and
+`sql:` must be sourced from raw files, `entity_details`, or a successful SQL
+probe.
+
+After every `sl_write_source`, call `sl_validate`. The warehouse will reject invented columns with `Unrecognized name: <name>` - treat as a hard failure and re-read the schema.
+
+## Cumulative metrics - sql-standalone fallback
+
+KTX SL has no first-class `window:` or `grain_to_date:` primitive in v1 (spec Non-goals). Translate a MetricFlow cumulative metric to a standalone SL source with a window-function SQL:
+
+```yaml
+# MetricFlow input:
+metrics:
+  - name: cum_revenue_7d
+    type: cumulative
+    type_params:
+      measure: gross_amount
+      window: 7 days
+```
+
+```yaml
+# KTX standalone output:
+name: cum_revenue_7d
+source_type: sql
+sql: |
+  SELECT
+    ordered_at,
+    SUM(amount) OVER (ORDER BY ordered_at RANGE BETWEEN INTERVAL '7' DAY PRECEDING AND CURRENT ROW) AS cum_revenue_7d,
+    order_id
+  FROM analytics.orders
+grain: [order_id]
+columns:
+  - {name: ordered_at, type: time, role: time}
+  - {name: cum_revenue_7d, type: number}
+  - {name: order_id, type: string}
+measures:
+  - {name: cum_revenue_7d, expr: "max(cum_revenue_7d)"}
+```
+
+Pick the time column based on the semantic_model's `defaults.agg_time_dimension` (e.g. `ordered_at`). If the MetricFlow config omits it, probe the base table for time-typed columns and choose the most obvious. After writing the standalone SQL source, call `emit_unmapped_fallback` with `rawPath` set to the MetricFlow file path, `reason: "cumulative_metric_unsupported"`, and `fallback: "sql_standalone"`.
+
+## Conversion metrics - flag for human
+
+```yaml
+metrics:
+  - name: signup_to_first_order
+    type: conversion
+    type_params:
+      conversion_type_params:
+        entity: customer
+        base_measure: signup_count
+        conversion_measure: first_order_count
+        window: 30 days
+```
+
+Do NOT emit SL for this. Instead:
+- Write a wiki page at `wiki/global/<metric_name>-intent.md` quoting the full YAML body and a one-line explanation of the intended semantics (base event → conversion event within window).
+- Call `emit_unmapped_fallback` with `rawPath` set to the MetricFlow file path, `reason: "conversion_metric_unsupported"`, and `fallback: "flagged"`.
+
+When KTX SL gains conversion primitives, re-ingesting will find the prior wiki note (via `priorProvenance`) and replace it with an SL source.
+
+## Provenance markers
+
+Every overlay/standalone/wiki page emitted from a MetricFlow source carries HTML-comment provenance tags. When one overlay derives from multiple files (e.g. an extends chain), emit one tag per contributing file:
+
+```yaml
+# <!-- from: raw-sources/conn-1/metricflow/<syncId>/models/orders.yml#L1-20 -->
+# <!-- from: raw-sources/conn-1/metricflow/<syncId>/models/orders_ext.yml#L1-12 -->
+# <!-- from: raw-sources/conn-1/metricflow/<syncId>/metrics/orders_final.yml#L1-10 -->
+name: orders_ext
+...
+```
+
+Line ranges (`#L<start>-<end>`) point to the exact YAML span within the file (the `semantic_models:` entry for its own `name`). Use `read_raw_span` to identify those ranges before writing.
+
+## Example 1 - single semantic_model to overlay
+
+```yaml
+# MetricFlow:
+semantic_models:
+  - name: orders
+    model: ref('orders')
+    entities:
+      - {name: order_id, type: primary}
+    measures:
+      - {name: order_count, agg: count, expr: order_id}
+      - {name: gross_amount, agg: sum, expr: amount}
+```
+
+```yaml
+# KTX overlay at <connId>/orders.yaml:
+# <!-- from: raw-sources/.../models/orders.yml#L1-10 -->
+name: orders
+descriptions:
+  user: Order fact table.
+measures:
+  - {name: order_count, expr: "count(order_id)"}
+  - {name: gross_amount, expr: "sum(amount)"}
+grain: [order_id]
+```
+
+## Example 2 - extends chain → one flattened overlay
+
+```yaml
+# MetricFlow:
+# models/orders.yml
+semantic_models:
+  - name: orders
+    model: ref('orders')
+    measures:
+      - {name: order_count, agg: count, expr: order_id}
+      - {name: gross_amount, agg: sum, expr: amount}
+
+# models/orders_ext.yml
+semantic_models:
+  - name: orders_ext
+    model: ref('orders_ext')
+    extends: orders
+    measures:
+      - {name: refund_amount, agg: sum, expr: refund_amt}
+
+# metrics/orders_final.yml
+metrics:
+  - name: revenue
+    type: derived
+    type_params:
+      expr: gross_amount - refund_amount
+      metrics:
+        - {name: gross_amount}
+        - {name: refund_amount}
+```
+
+```yaml
+# KTX overlay at <connId>/orders_ext.yaml (one file; inheritance flattened):
+# <!-- from: raw-sources/.../models/orders.yml#L1-10 -->
+# <!-- from: raw-sources/.../models/orders_ext.yml#L1-8 -->
+# <!-- from: raw-sources/.../metrics/orders_final.yml#L1-10 -->
+name: orders_ext
+descriptions:
+  user: Extended order fact including refund handling; `revenue` = gross - refund.
+measures:
+  - {name: order_count, expr: "count(order_id)"}
+  - {name: gross_amount, expr: "sum(amount)"}
+  - {name: refund_amount, expr: "sum(refund_amt)"}
+  - {name: revenue, expr: "gross_amount - refund_amount"}
+grain: [order_id]
+```
+
+## Example 3 - derived metric spanning two semantic_models
+
+```yaml
+# models/sales.yml
+semantic_models:
+  - name: sales
+    model: ref('sales')
+    measures:
+      - {name: revenue, agg: sum, expr: revenue_cents}
+# models/costs.yml
+semantic_models:
+  - name: costs
+    model: ref('costs')
+    measures:
+      - {name: cost, agg: sum, expr: cost_cents}
+# metrics/margin.yml
+metrics:
+  - name: margin
+    type: derived
+    type_params:
+      expr: revenue - cost
+      metrics: [{name: revenue}, {name: cost}]
+```
+
+Because the WorkUnit bundles all three files (cross-component union via the metric), write the derived measure on ONE of the two sources - pick the source whose domain "owns" the metric (here, `sales` - margin is inherently a sales metric). Cross-source references aren't native in KTX SL; treat the metric's operands as already-resolvable in the target source's query context OR emit a standalone SQL that joins the two tables:
+
+```yaml
+# <connId>/sales.yaml
+# <!-- from: .../models/sales.yml#L1-8 -->
+# <!-- from: .../models/costs.yml#L1-8 -->
+# <!-- from: .../metrics/margin.yml#L1-8 -->
+name: sales
+measures:
+  - {name: revenue, expr: "sum(revenue_cents)"}
+```
+
+```yaml
+# <connId>/margin.yaml - standalone because it spans two tables
+# <!-- from: .../models/sales.yml#L1-8 -->
+# <!-- from: .../models/costs.yml#L1-8 -->
+# <!-- from: .../metrics/margin.yml#L1-8 -->
+name: margin
+source_type: sql
+sql: |
+  SELECT s.period_id, s.revenue_cents, COALESCE(c.cost_cents, 0) AS cost_cents
+  FROM analytics.sales s
+  LEFT JOIN analytics.costs c ON c.period_id = s.period_id
+grain: [period_id]
+columns:
+  - {name: period_id, type: string}
+  - {name: revenue_cents, type: number}
+  - {name: cost_cents, type: number}
+measures:
+  - {name: revenue, expr: "sum(revenue_cents)"}
+  - {name: cost, expr: "sum(cost_cents)"}
+  - {name: margin, expr: "sum(revenue_cents) - sum(cost_cents)"}
+```
+
+Also write a wiki page at `wiki/global/margin-metric.md` explaining the cross-source origin.
+
+## Example 4 - filtered metric creates a new measure
+
+```yaml
+metrics:
+  - name: paid_order_count
+    type: simple
+    type_params:
+      measure: order_count
+    filter: "{{ Dimension('orders__status') }} = 'paid'"
+```
+
+```yaml
+# <connId>/orders.yaml
+measures:
+  - {name: order_count, expr: "count(order_id)"}
+  - {name: paid_order_count, expr: "count(order_id)", filter: "status = 'paid'"}
+```
+
+Translate `{{ Dimension('orders__status') }}` to the bare column name `status` (the table alias prefix is implicit within the SL source's scope).