modscape 1.3.0 → 2.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.ja.md CHANGED
@@ -92,6 +92,7 @@ YAMLのルートレベル構造は以下の通りです:
92
92
  domains – 関連テーブルをまとめるビジュアルコンテナ
93
93
  tables – 3階層メタデータを持つエンティティ定義
94
94
  relationships – テーブル間のERカーディナリティ
95
+ lineage – データの流れ / 変換パス
95
96
  annotations – キャンバス上のスティッキーノート・吹き出し
96
97
  layout – 全座標データ(tables/domains の中に x/y を書いてはいけない)
97
98
  ```
@@ -104,7 +105,7 @@ domains:
104
105
  name: "主要売上"
105
106
  description: "営業チームのトランザクションデータ。" # 任意
106
107
  color: "rgba(59, 130, 246, 0.1)" # 背景色
107
- tables: [orders, dim_customers]
108
+ tables: [orders, dim_customers] # 論理的な所属リスト
108
109
  isLocked: false # true にするとキャンバスでの誤ドラッグを防止
109
110
  ```
110
111
 
@@ -130,11 +131,6 @@ tables:
130
131
  businessDefinitions:
131
132
  revenue: "割引・返品後の純売上"
132
133
 
133
- lineage: # mart/集計テーブルのみに定義
134
- upstream:
135
- - fct_sales
136
- - dim_dates
137
-
138
134
  implementation: # 任意 – AIコード生成へのヒント
139
135
  materialization: incremental # table | view | incremental | ephemeral
140
136
  incremental_strategy: merge # merge | append | delete+insert
@@ -165,23 +161,22 @@ tables:
165
161
  type: "BIGINT"
166
162
  constraints: [NOT NULL]
167
163
 
168
- sampleData: # 2次元配列。先頭行 = カラムID
169
- - [order_id, amount, status]
164
+ sampleData: # 実数値の2次元配列
170
165
  - [1001, 50.0, "COMPLETED"]
171
166
  - [1002, 120.5, "PENDING"]
172
167
  ```
173
168
 
174
- **テーブルタイプと `appearance.type` の使い分け:**
169
+ ### Data Lineage(データリネージ)
170
+
171
+ ルートレベルの `lineage` セクションでテーブル間のデータの流れ(どのソースからどの集計テーブルが作られるか)を定義します。リネージモードではアニメーション付きの点線矢印として表示されます。
175
172
 
176
- | type | 用途 |
177
- |------|------|
178
- | `fact` | 取引・イベント・測定値 |
179
- | `dimension` | エンティティ・マスタ・参照リスト |
180
- | `mart` | 集計・消費者向けテーブル(`lineage.upstream` を必ず定義) |
181
- | `hub` | Data Vault のビジネスキー |
182
- | `link` | Data Vault のハブ間結合・トランザクション |
183
- | `satellite` | Data Vault のハブに紐づく履歴属性 |
184
- | `table` | 汎用 |
173
+ ```yaml
174
+ lineage:
175
+ - from: fct_orders # ソーステーブル ID
176
+ to: mart_revenue # 派生テーブル ID
177
+ - from: dim_dates
178
+ to: mart_revenue
179
+ ```
185
180
 
186
181
  ### Relationships(リレーションシップ)
187
182
 
@@ -196,7 +191,7 @@ relationships:
196
191
  type: one-to-many # one-to-one | one-to-many | many-to-one | many-to-many
197
192
  ```
198
193
 
199
- > **データリネージ**は `lineage.upstream` で定義し、リネージモードでアニメーション矢印として表示されます。`relationships` に重複して記載しないでください。
194
+ > **ER関係** vs **リネージ**: 構造的な結合(外部キーなど)には `relationships` を、データの加工・変換の流れには `lineage` を使用してください。両方に同じ接続を記述しないでください。
200
195
 
201
196
  ### Annotations(アノテーション)
202
197
 
@@ -270,7 +265,6 @@ modscape export ./models -o docs/ARCHITECTURE.md
270
265
 
271
266
  Modscape は以下の素晴らしいオープンソースプロジェクトによって支えられています:
272
267
 
273
- - [React Flow](https://reactflow.dev/) - インタラクティブなグラフ UI フレームワーク。
274
268
  - [CodeMirror 6](https://codemirror.net/) - 次世代のウェブベース・コードエディタ。
275
269
  - [Dagre](https://github.com/dagrejs/dagre) - 階層型グラフ・レイアウトエンジン。
276
270
  - [Lucide React](https://lucide.dev/) - シンプルで美しいアイコンセット。
package/README.md CHANGED
@@ -95,6 +95,7 @@ Modscape uses a schema designed for data analysis contexts. The full YAML struct
95
95
  domains – visual containers grouping related tables
96
96
  tables – entity definitions with tri-layer metadata
97
97
  relationships – ER cardinality between tables
98
+ lineage – data flow / transformation paths
98
99
  annotations – sticky notes / callouts on the canvas
99
100
  layout – ALL coordinate data (never put x/y inside tables or domains)
100
101
  ```
@@ -107,7 +108,7 @@ domains:
107
108
  name: "Core Sales"
108
109
  description: "Transactional data for the sales team." # optional
109
110
  color: "rgba(59, 130, 246, 0.1)" # background fill
110
- tables: [orders, dim_customers]
111
+ tables: [orders, dim_customers] # logical membership
111
112
  isLocked: false # prevent accidental drag when true
112
113
  ```
113
114
 
@@ -133,11 +134,6 @@ tables:
133
134
  businessDefinitions:
134
135
  revenue: "Net revenue after discounts"
135
136
 
136
- lineage: # for mart/aggregated tables only
137
- upstream:
138
- - fct_sales
139
- - dim_dates
140
-
141
137
  implementation: # optional – hints for AI code generation
142
138
  materialization: incremental # table | view | incremental | ephemeral
143
139
  incremental_strategy: merge # merge | append | delete+insert
@@ -168,12 +164,23 @@ tables:
168
164
  type: "BIGINT"
169
165
  constraints: [NOT NULL]
170
166
 
171
- sampleData: # 2D array; first row = column IDs
172
- - [order_id, amount, status]
167
+ sampleData: # 2D array of realistic values
173
168
  - [1001, 50.0, "COMPLETED"]
174
169
  - [1002, 120.5, "PENDING"]
175
170
  ```
176
171
 
172
+ ### Data Lineage
173
+
174
+ Top-level `lineage` section declares data flow between tables (which source tables feed which derived tables). This is rendered as dashed arrows in **Lineage Mode**.
175
+
176
+ ```yaml
177
+ lineage:
178
+ - from: fct_orders # source table ID
179
+ to: mart_revenue # derived table ID
180
+ - from: dim_dates
181
+ to: mart_revenue
182
+ ```
183
+
177
184
  ### Relationships
178
185
 
179
186
  ```yaml
@@ -187,7 +194,7 @@ relationships:
187
194
  type: one-to-many # one-to-one | one-to-many | many-to-one | many-to-many
188
195
  ```
189
196
 
190
- > **Data Lineage** connections are driven by `lineage.upstream` and rendered as animated arrows in Lineage Mode. Do **not** duplicate them as `relationships` entries.
197
+ > **ER Relationships** vs **Lineage**: Use `relationships` for structural joins (FKs) and `lineage` for data flow (transformations). Do not duplicate them.
191
198
 
192
199
  ### Annotations
193
200
 
@@ -261,7 +268,6 @@ modscape export ./models -o docs/ARCHITECTURE.md
261
268
 
262
269
  Modscape is made possible by these incredible open-source projects:
263
270
 
264
- - [React Flow](https://reactflow.dev/) - Interactive node-based UI framework.
265
271
  - [CodeMirror 6](https://codemirror.net/) - Next-generation code editor for the web.
266
272
  - [Dagre](https://github.com/dagrejs/dagre) - Directed graph layout engine.
267
273
  - [Lucide React](https://lucide.dev/) - Beautifully simple pixel-perfect icons.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "modscape",
3
- "version": "1.3.0",
3
+ "version": "2.0.1",
4
4
  "description": "Modscape: A YAML-driven data modeling visualizer CLI",
5
5
  "repository": {
6
6
  "type": "git",
@@ -29,6 +29,7 @@
29
29
  "@inquirer/prompts": "^8.3.0",
30
30
  "chokidar": "^5.0.0",
31
31
  "commander": "^14.0.3",
32
+ "dagre": "^0.8.5",
32
33
  "express": "^5.2.1",
33
34
  "js-yaml": "^4.1.1",
34
35
  "open": "^11.0.0",
package/src/export.js CHANGED
@@ -67,16 +67,13 @@ function generateMermaidLineage(schema) {
67
67
  let mermaid = 'graph TD\n';
68
68
  let hasLineage = false;
69
69
 
70
- schema.tables.forEach(table => {
71
- if (table.lineage?.upstream && table.lineage.upstream.length > 0) {
72
- hasLineage = true;
73
- const targetName = sanitize(table.name);
74
- table.lineage.upstream.forEach(upId => {
75
- const sourceTable = schema.tables.find(t => t.id === upId);
76
- const sourceName = sanitize(sourceTable?.name || upId);
77
- mermaid += ` ${sourceName} --> ${targetName}\n`;
78
- });
79
- }
70
+ (schema.lineage || []).forEach(edge => {
71
+ hasLineage = true;
72
+ const sourceTable = schema.tables.find(t => t.id === edge.from);
73
+ const targetTable = schema.tables.find(t => t.id === edge.to);
74
+ const sourceName = sanitize(sourceTable?.name || edge.from);
75
+ const targetName = sanitize(targetTable?.name || edge.to);
76
+ mermaid += ` ${sourceName} --> ${targetName}\n`;
80
77
  });
81
78
 
82
79
  return hasLineage ? mermaid : null;
@@ -197,10 +194,11 @@ export function generateMarkdown(schema, modelName) {
197
194
  }
198
195
 
199
196
  // Lineage
200
- if (table.lineage?.upstream?.length > 0) {
201
- const upstreamNames = table.lineage.upstream.map(upId => {
202
- const t = schema.tables.find(t => t.id === upId);
203
- return t ? t.name : upId;
197
+ const upstreamEdges = (schema.lineage || []).filter(e => e.to === table.id);
198
+ if (upstreamEdges.length > 0) {
199
+ const upstreamNames = upstreamEdges.map(e => {
200
+ const t = schema.tables.find(t => t.id === e.from);
201
+ return t ? t.name : e.from;
204
202
  });
205
203
  md += `**Upstream**: ${upstreamNames.map(n => `\`${n}\``).join(' → ')}\n\n`;
206
204
  }
package/src/import-dbt.js CHANGED
@@ -34,6 +34,7 @@ export async function importDbt(projectDir, options) {
34
34
  console.log(` 🔍 Parsing dbt manifest: ${manifestPath}`);
35
35
 
36
36
  const tables = [];
37
+ const lineage = [];
37
38
  const domainsMap = new Map();
38
39
  const tableSplitKeyMap = new Map();
39
40
 
@@ -66,7 +67,6 @@ export async function importDbt(projectDir, options) {
66
67
  appearance: { type: 'table' },
67
68
  conceptual: { description: node.description || '' },
68
69
  columns,
69
- lineage: { upstream: [] }
70
70
  };
71
71
 
72
72
  tables.push(tableEntry);
@@ -92,13 +92,12 @@ export async function importDbt(projectDir, options) {
92
92
  }
93
93
 
94
94
  // lineage
95
- for (const [uniqueId, node] of Object.entries(allNodes)) {
95
+ for (const [, node] of Object.entries(allNodes)) {
96
96
  if (!['model', 'seed', 'snapshot', 'source'].includes(node.resource_type)) continue;
97
- const tableEntry = tables.find(t => t.id === node.unique_id);
98
- if (tableEntry && node.depends_on?.nodes) {
97
+ if (node.depends_on?.nodes) {
99
98
  for (const upstreamId of node.depends_on.nodes) {
100
99
  if (allNodes[upstreamId]) {
101
- tableEntry.lineage.upstream.push(upstreamId);
100
+ lineage.push({ from: upstreamId, to: node.unique_id });
102
101
  }
103
102
  }
104
103
  }
@@ -122,11 +121,10 @@ export async function importDbt(projectDir, options) {
122
121
  const tableIds = new Set(splitTables.map(t => t.id));
123
122
  let internal = 0;
124
123
  let external = 0;
125
- for (const table of splitTables) {
126
- for (const upstreamId of table.lineage?.upstream || []) {
127
- if (tableIds.has(upstreamId)) internal++;
128
- else external++;
129
- }
124
+ for (const edge of lineage) {
125
+ if (!tableIds.has(edge.to)) continue;
126
+ if (tableIds.has(edge.from)) internal++;
127
+ else external++;
130
128
  }
131
129
  const total = internal + external;
132
130
  const rate = total > 0 ? Math.round(internal / total * 100) : 100;
@@ -145,9 +143,12 @@ export async function importDbt(projectDir, options) {
145
143
  tables: d.tables.filter(tid => splitTables.some(t => t.id === tid))
146
144
  }));
147
145
 
146
+ const splitTableIds = new Set(splitTables.map(t => t.id));
147
+ const splitLineage = lineage.filter(e => splitTableIds.has(e.from) || splitTableIds.has(e.to));
148
148
  const outputModel = {
149
149
  tables: splitTables,
150
150
  relationships: [],
151
+ lineage: splitLineage,
151
152
  domains: splitDomains
152
153
  };
153
154
 
@@ -167,6 +168,7 @@ export async function importDbt(projectDir, options) {
167
168
  const outputModel = {
168
169
  tables,
169
170
  relationships: [],
171
+ lineage,
170
172
  domains: Array.from(domainsMap.values())
171
173
  };
172
174
 
package/src/index.js CHANGED
@@ -10,6 +10,7 @@ import { exportModel } from './export.js';
10
10
  import { createModel } from './create.js';
11
11
  import { importDbt } from './import-dbt.js';
12
12
  import { syncDbt } from './sync-dbt.js';
13
+ import { applyLayout } from './layout.js';
13
14
  import { createRequire } from 'module';
14
15
  import { mergeModels } from './merge.js';
15
16
 
@@ -104,4 +105,13 @@ program
104
105
  mergeModels(paths, options);
105
106
  });
106
107
 
108
+ program
109
+ .command('layout')
110
+ .description('Perform automatic layout calculation and update the YAML file')
111
+ .argument('<path>', 'path to the YAML model file')
112
+ .option('-o, --output <path>', 'output file path (defaults to overwriting input)')
113
+ .action((path, options) => {
114
+ applyLayout(path, options);
115
+ });
116
+
107
117
  program.parse();
package/src/layout.js ADDED
@@ -0,0 +1,151 @@
1
+ import fs from 'fs';
2
+ import path from 'path';
3
+ import yaml from 'js-yaml';
4
+ import dagre from 'dagre';
5
+
6
+ /**
7
+ * CLI-based layout engine for Modscape.
8
+ * Replicates the logic from the visualizer to ensure consistency.
9
+ */
10
+ export function applyLayout(inputPath, options = {}) {
11
+ const absolutePath = path.resolve(process.cwd(), inputPath);
12
+ if (!fs.existsSync(absolutePath)) {
13
+ console.error(` ❌ File not found: ${inputPath}`);
14
+ return;
15
+ }
16
+
17
+ const raw = fs.readFileSync(absolutePath, 'utf8');
18
+ let schema;
19
+ try {
20
+ schema = yaml.load(raw);
21
+ } catch (e) {
22
+ console.error(` ❌ Failed to parse YAML: ${e.message}`);
23
+ return;
24
+ }
25
+
26
+ if (!schema || !schema.tables) {
27
+ console.error(' ❌ Invalid schema: No tables found.');
28
+ return;
29
+ }
30
+
31
+ console.log(` 🏗️ Calculating layout for ${schema.tables.length} tables...`);
32
+
33
+ // Initialize Dagre Graph
34
+ const g = new dagre.graphlib.Graph({ compound: true });
35
+ g.setGraph({
36
+ rankdir: 'LR',
37
+ nodesep: 80,
38
+ ranksep: 200,
39
+ marginx: 80,
40
+ marginy: 80,
41
+ });
42
+ g.setDefaultEdgeLabel(() => ({}));
43
+
44
+ // 1. Add Tables
45
+ schema.tables.forEach((table) => {
46
+ // Standard table size in canvas units
47
+ g.setNode(table.id, { width: 280, height: 160 });
48
+ });
49
+
50
+ // 2. Add Lineage Edges
51
+ if (schema.lineage) {
52
+ schema.lineage.forEach((edge) => {
53
+ if (g.hasNode(edge.from) && g.hasNode(edge.to)) {
54
+ g.setEdge(edge.from, edge.to);
55
+ }
56
+ });
57
+ }
58
+
59
+ // 3. Add ER Relationships
60
+ if (schema.relationships) {
61
+ schema.relationships.forEach((rel) => {
62
+ if (g.hasNode(rel.from.table) && g.hasNode(rel.to.table)) {
63
+ g.setEdge(rel.from.table, rel.to.table);
64
+ }
65
+ });
66
+ }
67
+
68
+ // 4. Setup Domains
69
+ const domainTableMap = new Map();
70
+ if (schema.domains) {
71
+ schema.domains.forEach((domain) => {
72
+ g.setNode(domain.id, { label: domain.name, cluster: true });
73
+ domain.tables.forEach((tableId) => {
74
+ if (g.hasNode(tableId)) {
75
+ g.setParent(tableId, domain.id);
76
+ }
77
+ });
78
+ domainTableMap.set(domain.id, domain.tables);
79
+ });
80
+ }
81
+
82
+ // Execute Layout Calculation
83
+ dagre.layout(g);
84
+
85
+ // 5. Post-process: Convert Dagre results to Modscape Layout format
86
+ const newLayout = {};
87
+ const PAD = 48;
88
+ const TABLE_W = 280;
89
+ const TABLE_H = 160; // Default height for layout estimation
90
+ const GAP = 40;
91
+
92
+ // Process Domains (Grid packing)
93
+ if (schema.domains) {
94
+ schema.domains.forEach(domain => {
95
+ const members = domain.tables.filter(tid => g.hasNode(tid));
96
+ if (members.length === 0) return;
97
+
98
+ // Sort by dagre rank (left -> right)
99
+ members.sort((a, b) => g.node(a).x - g.node(b).x);
100
+
101
+ const cols = Math.min(3, Math.ceil(Math.sqrt(members.length)));
102
+ const rowCount = Math.ceil(members.length / cols);
103
+
104
+ const gridW = cols * (TABLE_W + GAP) - GAP;
105
+ const gridH = rowCount * (TABLE_H + GAP) - GAP;
106
+
107
+ // Anchor grid to dagre centroid
108
+ const cx = members.reduce((s, tid) => s + g.node(tid).x, 0) / members.length;
109
+ const cy = members.reduce((s, tid) => s + g.node(tid).y, 0) / members.length;
110
+ const originX = cx - gridW / 2;
111
+ const originY = cy - gridH / 2;
112
+
113
+ members.forEach((tid, idx) => {
114
+ const col = idx % cols;
115
+ const row = Math.floor(idx / cols);
116
+ const xOff = col * (TABLE_W + GAP) + TABLE_W / 2;
117
+ const yOff = row * (TABLE_H + GAP) + TABLE_H / 2;
118
+ newLayout[tid] = {
119
+ x: Math.round(originX + xOff),
120
+ y: Math.round(originY + yOff),
121
+ parentId: domain.id
122
+ };
123
+ });
124
+
125
+ // Domain bounding box
126
+ const HEADER = 28;
127
+ newLayout[domain.id] = {
128
+ x: Math.round(originX - PAD),
129
+ y: Math.round(originY - PAD - HEADER),
130
+ width: Math.round(gridW + PAD * 2),
131
+ height: Math.round(gridH + PAD * 2 + HEADER)
132
+ };
133
+ });
134
+ }
135
+
136
+ // Standalone Tables
137
+ const domainTableIds = new Set(schema.domains?.flatMap(d => d.tables) ?? []);
138
+ schema.tables.forEach(t => {
139
+ if (!domainTableIds.has(t.id)) {
140
+ const pos = g.node(t.id);
141
+ newLayout[t.id] = { x: Math.round(pos.x), y: Math.round(pos.y) };
142
+ }
143
+ });
144
+
145
+ // 6. Save back to YAML
146
+ schema.layout = newLayout;
147
+ const outputPath = options.output ? path.resolve(process.cwd(), options.output) : absolutePath;
148
+
149
+ fs.writeFileSync(outputPath, yaml.dump(schema, { indent: 2, lineWidth: -1, noRefs: true }), 'utf8');
150
+ console.log(` ✅ Layout complete! Saved to: ${path.relative(process.cwd(), outputPath)}`);
151
+ }
package/src/sync-dbt.js CHANGED
@@ -59,15 +59,6 @@ export async function syncDbt(projectDir, options) {
59
59
  }
60
60
  }
61
61
 
62
- const lineageUpstream = [];
63
- if (node.depends_on?.nodes) {
64
- for (const upstreamId of node.depends_on.nodes) {
65
- if (allNodes[upstreamId]) {
66
- lineageUpstream.push(upstreamId);
67
- }
68
- }
69
- }
70
-
71
62
  latestTablesMap.set(tableId, {
72
63
  id: tableId,
73
64
  name: node.name,
@@ -76,7 +67,6 @@ export async function syncDbt(projectDir, options) {
76
67
  appearance: { type: 'table' },
77
68
  conceptual: { description: node.description || '' },
78
69
  columns,
79
- lineage: { upstream: lineageUpstream }
80
70
  });
81
71
  }
82
72
 
@@ -112,11 +102,24 @@ export async function syncDbt(projectDir, options) {
112
102
  physical_name: latest.physical_name,
113
103
  conceptual: latest.conceptual,
114
104
  columns: latest.columns,
115
- lineage: latest.lineage
116
105
  };
117
106
  });
118
107
 
119
- const updated = { ...existing, tables: newTables };
108
+ // Rebuild lineage from manifest for tables in this file
109
+ const fileTableIds = new Set(newTables.map(t => t.id));
110
+ const newLineage = [];
111
+ for (const [, node] of Object.entries(allNodes)) {
112
+ if (!fileTableIds.has(node.unique_id)) continue;
113
+ if (node.depends_on?.nodes) {
114
+ for (const upstreamId of node.depends_on.nodes) {
115
+ if (allNodes[upstreamId]) {
116
+ newLineage.push({ from: upstreamId, to: node.unique_id });
117
+ }
118
+ }
119
+ }
120
+ }
121
+
122
+ const updated = { ...existing, tables: newTables, lineage: newLineage };
120
123
  fs.writeFileSync(yamlPath, yaml.dump(updated), 'utf8');
121
124
  console.log(` 📄 Updated: ${yamlPath}`);
122
125
  }
@@ -8,11 +8,14 @@ Read this file alongside `.modscape/rules.md` (which defines the YAML schema) be
8
8
 
9
9
  ## 1. Dependency Order (DAG)
10
10
 
11
- Use `lineage.upstream` to determine build order. Always generate upstream models before downstream ones.
11
+ Use the top-level `lineage` section to determine build order. Always generate upstream (`from`) models before downstream (`to`) ones.
12
12
 
13
13
  ```yaml
14
14
  lineage:
15
- upstream: [stg_orders, stg_order_items] # these must be generated first
15
+ - from: stg_orders # must be generated first
16
+ to: fct_orders
17
+ - from: stg_order_items # must be generated first
18
+ to: fct_orders
16
19
  ```
17
20
 
18
21
  In dbt this becomes `{{ ref('stg_orders') }}`. In SQLMesh, `MODEL (... grain [...])` with `@this_model` references. Apply the equivalent pattern for your target tool.
@@ -123,7 +126,7 @@ Common TODO patterns:
123
126
 
124
127
  ## 8. Physical Table Names
125
128
 
126
- When `physical_name` is set on a table, use it as the actual table name in DDL or config blocks. The `id` field is the logical reference name used in `ref()` calls and `lineage.upstream`.
129
+ When `physical_name` is set on a table, use it as the actual table name in DDL or config blocks. The `id` field is the logical reference name used in `ref()` calls and the `lineage` section.
127
130
 
128
131
  ---
129
132
 
@@ -8,9 +8,9 @@
8
8
  ## QUICK REFERENCE (read this first)
9
9
 
10
10
  ```
11
- ROOT KEYS domains | tables | relationships | annotations | layout
11
+ ROOT KEYS domains | tables | relationships | lineage | annotations | layout
12
12
  COORDINATES ONLY in `layout`. NEVER inside tables or domains.
13
- LINEAGE Use lineage.upstream (not relationships) for mart/aggregated tables.
13
+ LINEAGE Use top-level `lineage` section (not relationships, not table.lineage.upstream).
14
14
  parentId Declare a table's domain membership inside layout, not inside domains.
15
15
  IDs Every object (table, domain, annotation) needs a unique `id`.
16
16
  sampleData First row = column IDs. At least 3 realistic data rows.
@@ -27,6 +27,7 @@ A valid `model.yaml` has exactly these top-level keys.
27
27
  domains: # (array) visual containers — OPTIONAL but recommended
28
28
  tables: # (array) entity definitions — REQUIRED
29
29
  relationships: # (array) ER cardinality edges — OPTIONAL
30
+ lineage: # (array) data lineage edges — OPTIONAL
30
31
  annotations: # (array) sticky notes / callouts — OPTIONAL
31
32
  layout: # (object) ALL coordinates — REQUIRED if any objects exist
32
33
  ```
@@ -140,17 +141,15 @@ relationships:
140
141
 
141
142
  ## 4. Data Lineage
142
143
 
143
- `lineage.upstream` declares which source tables a derived table is built from.
144
- This is rendered as animated arrows in **Lineage Mode**. It is separate from ER relationships.
144
+ Top-level `lineage` section declares data flow between tables (which source tables feed which derived tables).
145
+ This is rendered as dashed arrows in **Lineage Mode**. It is separate from ER relationships.
145
146
 
146
147
  ```yaml
147
- tables:
148
- - id: mart_revenue
149
- appearance: { type: mart }
150
- lineage:
151
- upstream:
152
- - fct_orders # list of source table IDs
153
- - dim_dates
148
+ lineage:
149
+ - from: fct_orders # source table id
150
+ to: mart_revenue # derived table id
151
+ - from: dim_dates
152
+ to: mart_revenue
154
153
  ```
155
154
 
156
155
  ### When to use lineage vs relationships
@@ -158,21 +157,21 @@ tables:
158
157
  | Situation | Use |
159
158
  |-----------|-----|
160
159
  | `dim_customers` → `fct_orders` (FK join) | `relationships` |
161
- | `fct_orders` + `dim_dates` → `mart_revenue` (aggregation) | `lineage.upstream` |
160
+ | `fct_orders` + `dim_dates` → `mart_revenue` (aggregation) | `lineage` |
162
161
 
163
- **MUST** define `lineage.upstream` for every `mart` or aggregated table.
164
- **MUST NOT** define `lineage.upstream` for raw tables (`fact`, `dimension`, `hub`, `link`, `satellite`).
165
- **MUST NOT** add a `relationships` entry for a connection already expressed in `lineage.upstream`.
162
+ **MUST** define `lineage` entries for every `mart` or aggregated table.
163
+ **MUST NOT** define `lineage` entries for raw tables (`fact`, `dimension`, `hub`, `link`, `satellite`) as sources.
164
+ **MUST NOT** add a `relationships` entry for a connection already expressed in `lineage`.
166
165
 
167
166
  #### Example: correct separation
168
167
 
169
168
  ```yaml
170
169
  # CORRECT
171
- tables:
172
- - id: mart_revenue
173
- appearance: { type: mart }
174
- lineage:
175
- upstream: [fct_orders, dim_dates] # lineage only
170
+ lineage:
171
+ - from: fct_orders
172
+ to: mart_revenue
173
+ - from: dim_dates
174
+ to: mart_revenue
176
175
 
177
176
  relationships:
178
177
  - from: { table: dim_customers, column: customer_key }
@@ -409,11 +408,9 @@ relationships:
409
408
 
410
409
  ```yaml
411
410
  # CORRECT
412
- tables:
413
- - id: mart_revenue
414
- appearance: { type: mart }
415
- lineage:
416
- upstream: [fct_orders] # ✅ express lineage here
411
+ lineage:
412
+ - from: fct_orders
413
+ to: mart_revenue # express lineage in the top-level lineage section
417
414
  ```
418
415
 
419
416
  ---
@@ -663,10 +660,6 @@ tables:
663
660
  logical_name: "Executive Revenue Summary"
664
661
  physical_name: "mart_finance_monthly_revenue_agg"
665
662
  appearance: { type: mart, icon: "📈" }
666
- lineage: # mart → use lineage, not relationships
667
- upstream:
668
- - fct_orders
669
- - dim_customers
670
663
  implementation:
671
664
  materialization: table
672
665
  grain: [month_key]
@@ -684,6 +677,12 @@ tables:
684
677
  - ["2024-02", 15200.00]
685
678
  - ["2024-03", 18900.75]
686
679
 
680
+ lineage: # data flow — separate from ER
681
+ - from: fct_orders
682
+ to: mart_monthly_revenue
683
+ - from: dim_customers
684
+ to: mart_monthly_revenue
685
+
687
686
  relationships: # ER only — not for lineage
688
687
  - from: { table: dim_customers, column: customer_key }
689
688
  to: { table: fct_orders, column: customer_key }