modscape 1.3.0 → 2.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.ja.md +14 -20
- package/README.md +16 -10
- package/package.json +2 -1
- package/src/export.js +12 -14
- package/src/import-dbt.js +12 -10
- package/src/index.js +10 -0
- package/src/layout.js +151 -0
- package/src/sync-dbt.js +15 -12
- package/src/templates/codegen-rules.md +6 -3
- package/src/templates/rules.md +28 -29
- package/visualizer/package.json +9 -4
- package/visualizer-dist/assets/index-DTXT8yZr.js +432 -0
- package/visualizer-dist/assets/index-zFK5TNtm.css +1 -0
- package/visualizer-dist/index.html +2 -2
- package/visualizer-dist/assets/index-D-14ykQt.js +0 -63
- package/visualizer-dist/assets/index-DHPAF3El.css +0 -1
package/README.ja.md
CHANGED
|
@@ -92,6 +92,7 @@ YAMLのルートレベル構造は以下の通りです:
|
|
|
92
92
|
domains – 関連テーブルをまとめるビジュアルコンテナ
|
|
93
93
|
tables – 3階層メタデータを持つエンティティ定義
|
|
94
94
|
relationships – テーブル間のERカーディナリティ
|
|
95
|
+
lineage – データの流れ / 変換パス
|
|
95
96
|
annotations – キャンバス上のスティッキーノート・吹き出し
|
|
96
97
|
layout – 全座標データ(tables/domains の中に x/y を書いてはいけない)
|
|
97
98
|
```
|
|
@@ -104,7 +105,7 @@ domains:
|
|
|
104
105
|
name: "主要売上"
|
|
105
106
|
description: "営業チームのトランザクションデータ。" # 任意
|
|
106
107
|
color: "rgba(59, 130, 246, 0.1)" # 背景色
|
|
107
|
-
tables: [orders, dim_customers]
|
|
108
|
+
tables: [orders, dim_customers] # 論理的な所属リスト
|
|
108
109
|
isLocked: false # true にするとキャンバスでの誤ドラッグを防止
|
|
109
110
|
```
|
|
110
111
|
|
|
@@ -130,11 +131,6 @@ tables:
|
|
|
130
131
|
businessDefinitions:
|
|
131
132
|
revenue: "割引・返品後の純売上"
|
|
132
133
|
|
|
133
|
-
lineage: # mart/集計テーブルのみに定義
|
|
134
|
-
upstream:
|
|
135
|
-
- fct_sales
|
|
136
|
-
- dim_dates
|
|
137
|
-
|
|
138
134
|
implementation: # 任意 – AIコード生成へのヒント
|
|
139
135
|
materialization: incremental # table | view | incremental | ephemeral
|
|
140
136
|
incremental_strategy: merge # merge | append | delete+insert
|
|
@@ -165,23 +161,22 @@ tables:
|
|
|
165
161
|
type: "BIGINT"
|
|
166
162
|
constraints: [NOT NULL]
|
|
167
163
|
|
|
168
|
-
sampleData: # 2
|
|
169
|
-
- [order_id, amount, status]
|
|
164
|
+
sampleData: # 実数値の2次元配列
|
|
170
165
|
- [1001, 50.0, "COMPLETED"]
|
|
171
166
|
- [1002, 120.5, "PENDING"]
|
|
172
167
|
```
|
|
173
168
|
|
|
174
|
-
|
|
169
|
+
### Data Lineage(データリネージ)
|
|
170
|
+
|
|
171
|
+
ルートレベルの `lineage` セクションでテーブル間のデータの流れ(どのソースからどの集計テーブルが作られるか)を定義します。リネージモードではアニメーション付きの点線矢印として表示されます。
|
|
175
172
|
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
| `satellite` | Data Vault のハブに紐づく履歴属性 |
|
|
184
|
-
| `table` | 汎用 |
|
|
173
|
+
```yaml
|
|
174
|
+
lineage:
|
|
175
|
+
- from: fct_orders # ソーステーブル ID
|
|
176
|
+
to: mart_revenue # 派生テーブル ID
|
|
177
|
+
- from: dim_dates
|
|
178
|
+
to: mart_revenue
|
|
179
|
+
```
|
|
185
180
|
|
|
186
181
|
### Relationships(リレーションシップ)
|
|
187
182
|
|
|
@@ -196,7 +191,7 @@ relationships:
|
|
|
196
191
|
type: one-to-many # one-to-one | one-to-many | many-to-one | many-to-many
|
|
197
192
|
```
|
|
198
193
|
|
|
199
|
-
>
|
|
194
|
+
> **ER関係** vs **リネージ**: 構造的な結合(外部キーなど)には `relationships` を、データの加工・変換の流れには `lineage` を使用してください。両方に同じ接続を記述しないでください。
|
|
200
195
|
|
|
201
196
|
### Annotations(アノテーション)
|
|
202
197
|
|
|
@@ -270,7 +265,6 @@ modscape export ./models -o docs/ARCHITECTURE.md
|
|
|
270
265
|
|
|
271
266
|
Modscape は以下の素晴らしいオープンソースプロジェクトによって支えられています:
|
|
272
267
|
|
|
273
|
-
- [React Flow](https://reactflow.dev/) - インタラクティブなグラフ UI フレームワーク。
|
|
274
268
|
- [CodeMirror 6](https://codemirror.net/) - 次世代のウェブベース・コードエディタ。
|
|
275
269
|
- [Dagre](https://github.com/dagrejs/dagre) - 階層型グラフ・レイアウトエンジン。
|
|
276
270
|
- [Lucide React](https://lucide.dev/) - シンプルで美しいアイコンセット。
|
package/README.md
CHANGED
|
@@ -95,6 +95,7 @@ Modscape uses a schema designed for data analysis contexts. The full YAML struct
|
|
|
95
95
|
domains – visual containers grouping related tables
|
|
96
96
|
tables – entity definitions with tri-layer metadata
|
|
97
97
|
relationships – ER cardinality between tables
|
|
98
|
+
lineage – data flow / transformation paths
|
|
98
99
|
annotations – sticky notes / callouts on the canvas
|
|
99
100
|
layout – ALL coordinate data (never put x/y inside tables or domains)
|
|
100
101
|
```
|
|
@@ -107,7 +108,7 @@ domains:
|
|
|
107
108
|
name: "Core Sales"
|
|
108
109
|
description: "Transactional data for the sales team." # optional
|
|
109
110
|
color: "rgba(59, 130, 246, 0.1)" # background fill
|
|
110
|
-
tables: [orders, dim_customers]
|
|
111
|
+
tables: [orders, dim_customers] # logical membership
|
|
111
112
|
isLocked: false # prevent accidental drag when true
|
|
112
113
|
```
|
|
113
114
|
|
|
@@ -133,11 +134,6 @@ tables:
|
|
|
133
134
|
businessDefinitions:
|
|
134
135
|
revenue: "Net revenue after discounts"
|
|
135
136
|
|
|
136
|
-
lineage: # for mart/aggregated tables only
|
|
137
|
-
upstream:
|
|
138
|
-
- fct_sales
|
|
139
|
-
- dim_dates
|
|
140
|
-
|
|
141
137
|
implementation: # optional – hints for AI code generation
|
|
142
138
|
materialization: incremental # table | view | incremental | ephemeral
|
|
143
139
|
incremental_strategy: merge # merge | append | delete+insert
|
|
@@ -168,12 +164,23 @@ tables:
|
|
|
168
164
|
type: "BIGINT"
|
|
169
165
|
constraints: [NOT NULL]
|
|
170
166
|
|
|
171
|
-
sampleData: # 2D array
|
|
172
|
-
- [order_id, amount, status]
|
|
167
|
+
sampleData: # 2D array of realistic values
|
|
173
168
|
- [1001, 50.0, "COMPLETED"]
|
|
174
169
|
- [1002, 120.5, "PENDING"]
|
|
175
170
|
```
|
|
176
171
|
|
|
172
|
+
### Data Lineage
|
|
173
|
+
|
|
174
|
+
Top-level `lineage` section declares data flow between tables (which source tables feed which derived tables). This is rendered as dashed arrows in **Lineage Mode**.
|
|
175
|
+
|
|
176
|
+
```yaml
|
|
177
|
+
lineage:
|
|
178
|
+
- from: fct_orders # source table ID
|
|
179
|
+
to: mart_revenue # derived table ID
|
|
180
|
+
- from: dim_dates
|
|
181
|
+
to: mart_revenue
|
|
182
|
+
```
|
|
183
|
+
|
|
177
184
|
### Relationships
|
|
178
185
|
|
|
179
186
|
```yaml
|
|
@@ -187,7 +194,7 @@ relationships:
|
|
|
187
194
|
type: one-to-many # one-to-one | one-to-many | many-to-one | many-to-many
|
|
188
195
|
```
|
|
189
196
|
|
|
190
|
-
> **
|
|
197
|
+
> **ER Relationships** vs **Lineage**: Use `relationships` for structural joins (FKs) and `lineage` for data flow (transformations). Do not duplicate them.
|
|
191
198
|
|
|
192
199
|
### Annotations
|
|
193
200
|
|
|
@@ -261,7 +268,6 @@ modscape export ./models -o docs/ARCHITECTURE.md
|
|
|
261
268
|
|
|
262
269
|
Modscape is made possible by these incredible open-source projects:
|
|
263
270
|
|
|
264
|
-
- [React Flow](https://reactflow.dev/) - Interactive node-based UI framework.
|
|
265
271
|
- [CodeMirror 6](https://codemirror.net/) - Next-generation code editor for the web.
|
|
266
272
|
- [Dagre](https://github.com/dagrejs/dagre) - Directed graph layout engine.
|
|
267
273
|
- [Lucide React](https://lucide.dev/) - Beautifully simple pixel-perfect icons.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "modscape",
|
|
3
|
-
"version": "
|
|
3
|
+
"version": "2.0.1",
|
|
4
4
|
"description": "Modscape: A YAML-driven data modeling visualizer CLI",
|
|
5
5
|
"repository": {
|
|
6
6
|
"type": "git",
|
|
@@ -29,6 +29,7 @@
|
|
|
29
29
|
"@inquirer/prompts": "^8.3.0",
|
|
30
30
|
"chokidar": "^5.0.0",
|
|
31
31
|
"commander": "^14.0.3",
|
|
32
|
+
"dagre": "^0.8.5",
|
|
32
33
|
"express": "^5.2.1",
|
|
33
34
|
"js-yaml": "^4.1.1",
|
|
34
35
|
"open": "^11.0.0",
|
package/src/export.js
CHANGED
|
@@ -67,16 +67,13 @@ function generateMermaidLineage(schema) {
|
|
|
67
67
|
let mermaid = 'graph TD\n';
|
|
68
68
|
let hasLineage = false;
|
|
69
69
|
|
|
70
|
-
schema.
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
mermaid += ` ${sourceName} --> ${targetName}\n`;
|
|
78
|
-
});
|
|
79
|
-
}
|
|
70
|
+
(schema.lineage || []).forEach(edge => {
|
|
71
|
+
hasLineage = true;
|
|
72
|
+
const sourceTable = schema.tables.find(t => t.id === edge.from);
|
|
73
|
+
const targetTable = schema.tables.find(t => t.id === edge.to);
|
|
74
|
+
const sourceName = sanitize(sourceTable?.name || edge.from);
|
|
75
|
+
const targetName = sanitize(targetTable?.name || edge.to);
|
|
76
|
+
mermaid += ` ${sourceName} --> ${targetName}\n`;
|
|
80
77
|
});
|
|
81
78
|
|
|
82
79
|
return hasLineage ? mermaid : null;
|
|
@@ -197,10 +194,11 @@ export function generateMarkdown(schema, modelName) {
|
|
|
197
194
|
}
|
|
198
195
|
|
|
199
196
|
// Lineage
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
|
|
197
|
+
const upstreamEdges = (schema.lineage || []).filter(e => e.to === table.id);
|
|
198
|
+
if (upstreamEdges.length > 0) {
|
|
199
|
+
const upstreamNames = upstreamEdges.map(e => {
|
|
200
|
+
const t = schema.tables.find(t => t.id === e.from);
|
|
201
|
+
return t ? t.name : e.from;
|
|
204
202
|
});
|
|
205
203
|
md += `**Upstream**: ${upstreamNames.map(n => `\`${n}\``).join(' → ')}\n\n`;
|
|
206
204
|
}
|
package/src/import-dbt.js
CHANGED
|
@@ -34,6 +34,7 @@ export async function importDbt(projectDir, options) {
|
|
|
34
34
|
console.log(` 🔍 Parsing dbt manifest: ${manifestPath}`);
|
|
35
35
|
|
|
36
36
|
const tables = [];
|
|
37
|
+
const lineage = [];
|
|
37
38
|
const domainsMap = new Map();
|
|
38
39
|
const tableSplitKeyMap = new Map();
|
|
39
40
|
|
|
@@ -66,7 +67,6 @@ export async function importDbt(projectDir, options) {
|
|
|
66
67
|
appearance: { type: 'table' },
|
|
67
68
|
conceptual: { description: node.description || '' },
|
|
68
69
|
columns,
|
|
69
|
-
lineage: { upstream: [] }
|
|
70
70
|
};
|
|
71
71
|
|
|
72
72
|
tables.push(tableEntry);
|
|
@@ -92,13 +92,12 @@ export async function importDbt(projectDir, options) {
|
|
|
92
92
|
}
|
|
93
93
|
|
|
94
94
|
// lineage
|
|
95
|
-
for (const [
|
|
95
|
+
for (const [, node] of Object.entries(allNodes)) {
|
|
96
96
|
if (!['model', 'seed', 'snapshot', 'source'].includes(node.resource_type)) continue;
|
|
97
|
-
|
|
98
|
-
if (tableEntry && node.depends_on?.nodes) {
|
|
97
|
+
if (node.depends_on?.nodes) {
|
|
99
98
|
for (const upstreamId of node.depends_on.nodes) {
|
|
100
99
|
if (allNodes[upstreamId]) {
|
|
101
|
-
|
|
100
|
+
lineage.push({ from: upstreamId, to: node.unique_id });
|
|
102
101
|
}
|
|
103
102
|
}
|
|
104
103
|
}
|
|
@@ -122,11 +121,10 @@ export async function importDbt(projectDir, options) {
|
|
|
122
121
|
const tableIds = new Set(splitTables.map(t => t.id));
|
|
123
122
|
let internal = 0;
|
|
124
123
|
let external = 0;
|
|
125
|
-
for (const
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
}
|
|
124
|
+
for (const edge of lineage) {
|
|
125
|
+
if (!tableIds.has(edge.to)) continue;
|
|
126
|
+
if (tableIds.has(edge.from)) internal++;
|
|
127
|
+
else external++;
|
|
130
128
|
}
|
|
131
129
|
const total = internal + external;
|
|
132
130
|
const rate = total > 0 ? Math.round(internal / total * 100) : 100;
|
|
@@ -145,9 +143,12 @@ export async function importDbt(projectDir, options) {
|
|
|
145
143
|
tables: d.tables.filter(tid => splitTables.some(t => t.id === tid))
|
|
146
144
|
}));
|
|
147
145
|
|
|
146
|
+
const splitTableIds = new Set(splitTables.map(t => t.id));
|
|
147
|
+
const splitLineage = lineage.filter(e => splitTableIds.has(e.from) || splitTableIds.has(e.to));
|
|
148
148
|
const outputModel = {
|
|
149
149
|
tables: splitTables,
|
|
150
150
|
relationships: [],
|
|
151
|
+
lineage: splitLineage,
|
|
151
152
|
domains: splitDomains
|
|
152
153
|
};
|
|
153
154
|
|
|
@@ -167,6 +168,7 @@ export async function importDbt(projectDir, options) {
|
|
|
167
168
|
const outputModel = {
|
|
168
169
|
tables,
|
|
169
170
|
relationships: [],
|
|
171
|
+
lineage,
|
|
170
172
|
domains: Array.from(domainsMap.values())
|
|
171
173
|
};
|
|
172
174
|
|
package/src/index.js
CHANGED
|
@@ -10,6 +10,7 @@ import { exportModel } from './export.js';
|
|
|
10
10
|
import { createModel } from './create.js';
|
|
11
11
|
import { importDbt } from './import-dbt.js';
|
|
12
12
|
import { syncDbt } from './sync-dbt.js';
|
|
13
|
+
import { applyLayout } from './layout.js';
|
|
13
14
|
import { createRequire } from 'module';
|
|
14
15
|
import { mergeModels } from './merge.js';
|
|
15
16
|
|
|
@@ -104,4 +105,13 @@ program
|
|
|
104
105
|
mergeModels(paths, options);
|
|
105
106
|
});
|
|
106
107
|
|
|
108
|
+
program
|
|
109
|
+
.command('layout')
|
|
110
|
+
.description('Perform automatic layout calculation and update the YAML file')
|
|
111
|
+
.argument('<path>', 'path to the YAML model file')
|
|
112
|
+
.option('-o, --output <path>', 'output file path (defaults to overwriting input)')
|
|
113
|
+
.action((path, options) => {
|
|
114
|
+
applyLayout(path, options);
|
|
115
|
+
});
|
|
116
|
+
|
|
107
117
|
program.parse();
|
package/src/layout.js
ADDED
|
@@ -0,0 +1,151 @@
|
|
|
1
|
+
import fs from 'fs';
|
|
2
|
+
import path from 'path';
|
|
3
|
+
import yaml from 'js-yaml';
|
|
4
|
+
import dagre from 'dagre';
|
|
5
|
+
|
|
6
|
+
/**
|
|
7
|
+
* CLI-based layout engine for Modscape.
|
|
8
|
+
* Replicates the logic from the visualizer to ensure consistency.
|
|
9
|
+
*/
|
|
10
|
+
export function applyLayout(inputPath, options = {}) {
|
|
11
|
+
const absolutePath = path.resolve(process.cwd(), inputPath);
|
|
12
|
+
if (!fs.existsSync(absolutePath)) {
|
|
13
|
+
console.error(` ❌ File not found: ${inputPath}`);
|
|
14
|
+
return;
|
|
15
|
+
}
|
|
16
|
+
|
|
17
|
+
const raw = fs.readFileSync(absolutePath, 'utf8');
|
|
18
|
+
let schema;
|
|
19
|
+
try {
|
|
20
|
+
schema = yaml.load(raw);
|
|
21
|
+
} catch (e) {
|
|
22
|
+
console.error(` ❌ Failed to parse YAML: ${e.message}`);
|
|
23
|
+
return;
|
|
24
|
+
}
|
|
25
|
+
|
|
26
|
+
if (!schema || !schema.tables) {
|
|
27
|
+
console.error(' ❌ Invalid schema: No tables found.');
|
|
28
|
+
return;
|
|
29
|
+
}
|
|
30
|
+
|
|
31
|
+
console.log(` 🏗️ Calculating layout for ${schema.tables.length} tables...`);
|
|
32
|
+
|
|
33
|
+
// Initialize Dagre Graph
|
|
34
|
+
const g = new dagre.graphlib.Graph({ compound: true });
|
|
35
|
+
g.setGraph({
|
|
36
|
+
rankdir: 'LR',
|
|
37
|
+
nodesep: 80,
|
|
38
|
+
ranksep: 200,
|
|
39
|
+
marginx: 80,
|
|
40
|
+
marginy: 80,
|
|
41
|
+
});
|
|
42
|
+
g.setDefaultEdgeLabel(() => ({}));
|
|
43
|
+
|
|
44
|
+
// 1. Add Tables
|
|
45
|
+
schema.tables.forEach((table) => {
|
|
46
|
+
// Standard table size in canvas units
|
|
47
|
+
g.setNode(table.id, { width: 280, height: 160 });
|
|
48
|
+
});
|
|
49
|
+
|
|
50
|
+
// 2. Add Lineage Edges
|
|
51
|
+
if (schema.lineage) {
|
|
52
|
+
schema.lineage.forEach((edge) => {
|
|
53
|
+
if (g.hasNode(edge.from) && g.hasNode(edge.to)) {
|
|
54
|
+
g.setEdge(edge.from, edge.to);
|
|
55
|
+
}
|
|
56
|
+
});
|
|
57
|
+
}
|
|
58
|
+
|
|
59
|
+
// 3. Add ER Relationships
|
|
60
|
+
if (schema.relationships) {
|
|
61
|
+
schema.relationships.forEach((rel) => {
|
|
62
|
+
if (g.hasNode(rel.from.table) && g.hasNode(rel.to.table)) {
|
|
63
|
+
g.setEdge(rel.from.table, rel.to.table);
|
|
64
|
+
}
|
|
65
|
+
});
|
|
66
|
+
}
|
|
67
|
+
|
|
68
|
+
// 4. Setup Domains
|
|
69
|
+
const domainTableMap = new Map();
|
|
70
|
+
if (schema.domains) {
|
|
71
|
+
schema.domains.forEach((domain) => {
|
|
72
|
+
g.setNode(domain.id, { label: domain.name, cluster: true });
|
|
73
|
+
domain.tables.forEach((tableId) => {
|
|
74
|
+
if (g.hasNode(tableId)) {
|
|
75
|
+
g.setParent(tableId, domain.id);
|
|
76
|
+
}
|
|
77
|
+
});
|
|
78
|
+
domainTableMap.set(domain.id, domain.tables);
|
|
79
|
+
});
|
|
80
|
+
}
|
|
81
|
+
|
|
82
|
+
// Execute Layout Calculation
|
|
83
|
+
dagre.layout(g);
|
|
84
|
+
|
|
85
|
+
// 5. Post-process: Convert Dagre results to Modscape Layout format
|
|
86
|
+
const newLayout = {};
|
|
87
|
+
const PAD = 48;
|
|
88
|
+
const TABLE_W = 280;
|
|
89
|
+
const TABLE_H = 160; // Default height for layout estimation
|
|
90
|
+
const GAP = 40;
|
|
91
|
+
|
|
92
|
+
// Process Domains (Grid packing)
|
|
93
|
+
if (schema.domains) {
|
|
94
|
+
schema.domains.forEach(domain => {
|
|
95
|
+
const members = domain.tables.filter(tid => g.hasNode(tid));
|
|
96
|
+
if (members.length === 0) return;
|
|
97
|
+
|
|
98
|
+
// Sort by dagre rank (left -> right)
|
|
99
|
+
members.sort((a, b) => g.node(a).x - g.node(b).x);
|
|
100
|
+
|
|
101
|
+
const cols = Math.min(3, Math.ceil(Math.sqrt(members.length)));
|
|
102
|
+
const rowCount = Math.ceil(members.length / cols);
|
|
103
|
+
|
|
104
|
+
const gridW = cols * (TABLE_W + GAP) - GAP;
|
|
105
|
+
const gridH = rowCount * (TABLE_H + GAP) - GAP;
|
|
106
|
+
|
|
107
|
+
// Anchor grid to dagre centroid
|
|
108
|
+
const cx = members.reduce((s, tid) => s + g.node(tid).x, 0) / members.length;
|
|
109
|
+
const cy = members.reduce((s, tid) => s + g.node(tid).y, 0) / members.length;
|
|
110
|
+
const originX = cx - gridW / 2;
|
|
111
|
+
const originY = cy - gridH / 2;
|
|
112
|
+
|
|
113
|
+
members.forEach((tid, idx) => {
|
|
114
|
+
const col = idx % cols;
|
|
115
|
+
const row = Math.floor(idx / cols);
|
|
116
|
+
const xOff = col * (TABLE_W + GAP) + TABLE_W / 2;
|
|
117
|
+
const yOff = row * (TABLE_H + GAP) + TABLE_H / 2;
|
|
118
|
+
newLayout[tid] = {
|
|
119
|
+
x: Math.round(originX + xOff),
|
|
120
|
+
y: Math.round(originY + yOff),
|
|
121
|
+
parentId: domain.id
|
|
122
|
+
};
|
|
123
|
+
});
|
|
124
|
+
|
|
125
|
+
// Domain bounding box
|
|
126
|
+
const HEADER = 28;
|
|
127
|
+
newLayout[domain.id] = {
|
|
128
|
+
x: Math.round(originX - PAD),
|
|
129
|
+
y: Math.round(originY - PAD - HEADER),
|
|
130
|
+
width: Math.round(gridW + PAD * 2),
|
|
131
|
+
height: Math.round(gridH + PAD * 2 + HEADER)
|
|
132
|
+
};
|
|
133
|
+
});
|
|
134
|
+
}
|
|
135
|
+
|
|
136
|
+
// Standalone Tables
|
|
137
|
+
const domainTableIds = new Set(schema.domains?.flatMap(d => d.tables) ?? []);
|
|
138
|
+
schema.tables.forEach(t => {
|
|
139
|
+
if (!domainTableIds.has(t.id)) {
|
|
140
|
+
const pos = g.node(t.id);
|
|
141
|
+
newLayout[t.id] = { x: Math.round(pos.x), y: Math.round(pos.y) };
|
|
142
|
+
}
|
|
143
|
+
});
|
|
144
|
+
|
|
145
|
+
// 6. Save back to YAML
|
|
146
|
+
schema.layout = newLayout;
|
|
147
|
+
const outputPath = options.output ? path.resolve(process.cwd(), options.output) : absolutePath;
|
|
148
|
+
|
|
149
|
+
fs.writeFileSync(outputPath, yaml.dump(schema, { indent: 2, lineWidth: -1, noRefs: true }), 'utf8');
|
|
150
|
+
console.log(` ✅ Layout complete! Saved to: ${path.relative(process.cwd(), outputPath)}`);
|
|
151
|
+
}
|
package/src/sync-dbt.js
CHANGED
|
@@ -59,15 +59,6 @@ export async function syncDbt(projectDir, options) {
|
|
|
59
59
|
}
|
|
60
60
|
}
|
|
61
61
|
|
|
62
|
-
const lineageUpstream = [];
|
|
63
|
-
if (node.depends_on?.nodes) {
|
|
64
|
-
for (const upstreamId of node.depends_on.nodes) {
|
|
65
|
-
if (allNodes[upstreamId]) {
|
|
66
|
-
lineageUpstream.push(upstreamId);
|
|
67
|
-
}
|
|
68
|
-
}
|
|
69
|
-
}
|
|
70
|
-
|
|
71
62
|
latestTablesMap.set(tableId, {
|
|
72
63
|
id: tableId,
|
|
73
64
|
name: node.name,
|
|
@@ -76,7 +67,6 @@ export async function syncDbt(projectDir, options) {
|
|
|
76
67
|
appearance: { type: 'table' },
|
|
77
68
|
conceptual: { description: node.description || '' },
|
|
78
69
|
columns,
|
|
79
|
-
lineage: { upstream: lineageUpstream }
|
|
80
70
|
});
|
|
81
71
|
}
|
|
82
72
|
|
|
@@ -112,11 +102,24 @@ export async function syncDbt(projectDir, options) {
|
|
|
112
102
|
physical_name: latest.physical_name,
|
|
113
103
|
conceptual: latest.conceptual,
|
|
114
104
|
columns: latest.columns,
|
|
115
|
-
lineage: latest.lineage
|
|
116
105
|
};
|
|
117
106
|
});
|
|
118
107
|
|
|
119
|
-
|
|
108
|
+
// Rebuild lineage from manifest for tables in this file
|
|
109
|
+
const fileTableIds = new Set(newTables.map(t => t.id));
|
|
110
|
+
const newLineage = [];
|
|
111
|
+
for (const [, node] of Object.entries(allNodes)) {
|
|
112
|
+
if (!fileTableIds.has(node.unique_id)) continue;
|
|
113
|
+
if (node.depends_on?.nodes) {
|
|
114
|
+
for (const upstreamId of node.depends_on.nodes) {
|
|
115
|
+
if (allNodes[upstreamId]) {
|
|
116
|
+
newLineage.push({ from: upstreamId, to: node.unique_id });
|
|
117
|
+
}
|
|
118
|
+
}
|
|
119
|
+
}
|
|
120
|
+
}
|
|
121
|
+
|
|
122
|
+
const updated = { ...existing, tables: newTables, lineage: newLineage };
|
|
120
123
|
fs.writeFileSync(yamlPath, yaml.dump(updated), 'utf8');
|
|
121
124
|
console.log(` 📄 Updated: ${yamlPath}`);
|
|
122
125
|
}
|
|
@@ -8,11 +8,14 @@ Read this file alongside `.modscape/rules.md` (which defines the YAML schema) be
|
|
|
8
8
|
|
|
9
9
|
## 1. Dependency Order (DAG)
|
|
10
10
|
|
|
11
|
-
Use `lineage
|
|
11
|
+
Use the top-level `lineage` section to determine build order. Always generate upstream (`from`) models before downstream (`to`) ones.
|
|
12
12
|
|
|
13
13
|
```yaml
|
|
14
14
|
lineage:
|
|
15
|
-
|
|
15
|
+
- from: stg_orders # must be generated first
|
|
16
|
+
to: fct_orders
|
|
17
|
+
- from: stg_order_items # must be generated first
|
|
18
|
+
to: fct_orders
|
|
16
19
|
```
|
|
17
20
|
|
|
18
21
|
In dbt this becomes `{{ ref('stg_orders') }}`. In SQLMesh, `MODEL (... grain [...])` with `@this_model` references. Apply the equivalent pattern for your target tool.
|
|
@@ -123,7 +126,7 @@ Common TODO patterns:
|
|
|
123
126
|
|
|
124
127
|
## 8. Physical Table Names
|
|
125
128
|
|
|
126
|
-
When `physical_name` is set on a table, use it as the actual table name in DDL or config blocks. The `id` field is the logical reference name used in `ref()` calls and `lineage.
|
|
129
|
+
When `physical_name` is set on a table, use it as the actual table name in DDL or config blocks. The `id` field is the logical reference name used in `ref()` calls and the `lineage` section.
|
|
127
130
|
|
|
128
131
|
---
|
|
129
132
|
|
package/src/templates/rules.md
CHANGED
|
@@ -8,9 +8,9 @@
|
|
|
8
8
|
## QUICK REFERENCE (read this first)
|
|
9
9
|
|
|
10
10
|
```
|
|
11
|
-
ROOT KEYS domains | tables | relationships | annotations | layout
|
|
11
|
+
ROOT KEYS domains | tables | relationships | lineage | annotations | layout
|
|
12
12
|
COORDINATES ONLY in `layout`. NEVER inside tables or domains.
|
|
13
|
-
LINEAGE Use lineage
|
|
13
|
+
LINEAGE Use top-level `lineage` section (not relationships, not table.lineage.upstream).
|
|
14
14
|
parentId Declare a table's domain membership inside layout, not inside domains.
|
|
15
15
|
IDs Every object (table, domain, annotation) needs a unique `id`.
|
|
16
16
|
sampleData First row = column IDs. At least 3 realistic data rows.
|
|
@@ -27,6 +27,7 @@ A valid `model.yaml` has exactly these top-level keys.
|
|
|
27
27
|
domains: # (array) visual containers — OPTIONAL but recommended
|
|
28
28
|
tables: # (array) entity definitions — REQUIRED
|
|
29
29
|
relationships: # (array) ER cardinality edges — OPTIONAL
|
|
30
|
+
lineage: # (array) data lineage edges — OPTIONAL
|
|
30
31
|
annotations: # (array) sticky notes / callouts — OPTIONAL
|
|
31
32
|
layout: # (object) ALL coordinates — REQUIRED if any objects exist
|
|
32
33
|
```
|
|
@@ -140,17 +141,15 @@ relationships:
|
|
|
140
141
|
|
|
141
142
|
## 4. Data Lineage
|
|
142
143
|
|
|
143
|
-
`lineage
|
|
144
|
-
This is rendered as
|
|
144
|
+
Top-level `lineage` section declares data flow between tables (which source tables feed which derived tables).
|
|
145
|
+
This is rendered as dashed arrows in **Lineage Mode**. It is separate from ER relationships.
|
|
145
146
|
|
|
146
147
|
```yaml
|
|
147
|
-
|
|
148
|
-
-
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
- fct_orders # list of source table IDs
|
|
153
|
-
- dim_dates
|
|
148
|
+
lineage:
|
|
149
|
+
- from: fct_orders # source table id
|
|
150
|
+
to: mart_revenue # derived table id
|
|
151
|
+
- from: dim_dates
|
|
152
|
+
to: mart_revenue
|
|
154
153
|
```
|
|
155
154
|
|
|
156
155
|
### When to use lineage vs relationships
|
|
@@ -158,21 +157,21 @@ tables:
|
|
|
158
157
|
| Situation | Use |
|
|
159
158
|
|-----------|-----|
|
|
160
159
|
| `dim_customers` → `fct_orders` (FK join) | `relationships` |
|
|
161
|
-
| `fct_orders` + `dim_dates` → `mart_revenue` (aggregation) | `lineage
|
|
160
|
+
| `fct_orders` + `dim_dates` → `mart_revenue` (aggregation) | `lineage` |
|
|
162
161
|
|
|
163
|
-
**MUST** define `lineage
|
|
164
|
-
**MUST NOT** define `lineage
|
|
165
|
-
**MUST NOT** add a `relationships` entry for a connection already expressed in `lineage
|
|
162
|
+
**MUST** define `lineage` entries for every `mart` or aggregated table.
|
|
163
|
+
**MUST NOT** define `lineage` entries for raw tables (`fact`, `dimension`, `hub`, `link`, `satellite`) as sources.
|
|
164
|
+
**MUST NOT** add a `relationships` entry for a connection already expressed in `lineage`.
|
|
166
165
|
|
|
167
166
|
#### Example: correct separation
|
|
168
167
|
|
|
169
168
|
```yaml
|
|
170
169
|
# CORRECT
|
|
171
|
-
|
|
172
|
-
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
170
|
+
lineage:
|
|
171
|
+
- from: fct_orders
|
|
172
|
+
to: mart_revenue
|
|
173
|
+
- from: dim_dates
|
|
174
|
+
to: mart_revenue
|
|
176
175
|
|
|
177
176
|
relationships:
|
|
178
177
|
- from: { table: dim_customers, column: customer_key }
|
|
@@ -409,11 +408,9 @@ relationships:
|
|
|
409
408
|
|
|
410
409
|
```yaml
|
|
411
410
|
# CORRECT
|
|
412
|
-
|
|
413
|
-
-
|
|
414
|
-
|
|
415
|
-
lineage:
|
|
416
|
-
upstream: [fct_orders] # ✅ express lineage here
|
|
411
|
+
lineage:
|
|
412
|
+
- from: fct_orders
|
|
413
|
+
to: mart_revenue # ✅ express lineage in the top-level lineage section
|
|
417
414
|
```
|
|
418
415
|
|
|
419
416
|
---
|
|
@@ -663,10 +660,6 @@ tables:
|
|
|
663
660
|
logical_name: "Executive Revenue Summary"
|
|
664
661
|
physical_name: "mart_finance_monthly_revenue_agg"
|
|
665
662
|
appearance: { type: mart, icon: "📈" }
|
|
666
|
-
lineage: # mart → use lineage, not relationships
|
|
667
|
-
upstream:
|
|
668
|
-
- fct_orders
|
|
669
|
-
- dim_customers
|
|
670
663
|
implementation:
|
|
671
664
|
materialization: table
|
|
672
665
|
grain: [month_key]
|
|
@@ -684,6 +677,12 @@ tables:
|
|
|
684
677
|
- ["2024-02", 15200.00]
|
|
685
678
|
- ["2024-03", 18900.75]
|
|
686
679
|
|
|
680
|
+
lineage: # data flow — separate from ER
|
|
681
|
+
- from: fct_orders
|
|
682
|
+
to: mart_monthly_revenue
|
|
683
|
+
- from: dim_customers
|
|
684
|
+
to: mart_monthly_revenue
|
|
685
|
+
|
|
687
686
|
relationships: # ER only — not for lineage
|
|
688
687
|
- from: { table: dim_customers, column: customer_key }
|
|
689
688
|
to: { table: fct_orders, column: customer_key }
|