@microsoft/rayfin-semantic-model-pack 1.0.0 → 2.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG +8 -0
- package/README.md +34 -10
- package/kit/hooks/use-query-factory-target.spec.ts +191 -0
- package/kit/hooks/use-query-factory-target.ts +123 -0
- package/{skills/analytics/kit → kit}/hooks/use-semantic-model-query.spec.ts +59 -23
- package/{skills/analytics/kit → kit}/hooks/use-semantic-model-query.ts +30 -54
- package/kit/lib/query-factory.spec.ts +97 -0
- package/kit/lib/query-factory.ts +68 -0
- package/kit/lib/semantic-model-errors.spec.ts +131 -0
- package/kit/lib/semantic-model-errors.ts +110 -0
- package/kit/lib/to-data-table.spec.ts +444 -0
- package/kit/lib/to-data-table.ts +221 -0
- package/pack.json +141 -0
- package/package.json +3 -1
- package/skills/analytics/SKILL.md +126 -421
- package/skills/dax-authoring/SKILL.md +99 -34
- package/skills/query-design/SKILL.md +22 -29
- package/skills/query-design/references/anti-patterns.md +13 -11
- package/skills/query-design/references/format-strings.md +6 -12
- package/skills/query-design/references/multi-grain-patterns.md +121 -108
- package/skills/schema-discovery/SKILL.md +197 -368
- package/skills/schema-discovery/references/discovery-queries.md +6 -8
- package/skills/visual-interactivity/SKILL.md +68 -0
- package/skills/visual-interactivity/references/advanced-query-shapes.md +155 -0
- package/skills/visual-interactivity/references/filter-aware-grid-totals.md +101 -0
- package/skills/visual-interactivity/references/highlight-queries.md +88 -0
- package/skills/visual-interactivity/references/interaction-manager.md +127 -0
- package/skills/visual-interactivity/references/native-selections.md +20 -0
- package/skills/visual-interactivity/references/query-factories.md +169 -0
- package/skills/analytics/kit/lib/connectors.ts +0 -94
- package/skills/analytics/kit/scripts/rayfin-connector.mjs +0 -265
- package/skills/analytics/kit/scripts/rayfin-connector.test.mjs +0 -243
- package/skills/analytics/kit/scripts/schema-sql-batch.test.mjs +0 -2097
- package/skills/analytics/kit/scripts/schema-sql-cli.test.mjs +0 -488
- package/skills/analytics/kit/scripts/schema-sql.mjs +0 -2650
- package/skills/analytics/kit/scripts/schema-sql.test.mjs +0 -1387
- package/skills/analytics/kit/scripts/schema.mjs +0 -166
- package/skills/analytics/kit/scripts/schema.test.mjs +0 -271
- package/skills/analytics/pack.json +0 -49
- package/skills/query-design/references/filter-strategy.md +0 -68
- package/skills/query-design/references/highlight-queries.md +0 -122
|
@@ -1,382 +1,211 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: schema-discovery
|
|
3
|
-
description:
|
|
4
|
-
Discover Power BI semantic-model tables, columns, measures, relationships,
|
|
5
|
-
and column statistics through the analytics pack's local SQLite catalog.
|
|
6
|
-
Use advanced INFO queries only when the fixed catalog does not contain the
|
|
7
|
-
required metadata.
|
|
3
|
+
description: Ground DAX work in the cached semantic-model catalog, shipped measures, relationships, and explicit filter values.
|
|
8
4
|
---
|
|
9
5
|
|
|
10
|
-
#
|
|
6
|
+
# Grounding semantic-model work
|
|
11
7
|
|
|
12
|
-
|
|
8
|
+
**Prerequisite:** register the task's semantic model as a Rayfin connector first:
|
|
9
|
+
`npx rayfin connector add --type fabric-semanticmodel --name <name> --workspace-id <workspaceId> --item-id <itemId>`.
|
|
10
|
+
Until then `query-schema` has no model to catalog and exits with that command.
|
|
13
11
|
|
|
14
|
-
|
|
15
|
-
[`analytics`](../analytics/SKILL.md) pack.
|
|
12
|
+
The semantic model's own catalog is cached as a local SQLite database and queried with SQL:
|
|
16
13
|
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
before planning around it. To turn it on, add the flag and re-apply:
|
|
20
|
-
|
|
21
|
-
```sh
|
|
22
|
-
# rayfin/.env
|
|
23
|
-
RAYFIN_FEATURE_FLAGS=schema-sqlite
|
|
24
|
-
```
|
|
25
|
-
|
|
26
|
-
```sh
|
|
27
|
-
npm run pack:add -- analytics
|
|
28
|
-
```
|
|
29
|
-
|
|
30
|
-
Without it, discover the model through the read-only queries in
|
|
31
|
-
[`discovery-queries.md`](./references/discovery-queries.md) instead, and say
|
|
32
|
-
that the catalog is unavailable rather than reporting the model as empty.
|
|
33
|
-
|
|
34
|
-
Apply the connectors and analytics packs, connect the intended semantic model,
|
|
35
|
-
then query the local catalog:
|
|
36
|
-
|
|
37
|
-
```sh
|
|
38
|
-
npm run --silent schema:sql -- "SELECT status, incomplete, notes FROM catalog_status"
|
|
39
|
-
npm run --silent schema:sql -- --search "requested business concept"
|
|
40
|
-
```
|
|
41
|
-
|
|
42
|
-
The first command builds the catalog automatically when it is missing or older
|
|
43
|
-
than 15 minutes.
|
|
44
|
-
An explicit rebuild is available when freshness matters:
|
|
45
|
-
|
|
46
|
-
```sh
|
|
47
|
-
npm run schema:refresh
|
|
48
|
-
```
|
|
49
|
-
|
|
50
|
-
One refresh runs five schema-only source queries from a hardcoded list, plus
|
|
51
|
-
retries if the model throttles:
|
|
52
|
-
|
|
53
|
-
- `INFO.VIEW.TABLES()`
|
|
54
|
-
- `INFO.VIEW.COLUMNS()`
|
|
55
|
-
- `INFO.VIEW.MEASURES()`
|
|
56
|
-
- `INFO.VIEW.RELATIONSHIPS()`
|
|
57
|
-
- `COLUMNSTATISTICS()`
|
|
58
|
-
|
|
59
|
-
The builder does not accept agent-supplied DAX.
|
|
60
|
-
Business-data queries cannot enter the catalog-build path.
|
|
61
|
-
After those five requests, discovery is read-only SQL against local SQLite.
|
|
62
|
-
|
|
63
|
-
This resolves the apparent conflict between full and progressive discovery.
|
|
64
|
-
The fixed sources are materialized once for local reuse, but each agent result
|
|
65
|
-
must still be a narrow SQL projection containing only evidence needed for the
|
|
66
|
-
current task.
|
|
67
|
-
Do not dump whole catalog tables into the conversation.
|
|
68
|
-
|
|
69
|
-
## Catalog contents
|
|
70
|
-
|
|
71
|
-
The database is `.schema/models/<itemId>/model.sqlite`.
|
|
72
|
-
Its tables preserve the model's native DAX column names:
|
|
73
|
-
|
|
74
|
-
| SQLite table | Contents |
|
|
75
|
-
| --- | --- |
|
|
76
|
-
| `catalog_status` | Persistent `complete` or `incomplete` build status and notes |
|
|
77
|
-
| `info_view_tables` | One row per semantic-model table |
|
|
78
|
-
| `info_view_columns` | One row per model column |
|
|
79
|
-
| `info_view_measures` | One row per shipped measure |
|
|
80
|
-
| `info_view_relationships` | One row per relationship |
|
|
81
|
-
| `column_statistics` | Actual Min/Max values, cardinality, and maximum text length |
|
|
82
|
-
| `semantic_search` | Searchable projection of tables, columns, measures, and relationships |
|
|
83
|
-
|
|
84
|
-
`column_statistics` is derived from real model rows.
|
|
85
|
-
The catalog is customer data, not metadata-only content.
|
|
86
|
-
The `.schema/` directory is gitignored and must never be copied, committed, or
|
|
87
|
-
exposed.
|
|
88
|
-
|
|
89
|
-
Read `.schema/models/<itemId>/catalog.md` once when orientation is needed.
|
|
90
|
-
It records every table and column and prominently marks an incomplete catalog.
|
|
91
|
-
|
|
92
|
-
## Completeness and truncation
|
|
93
|
-
|
|
94
|
-
Catalog completeness and SQL-result completeness are separate:
|
|
95
|
-
|
|
96
|
-
- `catalog_status.status = 'incomplete'` means at least one source view failed.
|
|
97
|
-
- `truncated: true` means the local SQL output was bounded by row count or byte size.
|
|
98
|
-
- `truncated: false` only means that one SQL result is complete.
|
|
99
|
-
It does not prove the catalog itself is complete.
|
|
100
|
-
|
|
101
|
-
Always check `catalog_status` before using absence as evidence.
|
|
102
|
-
If the catalog is incomplete, read its notes and run `npm run schema:refresh`
|
|
103
|
-
once to retry.
|
|
104
|
-
Concurrent refreshes serialize per model and reuse a complete catalog built by
|
|
105
|
-
the process that won the lock.
|
|
106
|
-
An incomplete refresh never replaces an existing complete catalog.
|
|
107
|
-
If it remains incomplete, continue only with the views that are present and
|
|
108
|
-
state which evidence is missing.
|
|
109
|
-
Do not bypass an incomplete catalog with exploratory `INFO.VIEW.*` calls.
|
|
110
|
-
|
|
111
|
-
`complete` describes which source views answered, not how much of each one you
|
|
112
|
-
were allowed to see. `INFO.VIEW.MEASURES()` returns `Expression` only to a
|
|
113
|
-
caller with write permission, so on a model you have Build access to, measure
|
|
114
|
-
expressions can come back blank while the catalog still reads `complete`. Treat
|
|
115
|
-
a blank `[Expression]` as unknown rather than as proof the measure has no
|
|
116
|
-
definition, and say so instead of rebuilding its logic from physical columns.
|
|
117
|
-
|
|
118
|
-
A SQL result with `truncated: true` cannot be used to choose model objects or
|
|
119
|
-
author DAX.
|
|
120
|
-
Narrow the `WHERE` clause and, for ordinary catalog tables, the projection.
|
|
121
|
-
Queries selecting model tables automatically append compact owned-measure names
|
|
122
|
-
and display folders, so a narrower projection does not reduce that enrichment;
|
|
123
|
-
narrow the selected table rows instead.
|
|
124
|
-
Rerun until the needed evidence is in a result with `truncated: false`.
|
|
125
|
-
Do not use `--allow-truncated` during discovery or DAX authoring.
|
|
126
|
-
|
|
127
|
-
### Reading a single expression that is larger than the budget
|
|
128
|
-
|
|
129
|
-
One cell can exceed the byte budget on its own — localization measures built
|
|
130
|
-
from `SWITCH(USERCULTURE(), ...)` routinely run to tens of thousands of
|
|
131
|
-
characters.
|
|
132
|
-
Narrowing cannot help there: a single cell has nothing left to narrow, so the
|
|
133
|
-
same `SELECT` returns `totalRows: 1` with `returnedRows: 0` however often it is
|
|
134
|
-
repeated.
|
|
135
|
-
|
|
136
|
-
Read it in slices instead, and fetch the length first so the last offset is
|
|
137
|
-
known.
|
|
138
|
-
Use `length([Expression])` as the bound rather than the character count in the
|
|
139
|
-
search summary: `substr` offsets and `length` both count code points, while the
|
|
140
|
-
summary counts UTF-16 units, so the two disagree for text containing emoji.
|
|
141
|
-
|
|
142
|
-
```sh
|
|
143
|
-
npm run --silent schema:sql -- "SELECT length([Expression]) AS chars FROM info_view_measures WHERE [Table] = 'Sales' AND [Name] = 'Localized Label'"
|
|
144
|
-
npm run --silent schema:sql -- "SELECT substr([Expression], 1, 1000) AS part FROM info_view_measures WHERE [Table] = 'Sales' AND [Name] = 'Localized Label'"
|
|
145
|
-
npm run --silent schema:sql -- "SELECT substr([Expression], 1001, 1000) AS part FROM info_view_measures WHERE [Table] = 'Sales' AND [Name] = 'Localized Label'"
|
|
146
|
-
```
|
|
147
|
-
|
|
148
|
-
Use a stride of 1000 characters.
|
|
149
|
-
The budget is counted in bytes, not characters, so a stride that fits ASCII can
|
|
150
|
-
still be three or four times too large for CJK or emoji text.
|
|
151
|
-
|
|
152
|
-
Two conditions both have to hold, and the second is the one that is easy to get
|
|
153
|
-
wrong:
|
|
154
|
-
|
|
155
|
-
- Every chunk returns `returnedRows: 1` and `truncated: false`.
|
|
156
|
-
`returnedRows: 0` means the slice was still too big — halve the stride and
|
|
157
|
-
retry that offset.
|
|
158
|
-
- The offsets cover `1` through the full length with no gap.
|
|
159
|
-
Stopping after one clean chunk looks like success, because the chunks that
|
|
160
|
-
were fetched are all complete; only the coverage is missing.
|
|
161
|
-
|
|
162
|
-
Concatenate the parts in offset order to reconstruct the definition exactly.
|
|
163
|
-
|
|
164
|
-
## Discovery workflow
|
|
165
|
-
|
|
166
|
-
1. Query `catalog_status`.
|
|
167
|
-
2. Run a catalog-wide semantic search for the user's concepts.
|
|
168
|
-
3. Inspect candidate tables and their owned measures.
|
|
169
|
-
4. Query focused columns and measures for every selected table.
|
|
170
|
-
5. Query relationships connecting the selected tables.
|
|
171
|
-
6. Use column statistics only after the exact column is known.
|
|
172
|
-
7. Reuse prior catalog results instead of issuing equivalent SQL again.
|
|
173
|
-
8. Stop only when every requested output has an exact source mapping, and the
|
|
174
|
-
shipped measures, grouping columns, relationship path, **filter grain** and
|
|
175
|
-
**explicit filter values** are all grounded in untruncated results.
|
|
176
|
-
|
|
177
|
-
The catalog holds metadata and per-column extrema, not the distinct values in a
|
|
178
|
-
column. Once the filter column is known and a literal value still has to be
|
|
179
|
-
confirmed, ground it with one bounded query against the model rather than
|
|
180
|
-
looping on catalog SQL that cannot answer it:
|
|
181
|
-
|
|
182
|
-
```sh
|
|
183
|
-
npx rayfin connector invoke <alias> executeQuery --file <probe>.query.json
|
|
184
|
-
```
|
|
185
|
-
|
|
186
|
-
Test the literal you were given, so the answer is decisive:
|
|
187
|
-
|
|
188
|
-
```dax
|
|
189
|
-
EVALUATE
|
|
190
|
-
FILTER(VALUES('Geography'[Country]), 'Geography'[Country] = "France")
|
|
191
|
-
```
|
|
192
|
-
|
|
193
|
-
Zero rows means there is no exact match for that value; report that instead of
|
|
194
|
-
widening the search. If the probe errors, or the mapping is still unresolved
|
|
195
|
-
after it, report the blocker rather than broadening discovery. Do not sample a
|
|
196
|
-
column and read absence from the sample — `TOPN` over a column with more
|
|
197
|
-
distinct values than the bound returns a complete-looking partial answer. This
|
|
198
|
-
is a value check, not more discovery: ask it once, for a column you have
|
|
199
|
-
already selected.
|
|
200
|
-
|
|
201
|
-
Before writing any DAX, write down the output plan: every output field with its
|
|
202
|
-
exact source (a fully qualified column, or an exact shipped measure for every
|
|
203
|
-
business value), the filters and their values, the relationship path, and the
|
|
204
|
-
expected row shape. Then compare what the query returns against that plan.
|
|
205
|
-
A query that runs is not the same as a query at the right grain.
|
|
206
|
-
|
|
207
|
-
One grain trap is worth checking by name. When a measure counts a column —
|
|
208
|
-
`COUNT`, `DISTINCTCOUNT` or `DISTINCTCOUNTNOBLANK` over `'Table'[Column]` —
|
|
209
|
-
filtering or grouping by that same column, or by one that reaches it through an
|
|
210
|
-
active relationship, collapses the result to a trivial one per group. It
|
|
211
|
-
returns rows and looks correct. The catalog already holds both inputs: read the
|
|
212
|
-
measure's `[Expression]` from `info_view_measures` and the path from
|
|
213
|
-
`info_view_relationships` before committing to that filter. If two filter
|
|
214
|
-
columns remain plausible, compare them once with the shipped measures
|
|
215
|
-
unchanged.
|
|
216
|
-
|
|
217
|
-
Start with shipped measures.
|
|
218
|
-
Never reconstruct business logic from physical columns when the model already
|
|
219
|
-
ships a measure.
|
|
220
|
-
Column statistics are not a shortcut around that: `Min`, `Max` and
|
|
221
|
-
`Cardinality` are the raw column extent, ignoring filter context,
|
|
222
|
-
relationships, and any business rule the measure encodes. A cardinality that
|
|
223
|
-
happens to match an expected row count is not evidence that reading the column
|
|
224
|
-
is correct.
|
|
225
|
-
Helper, selector, calculation, and parameter tables may own the measure that
|
|
226
|
-
supplies a displayed value, so do not restrict the first search to a fact-table
|
|
227
|
-
prefix.
|
|
228
|
-
|
|
229
|
-
A selected table is not fully inspected until its owned measures have been
|
|
230
|
-
read. When such a table owns a measure that dispatches on its column — a
|
|
231
|
-
`SELECTEDVALUE` / `SWITCH` over the selector, for example — group by that
|
|
232
|
-
column and evaluate the shipped measure once instead of building one expression
|
|
233
|
-
per row out of similarly named physical columns. That recipe does not apply to
|
|
234
|
-
every helper: a native field-parameter table carries field references that the
|
|
235
|
-
client resolves, so grouping by its label does not swap the underlying field.
|
|
236
|
-
If the owned measures do not explain what the table is for, search measure
|
|
237
|
-
expressions for references to it before falling back to raw aggregation:
|
|
238
|
-
|
|
239
|
-
```sh
|
|
240
|
-
npm run --silent schema:sql -- "SELECT [Table], [Name] FROM info_view_measures WHERE [Expression] LIKE '%ParameterTable%'"
|
|
241
|
-
```
|
|
242
|
-
|
|
243
|
-
Selecting from `info_view_tables` automatically includes a bounded
|
|
244
|
-
`ownedMeasures` preview when `info_view_measures` is available.
|
|
245
|
-
The preview carries names and display folders only so it cannot consume the
|
|
246
|
-
result byte budget with repeated expressions.
|
|
247
|
-
Query `info_view_measures` for each selected table to read expressions,
|
|
248
|
-
descriptions, and format strings.
|
|
249
|
-
Review it before choosing raw columns.
|
|
250
|
-
|
|
251
|
-
## SQL examples
|
|
252
|
-
|
|
253
|
-
Search globally before narrowing:
|
|
254
|
-
|
|
255
|
-
```sh
|
|
256
|
-
npm run --silent schema:sql -- --search "revenue region comparison"
|
|
257
|
-
```
|
|
258
|
-
|
|
259
|
-
Search is an AND across names, descriptions, expressions, and display folders,
|
|
260
|
-
so prefer a few specific terms over a whole sentence.
|
|
261
|
-
When no object matches every term, terms that appear nowhere in the model are
|
|
262
|
-
dropped and the search is answered without them, and the result says which ones
|
|
263
|
-
were dropped.
|
|
264
|
-
A statement sourced with `--file` must live inside the app, and is labelled by
|
|
265
|
-
its path rather than by its contents.
|
|
266
|
-
|
|
267
|
-
Inspect candidate tables:
|
|
268
|
-
|
|
269
|
-
```sh
|
|
270
|
-
npm run --silent schema:sql -- "SELECT [Name], [Description], [IsHidden] FROM info_view_tables WHERE lower([Name]) LIKE '%sales%'"
|
|
14
|
+
```bash
|
|
15
|
+
npx fabric-app-semantic-model-tools query-schema "SELECT [Name], [Expression] FROM info_view_measures WHERE [Table] = 'Sales'"
|
|
271
16
|
```
|
|
272
17
|
|
|
273
|
-
|
|
18
|
+
Quote identifiers in inline SQL with SQLite brackets (`[Table]`, `[Max Length]`), which pass
|
|
19
|
+
unchanged through both bash and PowerShell. Inline SQL is fine for model identifiers and literals
|
|
20
|
+
quoted as `[...]` / `'...'` (double any `'` inside a literal), including LIKE patterns with `%`,
|
|
21
|
+
even when they came from earlier catalog output. Put the SQL in a `.sql` file and pass
|
|
22
|
+
`--file <file.sql>` only when a name or value contains `"`, `` ` ``, `$`, `\`, or a newline, or
|
|
23
|
+
when it embeds free-form text from the user. In a LIKE pattern, don't write a whole environment
|
|
24
|
+
variable name between two `%` signs (e.g. `%PATH%`, `%DATE%`); use
|
|
25
|
+
`instr(lower([Name]), 'path') > 0` instead.
|
|
274
26
|
|
|
275
|
-
|
|
276
|
-
|
|
277
|
-
```
|
|
27
|
+
`--search` and `--sql` are repeatable and run in one process, so ask everything you already know
|
|
28
|
+
you need in one command rather than one command per question:
|
|
278
29
|
|
|
279
|
-
|
|
280
|
-
|
|
281
|
-
```sh
|
|
282
|
-
npm run --silent schema:sql -- "SELECT [Table], [Name], [DataType], [Description], [FormatString], [IsHidden] FROM info_view_columns WHERE [Table] IN ('Sales', 'Date') ORDER BY [Table], [Name]"
|
|
30
|
+
```bash
|
|
31
|
+
npx fabric-app-semantic-model-tools query-schema --search "revenue margin" --sql "SELECT [Name], [Expression] FROM info_view_measures WHERE [Table] = 'Sales'" --sql "SELECT [Name], [DataType] FROM info_view_columns WHERE [Table] = 'Sales'"
|
|
283
32
|
```
|
|
284
33
|
|
|
285
|
-
|
|
286
|
-
|
|
287
|
-
|
|
288
|
-
|
|
289
|
-
```
|
|
290
|
-
|
|
291
|
-
Inspect statistics for exact columns:
|
|
292
|
-
|
|
293
|
-
```sh
|
|
294
|
-
npm run --silent schema:sql -- "SELECT [Table Name], [Column Name], [Min], [Max], [Cardinality], [Max Length] FROM column_statistics WHERE [Table Name] = 'Date' AND [Column Name] IN ('Date', 'Year')"
|
|
295
|
-
```
|
|
296
|
-
|
|
297
|
-
Column names retain spaces and reserved words.
|
|
298
|
-
Quote them with SQL double quotes or square brackets.
|
|
299
|
-
|
|
300
|
-
## Batch several statements in one invocation
|
|
301
|
-
|
|
302
|
-
Each extra invocation costs a whole model round trip, and a round trip re-sends
|
|
303
|
-
the entire conversation.
|
|
304
|
-
The five statements above are normally run back to back, so run them together
|
|
305
|
-
instead.
|
|
306
|
-
`--search`, `--sql`, and `--file` are repeatable, and searches run before SQL:
|
|
307
|
-
|
|
308
|
-
```sh
|
|
309
|
-
npm run --silent schema:sql -- --search "revenue region comparison" --sql "SELECT [Name], [Description], [IsHidden] FROM info_view_tables WHERE lower([Name]) LIKE '%sales%'" --sql "SELECT [Table], [Name], [Expression], [Description], [FormatString], [DisplayFolder] FROM info_view_measures WHERE [Table] IN ('Sales', 'Date') ORDER BY [Table], [Name]"
|
|
310
|
-
```
|
|
311
|
-
|
|
312
|
-
The reply is one payload: a `#` header naming the batch, then a `##` label line
|
|
313
|
-
per statement followed by that statement's ordinary result block.
|
|
314
|
-
A failed statement stays in the payload beside the ones that worked, so a single
|
|
315
|
-
mistake does not cost the whole batch.
|
|
316
|
-
|
|
317
|
-
At most eight statements per invocation.
|
|
318
|
-
The whole batch shares one byte budget, so batching never costs more context
|
|
319
|
-
than a single statement would.
|
|
320
|
-
Ask everything you already know you need, but keep each statement narrow: the
|
|
321
|
-
shared budget is divided across them, and unused room rolls forward.
|
|
322
|
-
|
|
323
|
-
## Advanced metadata outside the catalog
|
|
324
|
-
|
|
325
|
-
The fixed catalog intentionally covers the common DAX-authoring path.
|
|
326
|
-
It does not include every elevated `INFO.*` function.
|
|
327
|
-
|
|
328
|
-
Only when the current task specifically requires calculation groups, calendars,
|
|
329
|
-
user-defined functions, variations, or model configuration should you use the
|
|
330
|
-
advanced read-only queries in
|
|
331
|
-
[`discovery-queries.md`](./references/discovery-queries.md).
|
|
332
|
-
Run the minimum advanced query needed through:
|
|
333
|
-
|
|
334
|
-
```sh
|
|
335
|
-
npx rayfin connector invoke <alias> executeQuery --file <query>.query.json
|
|
336
|
-
```
|
|
337
|
-
|
|
338
|
-
The file is a JSON operation input such as:
|
|
339
|
-
|
|
340
|
-
```json
|
|
341
|
-
{
|
|
342
|
-
"query": "EVALUATE INFO.CALCULATIONGROUPS()"
|
|
343
|
-
}
|
|
344
|
-
```
|
|
34
|
+
Each invocation costs a full model round trip, and a round trip re-sends the whole conversation, so
|
|
35
|
+
three separate commands cost roughly three times what one batch does. Batch the questions you can
|
|
36
|
+
write down now; keep a question whose *content* depends on an earlier answer for the next command.
|
|
37
|
+
Up to 8 statements per invocation.
|
|
345
38
|
|
|
346
|
-
|
|
347
|
-
|
|
348
|
-
|
|
349
|
-
|
|
350
|
-
|
|
351
|
-
|
|
352
|
-
|
|
353
|
-
|
|
354
|
-
|
|
355
|
-
|
|
356
|
-
|
|
357
|
-
|
|
358
|
-
|
|
359
|
-
|
|
360
|
-
|
|
361
|
-
|
|
362
|
-
|
|
363
|
-
|
|
364
|
-
|
|
365
|
-
|
|
366
|
-
|
|
367
|
-
|
|
368
|
-
|
|
369
|
-
-
|
|
370
|
-
|
|
371
|
-
|
|
372
|
-
|
|
373
|
-
|
|
374
|
-
|
|
375
|
-
|
|
376
|
-
|
|
377
|
-
|
|
378
|
-
|
|
379
|
-
|
|
380
|
-
|
|
381
|
-
|
|
382
|
-
|
|
39
|
+
The whole batch shares one result-size budget — the same one a single statement gets — divided
|
|
40
|
+
between the statements, with each statement's unused share rolling forward to the next. So batch
|
|
41
|
+
**focused** statements. A statement that returns a lot is truncated to its own share; later
|
|
42
|
+
statements keep their share of the budget. Run a query you expect to return a lot in its own
|
|
43
|
+
command so it gets the whole budget.
|
|
44
|
+
|
|
45
|
+
Each statement's result is labelled with the query that produced it, searches first, then SQL. A
|
|
46
|
+
statement that fails stays in the output beside the ones that succeeded. The exit code is the worst
|
|
47
|
+
outcome across the batch: 1 if any statement failed, 2 if any was truncated, 0 otherwise.
|
|
48
|
+
|
|
49
|
+
The database builds itself on first use and rebuilds when stale, so ask a question rather than
|
|
50
|
+
refreshing it by hand. `.schema/models/<itemId>/catalog.md` names every table, its real row and
|
|
51
|
+
column counts, and how the tables join; read it once and the rest is ordinary SQL.
|
|
52
|
+
|
|
53
|
+
The tables hold the untouched output of `INFO.VIEW.TABLES()`, `INFO.VIEW.COLUMNS()`,
|
|
54
|
+
`INFO.VIEW.MEASURES()`, `INFO.VIEW.RELATIONSHIPS()` and `COLUMNSTATISTICS()`, under the model's own
|
|
55
|
+
column names. Nothing is renamed, flattened, or summarized, so a column you see in the catalog is a
|
|
56
|
+
column DAX returned.
|
|
57
|
+
|
|
58
|
+
Besides metadata, the catalog stores the `COLUMNSTATISTICS()` output, including each column's
|
|
59
|
+
`Min`/`Max` sample values. It lives locally under `.schema/models/`, which is git-ignored and
|
|
60
|
+
rebuilt on demand; SQL queries read only that local copy.
|
|
61
|
+
|
|
62
|
+
Do not hand-run exploratory `INFO.*` queries, read the catalog database directly, bypass the catalog
|
|
63
|
+
with connector probes, or ask the user for metadata the catalog can answer.
|
|
64
|
+
|
|
65
|
+
Check `catalog_status` and `catalog.md` for completeness before treating absence
|
|
66
|
+
as evidence. A failed view differs from a successful zero-row view.
|
|
67
|
+
An explicit `--refresh` retries now but never replaces a complete catalog with
|
|
68
|
+
a build that loses a view. Empty Direct Lake statistics are expected; do not
|
|
69
|
+
reconstruct them with expensive MIN/MAX/DISTINCTCOUNT queries.
|
|
70
|
+
|
|
71
|
+
For long measure expressions, read bounded chunks with
|
|
72
|
+
`SELECT substr([Expression], 1, 2000) ...`, then offsets 2001, 4001, etc.,
|
|
73
|
+
using a focused WHERE clause until the expression is complete.
|
|
74
|
+
An empty Expression on an object that should expose one means detail is unavailable,
|
|
75
|
+
not proof that no logic exists. Search reports dropped terms; refine them rather
|
|
76
|
+
than treating broad or punctuation-only searches as absence.
|
|
77
|
+
For advanced metadata (calculation groups, calendars, UDFs) not in the five catalog
|
|
78
|
+
views, follow [discovery-queries.md](references/discovery-queries.md) narrowly;
|
|
79
|
+
do not use that exception to bypass ordinary cached discovery.
|
|
80
|
+
|
|
81
|
+
## Querying
|
|
82
|
+
|
|
83
|
+
Every query starts with result metadata including `totalRows`, `returnedRows`, and `truncated`.
|
|
84
|
+
Zero rows is a successful answer, not an error; it means the model has no such entity.
|
|
85
|
+
|
|
86
|
+
Results are capped by row count and by size, so filter in SQL instead of selecting whole tables and
|
|
87
|
+
reading the overflow. **A result with `truncated: true` is incomplete and cannot be used to choose
|
|
88
|
+
model objects or author DAX.** The command exits with code 2 after printing the bounded preview.
|
|
89
|
+
Narrow the `WHERE` clause and/or projection and rerun until the entities needed by the plan appear
|
|
90
|
+
in a result with `truncated: false`. `SELECT *` on `info_view_measures` will truncate on a large
|
|
91
|
+
model; selecting only the required metadata and filtering by table, name, folder, description, or
|
|
92
|
+
expression will not.
|
|
93
|
+
|
|
94
|
+
The connection is read-only. A failing query comes back as an error message, so a malformed query
|
|
95
|
+
costs one retry, not a corrupted cache.
|
|
96
|
+
|
|
97
|
+
## Column statistics
|
|
98
|
+
|
|
99
|
+
`column_statistics` carries `Min`, `Max`, `Cardinality` and `Max Length` for each column. Use them
|
|
100
|
+
to choose grouping columns, bound date and numeric filters, and anticipate result size.
|
|
101
|
+
|
|
102
|
+
They are not a substitute for the model's logic. When a shipped measure computes the same quantity,
|
|
103
|
+
use the measure: a statistic is the raw column extent, ignoring filter context, relationships, and
|
|
104
|
+
any business rule the measure encodes. A column whose cardinality happens to match an expected row
|
|
105
|
+
count is not evidence that reading it is correct.
|
|
106
|
+
|
|
107
|
+
If `column_statistics` is missing from the catalog, the model did not return statistics. Continue
|
|
108
|
+
without them.
|
|
109
|
+
|
|
110
|
+
## Global semantic discovery
|
|
111
|
+
|
|
112
|
+
Search the whole catalog before restricting discovery to a domain-looking table prefix. Helper,
|
|
113
|
+
parameter, selector, calculation, and metadata tables may have names that do not share the user's
|
|
114
|
+
term or the primary fact-table prefix. Search table names and descriptions, measure names,
|
|
115
|
+
descriptions, expressions, and display folders, and column names and descriptions in one query.
|
|
116
|
+
Start with the ranked catalog-wide search, replacing the terms with the user's concepts:
|
|
117
|
+
|
|
118
|
+
```bash
|
|
119
|
+
npx fabric-app-semantic-model-tools query-schema --search "requested concept comparison"
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
Every search token must appear somewhere in an entity's name, parent table, description, display
|
|
123
|
+
folder, expression, or type. If there are no all-term matches, model-wide absent terms may
|
|
124
|
+
be dropped only with an explicit diagnostic in the result. Exact and prefix name matches rank above descriptive or expression
|
|
125
|
+
matches, and shipped measures rank above columns at the same lexical relevance. After the global
|
|
126
|
+
result identifies candidate tables, run a focused, untruncated SQL query for their measures,
|
|
127
|
+
columns, descriptions, and expressions. Do not begin with a domain-prefix restriction: that can
|
|
128
|
+
hide a helper, parameter, selector, or measure owned elsewhere.
|
|
129
|
+
|
|
130
|
+
Search is a findability surface, not a reading surface. Ranking sees every field in full, but the
|
|
131
|
+
printed rows omit a `description` that is identical across the model and shorten a long
|
|
132
|
+
`expression` to its head plus its length. The result's `trimmed` counts say when either happened.
|
|
133
|
+
Both fields are still matched, so nothing becomes unfindable — `SELECT` the row when you need the
|
|
134
|
+
full text. Do not treat a shortened expression as the measure's definition.
|
|
135
|
+
|
|
136
|
+
## Model-facing workflow
|
|
137
|
+
|
|
138
|
+
1. Read `catalog.md` once when orientation is needed. Do not fetch it again in the same task.
|
|
139
|
+
2. Run global semantic discovery across tables, measures, and columns before applying a
|
|
140
|
+
domain-specific table-name restriction.
|
|
141
|
+
3. Start with shipped measures. Whenever a table becomes part of the plan, inspect measures owned
|
|
142
|
+
by that table before authoring DAX. Query all selected tables in one statement using `IN` or
|
|
143
|
+
`OR`, returning at least `[Table]`, `[Name]`, `[Expression]`, `[Description]`,
|
|
144
|
+
`[FormatString]`, and `[DisplayFolder]`. Never reconstruct business logic from columns when the
|
|
145
|
+
model already ships the measure. A single-table query on `info_view_tables` (no JOIN, comma
|
|
146
|
+
join, or UNION) automatically includes bounded `ownedMeasures` for each returned table; review
|
|
147
|
+
those before choosing physical columns.
|
|
148
|
+
4. Answer several known metadata questions in one command. Combine them into one statement with a
|
|
149
|
+
join or an `OR` where they share a shape; where they do not, pass them as repeated `--sql`
|
|
150
|
+
(and `--search`) flags in a single invocation. Either way it is one round trip. One command per
|
|
151
|
+
question is the single most expensive habit in discovery.
|
|
152
|
+
5. Take explicit filter values from the request, and bind every other qualifier in it to a filter
|
|
153
|
+
too. A qualifier can imply a value the request never states: the complement of a named value,
|
|
154
|
+
one section of a parameter or selector table, a status or flag. Take such values from the model
|
|
155
|
+
(a parameter table's `[Expression]`, a column's `column_statistics` `[Min]`/`[Max]`), not from
|
|
156
|
+
guesses. Apart from those statistics the catalog holds no other row values, but a
|
|
157
|
+
parameter or calculated table (for example a `DATATABLE`) shows its literal rows in
|
|
158
|
+
`info_view_tables.[Expression]`; read that expression before inventing labels or keys. Do not
|
|
159
|
+
probe the model for row values during discovery. When a value needs confirmation, add one small
|
|
160
|
+
value-check query file
|
|
161
|
+
(for example `EVALUATE FILTER(VALUES('Customer'[Country]), 'Customer'[Country] = "Canada")`) to
|
|
162
|
+
the same final `run-dax-query` call. The value check counts toward that call's four-file limit;
|
|
163
|
+
when the plan already has four queries, fold the check into one of them or run it in the next
|
|
164
|
+
call. An empty value-check result is the signal to correct that value once.
|
|
165
|
+
6. Before another schema request, check whether an earlier result already answers the question.
|
|
166
|
+
Never rerun an identical or equivalent schema query.
|
|
167
|
+
7. Scope discovery to the current visual batch, reusing earlier mappings. Stop only after
|
|
168
|
+
every output field in that batch has an exact source mapping; the shipped
|
|
169
|
+
measures, grouping columns, relationship path, and filter grain are grounded; every explicit
|
|
170
|
+
filter value is bound to its exact column (its value comes from the request and, if needed, is
|
|
171
|
+
confirmed by the value check in step 5, not by more discovery); every qualifier in the request
|
|
172
|
+
relevant to that batch is bound to a filter, including implied ones (step 5); and no mapping
|
|
173
|
+
depends on a truncated result.
|
|
174
|
+
8. Write and validate the DAX with the [dax-authoring](../dax-authoring/SKILL.md) skill.
|
|
175
|
+
|
|
176
|
+
## DAX plan and semantic grain
|
|
177
|
+
|
|
178
|
+
Before writing DAX for the current visual batch, form a compact plan containing:
|
|
179
|
+
|
|
180
|
+
- every requested output field;
|
|
181
|
+
- the exact source for each field: a fully qualified grouping/label column, an exact shipped
|
|
182
|
+
measure name for every business value, or an explicit literal;
|
|
183
|
+
- exact filters and values, including the ones the request implies rather than names;
|
|
184
|
+
- relevant relationship path;
|
|
185
|
+
- expected output columns and row shape;
|
|
186
|
+
- unresolved filter candidates, if any.
|
|
187
|
+
|
|
188
|
+
Keep this plan in your working notes; do not write it to a file. `run-dax-query` does not know the
|
|
189
|
+
plan, so compare the columns, row count, and warnings it reports with the plan yourself.
|
|
190
|
+
|
|
191
|
+
Do not generate DAX until every business value needed by this visual is bound to an exact shipped measure and
|
|
192
|
+
every label/group is bound to an exact column. If exhaustive, untruncated measure discovery
|
|
193
|
+
establishes that no suitable measure exists, record both the missing measure and the catalog
|
|
194
|
+
evidence before declaring a different source. Partial measure results are not evidence that a
|
|
195
|
+
measure is absent.
|
|
196
|
+
|
|
197
|
+
A selected table is not fully inspected until its owned measures have been reviewed. This is
|
|
198
|
+
especially important for helper, parameter, selector, calculation, and metadata tables: their rows
|
|
199
|
+
may provide labels or filter context while an owned measure returns the requested value. Group by
|
|
200
|
+
the helper column and evaluate the shipped measure; do not invent one expression per row from
|
|
201
|
+
similarly named physical columns. If owned measures do not explain the table's semantic role,
|
|
202
|
+
search measure expressions for references to that table before falling back to raw aggregation.
|
|
203
|
+
|
|
204
|
+
If a catalog refresh fails, or would drop a view the previous catalog has (for example a throttled
|
|
205
|
+
`INFO.VIEW.MEASURES()`), the previous database is kept rather than replaced by the partial
|
|
206
|
+
build, the query is answered from it, and stderr prints a warning naming the catalog's age and
|
|
207
|
+
what failed. A refresh that is incomplete only in views the previous catalog also lacked is
|
|
208
|
+
published with an incomplete-catalog warning. Continue discovery from that cache. Treat the warning
|
|
209
|
+
as a reason to avoid freshness-sensitive assumptions, not as permission to ignore the cached
|
|
210
|
+
catalog. When no previous catalog exists, a partial build is still used and a warning says which
|
|
211
|
+
views are missing.
|
|
@@ -2,14 +2,12 @@
|
|
|
2
2
|
|
|
3
3
|
Read-only DAX queries for metadata exploration using `INFO.VIEW.*` and `INFO.*` rowsets.
|
|
4
4
|
|
|
5
|
-
The
|
|
6
|
-
column statistics into
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
Use the advanced `INFO.*` sections only when the task requires metadata outside
|
|
12
|
-
the fixed catalog.
|
|
5
|
+
The semantic-model tools materialize tables, columns, measures, relationships,
|
|
6
|
+
and column statistics into the local SQLite catalog, always available for this
|
|
7
|
+
capability. Common `INFO.VIEW.*` examples below are output-shape reference only:
|
|
8
|
+
do not execute them, including the scope-estimation probe.
|
|
9
|
+
Use advanced `INFO.*` sections only for required metadata outside the fixed
|
|
10
|
+
catalog. A permission failure stops the remaining elevated queries.
|
|
13
11
|
|
|
14
12
|
## Scope Estimation Queries
|
|
15
13
|
|