@microsoft/rayfin-semantic-model-pack 1.1.0 → 2.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (39) hide show
  1. package/CHANGELOG +9 -0
  2. package/README.md +16 -5
  3. package/kit/hooks/use-query-factory-target.spec.ts +191 -0
  4. package/kit/hooks/use-query-factory-target.ts +123 -0
  5. package/kit/hooks/use-semantic-model-query.spec.ts +51 -34
  6. package/kit/hooks/use-semantic-model-query.ts +30 -49
  7. package/kit/lib/query-factory.spec.ts +97 -0
  8. package/kit/lib/query-factory.ts +68 -0
  9. package/kit/lib/semantic-model-errors.spec.ts +131 -0
  10. package/kit/lib/semantic-model-errors.ts +110 -0
  11. package/kit/lib/to-data-table.spec.ts +444 -0
  12. package/kit/lib/to-data-table.ts +221 -0
  13. package/pack.json +122 -25
  14. package/package.json +1 -1
  15. package/skills/analytics/SKILL.md +126 -422
  16. package/skills/dax-authoring/SKILL.md +98 -31
  17. package/skills/query-design/SKILL.md +22 -29
  18. package/skills/query-design/references/anti-patterns.md +13 -11
  19. package/skills/query-design/references/format-strings.md +6 -12
  20. package/skills/query-design/references/multi-grain-patterns.md +121 -108
  21. package/skills/schema-discovery/SKILL.md +197 -368
  22. package/skills/schema-discovery/references/discovery-queries.md +6 -8
  23. package/skills/visual-interactivity/SKILL.md +68 -0
  24. package/skills/visual-interactivity/references/advanced-query-shapes.md +155 -0
  25. package/skills/visual-interactivity/references/filter-aware-grid-totals.md +101 -0
  26. package/skills/visual-interactivity/references/highlight-queries.md +89 -0
  27. package/skills/visual-interactivity/references/interaction-manager.md +127 -0
  28. package/skills/visual-interactivity/references/native-selections.md +20 -0
  29. package/skills/visual-interactivity/references/query-factories.md +197 -0
  30. package/kit/scripts/rayfin-connector.mjs +0 -265
  31. package/kit/scripts/rayfin-connector.test.mjs +0 -243
  32. package/kit/scripts/schema-sql-batch.test.mjs +0 -2097
  33. package/kit/scripts/schema-sql-cli.test.mjs +0 -488
  34. package/kit/scripts/schema-sql.mjs +0 -2650
  35. package/kit/scripts/schema-sql.test.mjs +0 -1387
  36. package/kit/scripts/schema.mjs +0 -166
  37. package/kit/scripts/schema.test.mjs +0 -271
  38. package/skills/query-design/references/filter-strategy.md +0 -68
  39. package/skills/query-design/references/highlight-queries.md +0 -122
@@ -1,382 +1,211 @@
1
1
  ---
2
2
  name: schema-discovery
3
- description: >
4
- Discover Power BI semantic-model tables, columns, measures, relationships,
5
- and column statistics through the analytics pack's local SQLite catalog.
6
- Use advanced INFO queries only when the fixed catalog does not contain the
7
- required metadata.
3
+ description: Ground DAX work in the cached semantic-model catalog, shipped measures, relationships, and explicit filter values.
8
4
  ---
9
5
 
10
- # Schema discovery
6
+ # Grounding semantic-model work
11
7
 
12
- ## Use the analytics catalog
8
+ **Prerequisite:** register the task's semantic model as a Rayfin connector first:
9
+ `npx rayfin connector add --type fabric-semanticmodel --name <name> --workspace-id <workspaceId> --item-id <itemId>`.
10
+ Until then `query-schema` has no model to catalog and exits with that command.
13
11
 
14
- This skill is the schema-discovery route for the
15
- [`analytics`](../analytics/SKILL.md) pack.
12
+ The semantic model's own catalog is cached as a local SQLite database and queried with SQL:
16
13
 
17
- It is a preview capability and ships off. `schema:sql` exists only in an app
18
- that opted in before applying the pack, so check `package.json` for the script
19
- before planning around it. To turn it on, add the flag and re-apply:
20
-
21
- ```sh
22
- # rayfin/.env
23
- RAYFIN_FEATURE_FLAGS=schema-sqlite
24
- ```
25
-
26
- ```sh
27
- npm run pack:add -- analytics
28
- ```
29
-
30
- Without it, discover the model through the read-only queries in
31
- [`discovery-queries.md`](./references/discovery-queries.md) instead, and say
32
- that the catalog is unavailable rather than reporting the model as empty.
33
-
34
- Apply the connectors and analytics packs, connect the intended semantic model,
35
- then query the local catalog:
36
-
37
- ```sh
38
- npm run --silent schema:sql -- "SELECT status, incomplete, notes FROM catalog_status"
39
- npm run --silent schema:sql -- --search "requested business concept"
40
- ```
41
-
42
- The first command builds the catalog automatically when it is missing or older
43
- than 15 minutes.
44
- An explicit rebuild is available when freshness matters:
45
-
46
- ```sh
47
- npm run schema:refresh
48
- ```
49
-
50
- One refresh runs five schema-only source queries from a hardcoded list, plus
51
- retries if the model throttles:
52
-
53
- - `INFO.VIEW.TABLES()`
54
- - `INFO.VIEW.COLUMNS()`
55
- - `INFO.VIEW.MEASURES()`
56
- - `INFO.VIEW.RELATIONSHIPS()`
57
- - `COLUMNSTATISTICS()`
58
-
59
- The builder does not accept agent-supplied DAX.
60
- Business-data queries cannot enter the catalog-build path.
61
- After those five requests, discovery is read-only SQL against local SQLite.
62
-
63
- This resolves the apparent conflict between full and progressive discovery.
64
- The fixed sources are materialized once for local reuse, but each agent result
65
- must still be a narrow SQL projection containing only evidence needed for the
66
- current task.
67
- Do not dump whole catalog tables into the conversation.
68
-
69
- ## Catalog contents
70
-
71
- The database is `.schema/models/<itemId>/model.sqlite`.
72
- Its tables preserve the model's native DAX column names:
73
-
74
- | SQLite table | Contents |
75
- | --- | --- |
76
- | `catalog_status` | Persistent `complete` or `incomplete` build status and notes |
77
- | `info_view_tables` | One row per semantic-model table |
78
- | `info_view_columns` | One row per model column |
79
- | `info_view_measures` | One row per shipped measure |
80
- | `info_view_relationships` | One row per relationship |
81
- | `column_statistics` | Actual Min/Max values, cardinality, and maximum text length |
82
- | `semantic_search` | Searchable projection of tables, columns, measures, and relationships |
83
-
84
- `column_statistics` is derived from real model rows.
85
- The catalog is customer data, not metadata-only content.
86
- The `.schema/` directory is gitignored and must never be copied, committed, or
87
- exposed.
88
-
89
- Read `.schema/models/<itemId>/catalog.md` once when orientation is needed.
90
- It records every table and column and prominently marks an incomplete catalog.
91
-
92
- ## Completeness and truncation
93
-
94
- Catalog completeness and SQL-result completeness are separate:
95
-
96
- - `catalog_status.status = 'incomplete'` means at least one source view failed.
97
- - `truncated: true` means the local SQL output was bounded by row count or byte size.
98
- - `truncated: false` only means that one SQL result is complete.
99
- It does not prove the catalog itself is complete.
100
-
101
- Always check `catalog_status` before using absence as evidence.
102
- If the catalog is incomplete, read its notes and run `npm run schema:refresh`
103
- once to retry.
104
- Concurrent refreshes serialize per model and reuse a complete catalog built by
105
- the process that won the lock.
106
- An incomplete refresh never replaces an existing complete catalog.
107
- If it remains incomplete, continue only with the views that are present and
108
- state which evidence is missing.
109
- Do not bypass an incomplete catalog with exploratory `INFO.VIEW.*` calls.
110
-
111
- `complete` describes which source views answered, not how much of each one you
112
- were allowed to see. `INFO.VIEW.MEASURES()` returns `Expression` only to a
113
- caller with write permission, so on a model you have Build access to, measure
114
- expressions can come back blank while the catalog still reads `complete`. Treat
115
- a blank `[Expression]` as unknown rather than as proof the measure has no
116
- definition, and say so instead of rebuilding its logic from physical columns.
117
-
118
- A SQL result with `truncated: true` cannot be used to choose model objects or
119
- author DAX.
120
- Narrow the `WHERE` clause and, for ordinary catalog tables, the projection.
121
- Queries selecting model tables automatically append compact owned-measure names
122
- and display folders, so a narrower projection does not reduce that enrichment;
123
- narrow the selected table rows instead.
124
- Rerun until the needed evidence is in a result with `truncated: false`.
125
- Do not use `--allow-truncated` during discovery or DAX authoring.
126
-
127
- ### Reading a single expression that is larger than the budget
128
-
129
- One cell can exceed the byte budget on its own — localization measures built
130
- from `SWITCH(USERCULTURE(), ...)` routinely run to tens of thousands of
131
- characters.
132
- Narrowing cannot help there: a single cell has nothing left to narrow, so the
133
- same `SELECT` returns `totalRows: 1` with `returnedRows: 0` however often it is
134
- repeated.
135
-
136
- Read it in slices instead, and fetch the length first so the last offset is
137
- known.
138
- Use `length([Expression])` as the bound rather than the character count in the
139
- search summary: `substr` offsets and `length` both count code points, while the
140
- summary counts UTF-16 units, so the two disagree for text containing emoji.
141
-
142
- ```sh
143
- npm run --silent schema:sql -- "SELECT length([Expression]) AS chars FROM info_view_measures WHERE [Table] = 'Sales' AND [Name] = 'Localized Label'"
144
- npm run --silent schema:sql -- "SELECT substr([Expression], 1, 1000) AS part FROM info_view_measures WHERE [Table] = 'Sales' AND [Name] = 'Localized Label'"
145
- npm run --silent schema:sql -- "SELECT substr([Expression], 1001, 1000) AS part FROM info_view_measures WHERE [Table] = 'Sales' AND [Name] = 'Localized Label'"
146
- ```
147
-
148
- Use a stride of 1000 characters.
149
- The budget is counted in bytes, not characters, so a stride that fits ASCII can
150
- still be three or four times too large for CJK or emoji text.
151
-
152
- Two conditions both have to hold, and the second is the one that is easy to get
153
- wrong:
154
-
155
- - Every chunk returns `returnedRows: 1` and `truncated: false`.
156
- `returnedRows: 0` means the slice was still too big — halve the stride and
157
- retry that offset.
158
- - The offsets cover `1` through the full length with no gap.
159
- Stopping after one clean chunk looks like success, because the chunks that
160
- were fetched are all complete; only the coverage is missing.
161
-
162
- Concatenate the parts in offset order to reconstruct the definition exactly.
163
-
164
- ## Discovery workflow
165
-
166
- 1. Query `catalog_status`.
167
- 2. Run a catalog-wide semantic search for the user's concepts.
168
- 3. Inspect candidate tables and their owned measures.
169
- 4. Query focused columns and measures for every selected table.
170
- 5. Query relationships connecting the selected tables.
171
- 6. Use column statistics only after the exact column is known.
172
- 7. Reuse prior catalog results instead of issuing equivalent SQL again.
173
- 8. Stop only when every requested output has an exact source mapping, and the
174
- shipped measures, grouping columns, relationship path, **filter grain** and
175
- **explicit filter values** are all grounded in untruncated results.
176
-
177
- The catalog holds metadata and per-column extrema, not the distinct values in a
178
- column. Once the filter column is known and a literal value still has to be
179
- confirmed, ground it with one bounded query against the model rather than
180
- looping on catalog SQL that cannot answer it:
181
-
182
- ```sh
183
- npx rayfin connector invoke <alias> executeQuery --file <probe>.query.json
184
- ```
185
-
186
- Test the literal you were given, so the answer is decisive:
187
-
188
- ```dax
189
- EVALUATE
190
- FILTER(VALUES('Geography'[Country]), 'Geography'[Country] = "France")
191
- ```
192
-
193
- Zero rows means there is no exact match for that value; report that instead of
194
- widening the search. If the probe errors, or the mapping is still unresolved
195
- after it, report the blocker rather than broadening discovery. Do not sample a
196
- column and read absence from the sample — `TOPN` over a column with more
197
- distinct values than the bound returns a complete-looking partial answer. This
198
- is a value check, not more discovery: ask it once, for a column you have
199
- already selected.
200
-
201
- Before writing any DAX, write down the output plan: every output field with its
202
- exact source (a fully qualified column, or an exact shipped measure for every
203
- business value), the filters and their values, the relationship path, and the
204
- expected row shape. Then compare what the query returns against that plan.
205
- A query that runs is not the same as a query at the right grain.
206
-
207
- One grain trap is worth checking by name. When a measure counts a column —
208
- `COUNT`, `DISTINCTCOUNT` or `DISTINCTCOUNTNOBLANK` over `'Table'[Column]` —
209
- filtering or grouping by that same column, or by one that reaches it through an
210
- active relationship, collapses the result to a trivial one per group. It
211
- returns rows and looks correct. The catalog already holds both inputs: read the
212
- measure's `[Expression]` from `info_view_measures` and the path from
213
- `info_view_relationships` before committing to that filter. If two filter
214
- columns remain plausible, compare them once with the shipped measures
215
- unchanged.
216
-
217
- Start with shipped measures.
218
- Never reconstruct business logic from physical columns when the model already
219
- ships a measure.
220
- Column statistics are not a shortcut around that: `Min`, `Max` and
221
- `Cardinality` are the raw column extent, ignoring filter context,
222
- relationships, and any business rule the measure encodes. A cardinality that
223
- happens to match an expected row count is not evidence that reading the column
224
- is correct.
225
- Helper, selector, calculation, and parameter tables may own the measure that
226
- supplies a displayed value, so do not restrict the first search to a fact-table
227
- prefix.
228
-
229
- A selected table is not fully inspected until its owned measures have been
230
- read. When such a table owns a measure that dispatches on its column — a
231
- `SELECTEDVALUE` / `SWITCH` over the selector, for example — group by that
232
- column and evaluate the shipped measure once instead of building one expression
233
- per row out of similarly named physical columns. That recipe does not apply to
234
- every helper: a native field-parameter table carries field references that the
235
- client resolves, so grouping by its label does not swap the underlying field.
236
- If the owned measures do not explain what the table is for, search measure
237
- expressions for references to it before falling back to raw aggregation:
238
-
239
- ```sh
240
- npm run --silent schema:sql -- "SELECT [Table], [Name] FROM info_view_measures WHERE [Expression] LIKE '%ParameterTable%'"
241
- ```
242
-
243
- Selecting from `info_view_tables` automatically includes a bounded
244
- `ownedMeasures` preview when `info_view_measures` is available.
245
- The preview carries names and display folders only so it cannot consume the
246
- result byte budget with repeated expressions.
247
- Query `info_view_measures` for each selected table to read expressions,
248
- descriptions, and format strings.
249
- Review it before choosing raw columns.
250
-
251
- ## SQL examples
252
-
253
- Search globally before narrowing:
254
-
255
- ```sh
256
- npm run --silent schema:sql -- --search "revenue region comparison"
257
- ```
258
-
259
- Search is an AND across names, descriptions, expressions, and display folders,
260
- so prefer a few specific terms over a whole sentence.
261
- When no object matches every term, terms that appear nowhere in the model are
262
- dropped and the search is answered without them, and the result says which ones
263
- were dropped.
264
- A statement sourced with `--file` must live inside the app, and is labelled by
265
- its path rather than by its contents.
266
-
267
- Inspect candidate tables:
268
-
269
- ```sh
270
- npm run --silent schema:sql -- "SELECT [Name], [Description], [IsHidden] FROM info_view_tables WHERE lower([Name]) LIKE '%sales%'"
14
+ ```bash
15
+ npx fabric-app-semantic-model-tools query-schema "SELECT [Name], [Expression] FROM info_view_measures WHERE [Table] = 'Sales'"
271
16
  ```
272
17
 
273
- Inspect shipped measures across selected tables in one statement:
18
+ Quote identifiers in inline SQL with SQLite brackets (`[Table]`, `[Max Length]`), which pass
19
+ unchanged through both bash and PowerShell. Inline SQL is fine for model identifiers and literals
20
+ quoted as `[...]` / `'...'` (double any `'` inside a literal), including LIKE patterns with `%`,
21
+ even when they came from earlier catalog output. Put the SQL in a `.sql` file and pass
22
+ `--file <file.sql>` only when a name or value contains `"`, `` ` ``, `$`, `\`, or a newline, or
23
+ when it embeds free-form text from the user. In a LIKE pattern, don't write a whole environment
24
+ variable name between two `%` signs (e.g. `%PATH%`, `%DATE%`); use
25
+ `instr(lower([Name]), 'path') > 0` instead.
274
26
 
275
- ```sh
276
- npm run --silent schema:sql -- "SELECT [Table], [Name], [Expression], [Description], [FormatString], [DisplayFolder] FROM info_view_measures WHERE [Table] IN ('Sales', 'Date') ORDER BY [Table], [Name]"
277
- ```
27
+ `--search` and `--sql` are repeatable and run in one process, so ask everything you already know
28
+ you need in one command rather than one command per question:
278
29
 
279
- Inspect columns only after selecting tables:
280
-
281
- ```sh
282
- npm run --silent schema:sql -- "SELECT [Table], [Name], [DataType], [Description], [FormatString], [IsHidden] FROM info_view_columns WHERE [Table] IN ('Sales', 'Date') ORDER BY [Table], [Name]"
30
+ ```bash
31
+ npx fabric-app-semantic-model-tools query-schema --search "revenue margin" --sql "SELECT [Name], [Expression] FROM info_view_measures WHERE [Table] = 'Sales'" --sql "SELECT [Name], [DataType] FROM info_view_columns WHERE [Table] = 'Sales'"
283
32
  ```
284
33
 
285
- Inspect the relationship path:
286
-
287
- ```sh
288
- npm run --silent schema:sql -- "SELECT [FromTable], [FromColumn], [ToTable], [ToColumn], [IsActive], [CrossFilteringBehavior] FROM info_view_relationships WHERE [FromTable] IN ('Sales', 'Date') OR [ToTable] IN ('Sales', 'Date')"
289
- ```
290
-
291
- Inspect statistics for exact columns:
292
-
293
- ```sh
294
- npm run --silent schema:sql -- "SELECT [Table Name], [Column Name], [Min], [Max], [Cardinality], [Max Length] FROM column_statistics WHERE [Table Name] = 'Date' AND [Column Name] IN ('Date', 'Year')"
295
- ```
296
-
297
- Column names retain spaces and reserved words.
298
- Quote them with SQL double quotes or square brackets.
299
-
300
- ## Batch several statements in one invocation
301
-
302
- Each extra invocation costs a whole model round trip, and a round trip re-sends
303
- the entire conversation.
304
- The five statements above are normally run back to back, so run them together
305
- instead.
306
- `--search`, `--sql`, and `--file` are repeatable, and searches run before SQL:
307
-
308
- ```sh
309
- npm run --silent schema:sql -- --search "revenue region comparison" --sql "SELECT [Name], [Description], [IsHidden] FROM info_view_tables WHERE lower([Name]) LIKE '%sales%'" --sql "SELECT [Table], [Name], [Expression], [Description], [FormatString], [DisplayFolder] FROM info_view_measures WHERE [Table] IN ('Sales', 'Date') ORDER BY [Table], [Name]"
310
- ```
311
-
312
- The reply is one payload: a `#` header naming the batch, then a `##` label line
313
- per statement followed by that statement's ordinary result block.
314
- A failed statement stays in the payload beside the ones that worked, so a single
315
- mistake does not cost the whole batch.
316
-
317
- At most eight statements per invocation.
318
- The whole batch shares one byte budget, so batching never costs more context
319
- than a single statement would.
320
- Ask everything you already know you need, but keep each statement narrow: the
321
- shared budget is divided across them, and unused room rolls forward.
322
-
323
- ## Advanced metadata outside the catalog
324
-
325
- The fixed catalog intentionally covers the common DAX-authoring path.
326
- It does not include every elevated `INFO.*` function.
327
-
328
- Only when the current task specifically requires calculation groups, calendars,
329
- user-defined functions, variations, or model configuration should you use the
330
- advanced read-only queries in
331
- [`discovery-queries.md`](./references/discovery-queries.md).
332
- Run the minimum advanced query needed through:
333
-
334
- ```sh
335
- npx rayfin connector invoke <alias> executeQuery --file <query>.query.json
336
- ```
337
-
338
- The file is a JSON operation input such as:
339
-
340
- ```json
341
- {
342
- "query": "EVALUATE INFO.CALCULATIONGROUPS()"
343
- }
344
- ```
34
+ Each invocation costs a full model round trip, and a round trip re-sends the whole conversation, so
35
+ three separate commands cost roughly three times what one batch does. Batch the questions you can
36
+ write down now; keep a question whose *content* depends on an earlier answer for the next command.
37
+ Up to 8 statements per invocation.
345
38
 
346
- Do not use advanced queries to re-fetch tables, columns, measures,
347
- relationships, or column statistics already owned by the catalog.
348
- If an elevated query fails with a permission error after catalog discovery
349
- succeeded, mark elevated metadata unavailable and do not probe the remaining
350
- elevated functions.
351
-
352
- ## Must, prefer, avoid
353
-
354
- ### Must
355
-
356
- - Check `catalog_status` before treating absence as evidence.
357
- - Bind every business value to an exact shipped measure, and every label or
358
- grouping to an exact column, before writing DAX. If exhaustive untruncated
359
- discovery finds no suitable measure, record which one is missing and the
360
- catalog evidence for that before using another source.
361
- - Use fully qualified `'Table'[Column]` references in authored DAX.
362
- - Use simple `[Measure]` references for measures.
363
- - Keep every SQL result narrow and untruncated.
364
- - Reuse discovered metadata.
365
-
366
- ### Prefer
367
-
368
- - `--search` for the first semantic pass.
369
- - Batching known statements into one invocation over separate calls.
370
- - One SQL statement for several known metadata questions.
371
- - Exact relationship paths and column statistics after candidate selection.
372
-
373
- ### Avoid
374
-
375
- - Hand-running the catalog's five source queries.
376
- - Progressive `INFO.VIEW.*` connector round trips.
377
- - Whole-table SQL dumps.
378
- - Re-fetching metadata already present in the catalog or conversation.
379
- - Treating `truncated: false` as proof that an incomplete catalog is complete.
380
- - Rebuilding a helper table's logic one row at a time from physical columns.
381
- - Using unavailable `GetSemanticModelSchema`, `GenerateQuery`, or
382
- `DiscoverArtifacts` tools.
39
+ The whole batch shares one result-size budget — the same one a single statement gets — divided
40
+ between the statements, with each statement's unused share rolling forward to the next. So batch
41
+ **focused** statements. A statement that returns a lot is truncated to its own share; later
42
+ statements keep their share of the budget. Run a query you expect to return a lot in its own
43
+ command so it gets the whole budget.
44
+
45
+ Each statement's result is labelled with the query that produced it, searches first, then SQL. A
46
+ statement that fails stays in the output beside the ones that succeeded. The exit code is the worst
47
+ outcome across the batch: 1 if any statement failed, 2 if any was truncated, 0 otherwise.
48
+
49
+ The database builds itself on first use and rebuilds when stale, so ask a question rather than
50
+ refreshing it by hand. `.schema/models/<itemId>/catalog.md` names every table, its real row and
51
+ column counts, and how the tables join; read it once and the rest is ordinary SQL.
52
+
53
+ The tables hold the untouched output of `INFO.VIEW.TABLES()`, `INFO.VIEW.COLUMNS()`,
54
+ `INFO.VIEW.MEASURES()`, `INFO.VIEW.RELATIONSHIPS()` and `COLUMNSTATISTICS()`, under the model's own
55
+ column names. Nothing is renamed, flattened, or summarized, so a column you see in the catalog is a
56
+ column DAX returned.
57
+
58
+ Besides metadata, the catalog stores the `COLUMNSTATISTICS()` output, including each column's
59
+ `Min`/`Max` sample values. It lives locally under `.schema/models/`, which is git-ignored and
60
+ rebuilt on demand; SQL queries read only that local copy.
61
+
62
+ Do not hand-run exploratory `INFO.*` queries, read the catalog database directly, bypass the catalog
63
+ with connector probes, or ask the user for metadata the catalog can answer.
64
+
65
+ Check `catalog_status` and `catalog.md` for completeness before treating absence
66
+ as evidence. A failed view differs from a successful zero-row view.
67
+ An explicit `--refresh` retries now but never replaces a complete catalog with
68
+ a build that loses a view. Empty Direct Lake statistics are expected; do not
69
+ reconstruct them with expensive MIN/MAX/DISTINCTCOUNT queries.
70
+
71
+ For long measure expressions, read bounded chunks with
72
+ `SELECT substr([Expression], 1, 2000) ...`, then offsets 2001, 4001, etc.,
73
+ using a focused WHERE clause until the expression is complete.
74
+ An empty Expression on an object that should expose one means detail is unavailable,
75
+ not proof that no logic exists. Search reports dropped terms; refine them rather
76
+ than treating broad or punctuation-only searches as absence.
77
+ For advanced metadata (calculation groups, calendars, UDFs) not in the five catalog
78
+ views, follow [discovery-queries.md](references/discovery-queries.md) narrowly;
79
+ do not use that exception to bypass ordinary cached discovery.
80
+
81
+ ## Querying
82
+
83
+ Every query starts with result metadata including `totalRows`, `returnedRows`, and `truncated`.
84
+ Zero rows is a successful answer, not an error; it means the model has no such entity.
85
+
86
+ Results are capped by row count and by size, so filter in SQL instead of selecting whole tables and
87
+ reading the overflow. **A result with `truncated: true` is incomplete and cannot be used to choose
88
+ model objects or author DAX.** The command exits with code 2 after printing the bounded preview.
89
+ Narrow the `WHERE` clause and/or projection and rerun until the entities needed by the plan appear
90
+ in a result with `truncated: false`. `SELECT *` on `info_view_measures` will truncate on a large
91
+ model; selecting only the required metadata and filtering by table, name, folder, description, or
92
+ expression will not.
93
+
94
+ The connection is read-only. A failing query comes back as an error message, so a malformed query
95
+ costs one retry, not a corrupted cache.
96
+
97
+ ## Column statistics
98
+
99
+ `column_statistics` carries `Min`, `Max`, `Cardinality` and `Max Length` for each column. Use them
100
+ to choose grouping columns, bound date and numeric filters, and anticipate result size.
101
+
102
+ They are not a substitute for the model's logic. When a shipped measure computes the same quantity,
103
+ use the measure: a statistic is the raw column extent, ignoring filter context, relationships, and
104
+ any business rule the measure encodes. A column whose cardinality happens to match an expected row
105
+ count is not evidence that reading it is correct.
106
+
107
+ If `column_statistics` is missing from the catalog, the model did not return statistics. Continue
108
+ without them.
109
+
110
+ ## Global semantic discovery
111
+
112
+ Search the whole catalog before restricting discovery to a domain-looking table prefix. Helper,
113
+ parameter, selector, calculation, and metadata tables may have names that do not share the user's
114
+ term or the primary fact-table prefix. Search table names and descriptions, measure names,
115
+ descriptions, expressions, and display folders, and column names and descriptions in one query.
116
+ Start with the ranked catalog-wide search, replacing the terms with the user's concepts:
117
+
118
+ ```bash
119
+ npx fabric-app-semantic-model-tools query-schema --search "requested concept comparison"
120
+ ```
121
+
122
+ Every search token must appear somewhere in an entity's name, parent table, description, display
123
+ folder, expression, or type. If there are no all-term matches, model-wide absent terms may
124
+ be dropped only with an explicit diagnostic in the result. Exact and prefix name matches rank above descriptive or expression
125
+ matches, and shipped measures rank above columns at the same lexical relevance. After the global
126
+ result identifies candidate tables, run a focused, untruncated SQL query for their measures,
127
+ columns, descriptions, and expressions. Do not begin with a domain-prefix restriction: that can
128
+ hide a helper, parameter, selector, or measure owned elsewhere.
129
+
130
+ Search is a findability surface, not a reading surface. Ranking sees every field in full, but the
131
+ printed rows omit a `description` that is identical across the model and shorten a long
132
+ `expression` to its head plus its length. The result's `trimmed` counts say when either happened.
133
+ Both fields are still matched, so nothing becomes unfindable — `SELECT` the row when you need the
134
+ full text. Do not treat a shortened expression as the measure's definition.
135
+
136
+ ## Model-facing workflow
137
+
138
+ 1. Read `catalog.md` once when orientation is needed. Do not fetch it again in the same task.
139
+ 2. Run global semantic discovery across tables, measures, and columns before applying a
140
+ domain-specific table-name restriction.
141
+ 3. Start with shipped measures. Whenever a table becomes part of the plan, inspect measures owned
142
+ by that table before authoring DAX. Query all selected tables in one statement using `IN` or
143
+ `OR`, returning at least `[Table]`, `[Name]`, `[Expression]`, `[Description]`,
144
+ `[FormatString]`, and `[DisplayFolder]`. Never reconstruct business logic from columns when the
145
+ model already ships the measure. A single-table query on `info_view_tables` (no JOIN, comma
146
+ join, or UNION) automatically includes bounded `ownedMeasures` for each returned table; review
147
+ those before choosing physical columns.
148
+ 4. Answer several known metadata questions in one command. Combine them into one statement with a
149
+ join or an `OR` where they share a shape; where they do not, pass them as repeated `--sql`
150
+ (and `--search`) flags in a single invocation. Either way it is one round trip. One command per
151
+ question is the single most expensive habit in discovery.
152
+ 5. Take explicit filter values from the request, and bind every other qualifier in it to a filter
153
+ too. A qualifier can imply a value the request never states: the complement of a named value,
154
+ one section of a parameter or selector table, a status or flag. Take such values from the model
155
+ (a parameter table's `[Expression]`, a column's `column_statistics` `[Min]`/`[Max]`), not from
156
+ guesses. Apart from those statistics the catalog holds no other row values, but a
157
+ parameter or calculated table (for example a `DATATABLE`) shows its literal rows in
158
+ `info_view_tables.[Expression]`; read that expression before inventing labels or keys. Do not
159
+ probe the model for row values during discovery. When a value needs confirmation, add one small
160
+ value-check query file
161
+ (for example `EVALUATE FILTER(VALUES('Customer'[Country]), 'Customer'[Country] = "Canada")`) to
162
+ the same final `run-dax-query` call. The value check counts toward that call's four-file limit;
163
+ when the plan already has four queries, fold the check into one of them or run it in the next
164
+ call. An empty value-check result is the signal to correct that value once.
165
+ 6. Before another schema request, check whether an earlier result already answers the question.
166
+ Never rerun an identical or equivalent schema query.
167
+ 7. Scope discovery to the current visual batch, reusing earlier mappings. Stop only after
168
+ every output field in that batch has an exact source mapping; the shipped
169
+ measures, grouping columns, relationship path, and filter grain are grounded; every explicit
170
+ filter value is bound to its exact column (its value comes from the request and, if needed, is
171
+ confirmed by the value check in step 5, not by more discovery); every qualifier in the request
172
+ relevant to that batch is bound to a filter, including implied ones (step 5); and no mapping
173
+ depends on a truncated result.
174
+ 8. Write and validate the DAX with the [dax-authoring](../dax-authoring/SKILL.md) skill.
175
+
176
+ ## DAX plan and semantic grain
177
+
178
+ Before writing DAX for the current visual batch, form a compact plan containing:
179
+
180
+ - every requested output field;
181
+ - the exact source for each field: a fully qualified grouping/label column, an exact shipped
182
+ measure name for every business value, or an explicit literal;
183
+ - exact filters and values, including the ones the request implies rather than names;
184
+ - relevant relationship path;
185
+ - expected output columns and row shape;
186
+ - unresolved filter candidates, if any.
187
+
188
+ Keep this plan in your working notes; do not write it to a file. `run-dax-query` does not know the
189
+ plan, so compare the columns, row count, and warnings it reports with the plan yourself.
190
+
191
+ Do not generate DAX until every business value needed by this visual is bound to an exact shipped measure and
192
+ every label/group is bound to an exact column. If exhaustive, untruncated measure discovery
193
+ establishes that no suitable measure exists, record both the missing measure and the catalog
194
+ evidence before declaring a different source. Partial measure results are not evidence that a
195
+ measure is absent.
196
+
197
+ A selected table is not fully inspected until its owned measures have been reviewed. This is
198
+ especially important for helper, parameter, selector, calculation, and metadata tables: their rows
199
+ may provide labels or filter context while an owned measure returns the requested value. Group by
200
+ the helper column and evaluate the shipped measure; do not invent one expression per row from
201
+ similarly named physical columns. If owned measures do not explain the table's semantic role,
202
+ search measure expressions for references to that table before falling back to raw aggregation.
203
+
204
+ If a catalog refresh fails, or would drop a view the previous catalog has (for example a throttled
205
+ `INFO.VIEW.MEASURES()`), the previous database is kept rather than replaced by the partial
206
+ build, the query is answered from it, and stderr prints a warning naming the catalog's age and
207
+ what failed. A refresh that is incomplete only in views the previous catalog also lacked is
208
+ published with an incomplete-catalog warning. Continue discovery from that cache. Treat the warning
209
+ as a reason to avoid freshness-sensitive assumptions, not as permission to ignore the cached
210
+ catalog. When no previous catalog exists, a partial build is still used and a warning says which
211
+ views are missing.
@@ -2,14 +2,12 @@
2
2
 
3
3
  Read-only DAX queries for metadata exploration using `INFO.VIEW.*` and `INFO.*` rowsets.
4
4
 
5
- The analytics pack materializes tables, columns, measures, relationships, and
6
- column statistics into its local SQLite catalog.
7
- Do not run the common `INFO.VIEW.*` queries below when that catalog is
8
- available.
9
- They remain here as output-shape reference material and for environments that
10
- do not have the analytics pack.
11
- Use the advanced `INFO.*` sections only when the task requires metadata outside
12
- the fixed catalog.
5
+ The semantic-model tools materialize tables, columns, measures, relationships,
6
+ and column statistics into the local SQLite catalog, always available for this
7
+ capability. Common `INFO.VIEW.*` examples below are output-shape reference only:
8
+ do not execute them, including the scope-estimation probe.
9
+ Use advanced `INFO.*` sections only for required metadata outside the fixed
10
+ catalog. A permission failure stops the remaining elevated queries.
13
11
 
14
12
  ## Scope Estimation Queries
15
13