@microsoft/rayfin-semantic-model-pack 1.1.0 → 2.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (39) hide show
  1. package/CHANGELOG +4 -0
  2. package/README.md +16 -5
  3. package/kit/hooks/use-query-factory-target.spec.ts +191 -0
  4. package/kit/hooks/use-query-factory-target.ts +123 -0
  5. package/kit/hooks/use-semantic-model-query.spec.ts +51 -34
  6. package/kit/hooks/use-semantic-model-query.ts +30 -49
  7. package/kit/lib/query-factory.spec.ts +97 -0
  8. package/kit/lib/query-factory.ts +68 -0
  9. package/kit/lib/semantic-model-errors.spec.ts +131 -0
  10. package/kit/lib/semantic-model-errors.ts +110 -0
  11. package/kit/lib/to-data-table.spec.ts +444 -0
  12. package/kit/lib/to-data-table.ts +221 -0
  13. package/pack.json +122 -25
  14. package/package.json +1 -1
  15. package/skills/analytics/SKILL.md +126 -422
  16. package/skills/dax-authoring/SKILL.md +98 -31
  17. package/skills/query-design/SKILL.md +22 -29
  18. package/skills/query-design/references/anti-patterns.md +13 -11
  19. package/skills/query-design/references/format-strings.md +6 -12
  20. package/skills/query-design/references/multi-grain-patterns.md +121 -108
  21. package/skills/schema-discovery/SKILL.md +197 -368
  22. package/skills/schema-discovery/references/discovery-queries.md +6 -8
  23. package/skills/visual-interactivity/SKILL.md +68 -0
  24. package/skills/visual-interactivity/references/advanced-query-shapes.md +155 -0
  25. package/skills/visual-interactivity/references/filter-aware-grid-totals.md +101 -0
  26. package/skills/visual-interactivity/references/highlight-queries.md +88 -0
  27. package/skills/visual-interactivity/references/interaction-manager.md +127 -0
  28. package/skills/visual-interactivity/references/native-selections.md +20 -0
  29. package/skills/visual-interactivity/references/query-factories.md +169 -0
  30. package/kit/scripts/rayfin-connector.mjs +0 -265
  31. package/kit/scripts/rayfin-connector.test.mjs +0 -243
  32. package/kit/scripts/schema-sql-batch.test.mjs +0 -2097
  33. package/kit/scripts/schema-sql-cli.test.mjs +0 -488
  34. package/kit/scripts/schema-sql.mjs +0 -2650
  35. package/kit/scripts/schema-sql.test.mjs +0 -1387
  36. package/kit/scripts/schema.mjs +0 -166
  37. package/kit/scripts/schema.test.mjs +0 -271
  38. package/skills/query-design/references/filter-strategy.md +0 -68
  39. package/skills/query-design/references/highlight-queries.md +0 -122
@@ -1,159 +1,172 @@
1
1
  # Multi-Grain Patterns
2
2
 
3
- When a component needs data at multiple grains (e.g., region detail + grand total, monthly trend + YTD), use separate `.dax` files and separate hook calls.
3
+ When a component needs data at multiple grains, return each metric at the grain where DAX must evaluate it. DataGrid body rows and a grand total can share one flagged result table; unrelated datasets such as a monthly trend and YTD annotation should use separate queries.
4
4
 
5
5
  ## Contents
6
6
 
7
- - [File Organization](#file-organization) — `.dax` + `.ts` layout per visualization
8
- - [DAX Query Shapes](#dax-query-shapes) — separate detail and one-row summary queries
9
- - [Rendering in VegaVisual (multi-DataTable)](#rendering-in-vegavisual-multi-datatable) — pass named datasets, layer in spec
10
- - [Rendering in DataGrid (provided grand total)](#rendering-in-datagrid-provided-grand-total) — pass the summary `DataTable` through `grandTotals`
11
- - [Consistency Rule](#consistency-rule) — shared filters, measures, scope across split-grain queries
12
-
13
- ## File Organization
14
-
15
- ```text
16
- packages/frontend/src/queries/sales/
17
- ├── revenue-by-region.dax # Detail grain
18
- ├── revenue-by-region.json # Vega-Lite spec (layers both datasets)
19
- ├── revenue-by-region.ts # Factory: returns detailQuery + vegaLiteSpec
20
- ├── revenue-total.dax # Summary grain (single-row total)
21
- ├── revenue-total.ts # Factory: returns totalQuery (no spec needed)
22
- └── index.ts
23
- ```
7
+ - [File organization](#file-organization) — keep rollups together and independent result tables in separate query files
8
+ - [DataGrid grand-total query](#datagrid-grand-total-query) — return flagged body and optional grand-total rows in one table
9
+ - [Rendering in DataGrid](#rendering-in-datagrid) — split the result into body and grand-total `DataTable`s
10
+ - [Independent query organization and wiring](#independent-query-organization-and-wiring) — connect multiple factories and hook calls
11
+ - [Rendering in VegaVisual](#rendering-in-vegavisual) — use separate queries for independently consumed datasets
12
+ - [Consistency rules](#consistency-rules) — keep measures, filters, metadata, and row classification aligned
13
+
14
+ ## File organization
15
+
16
+ Follow the frontend-relative `src/queries/<page>/` factory organization described in analytics.
17
+
18
+ - **DataGrid body and grand total:** Use one `.dax` file and one factory. The query returns both as flagged rows.
19
+ - **Independent datasets:** Use one `.dax` file and factory per result table. Keep the shared Vega-Lite spec with the primary visualization factory and re-export every factory through the appropriate barrel.
24
20
 
25
- ## DAX Query Shapes
21
+ ## DataGrid grand-total query
26
22
 
27
- Keep the detail and summary grains in separate query files.
23
+ Use `ROLLUPADDISSUBTOTAL` to evaluate every metric at both the body grain and grand-total grain in one DAX query. Wrap all current body grouping columns in `ROLLUPGROUP` so they roll up as one unit and do not produce intermediate subtotal rows.
28
24
 
29
- **Detail query:**
25
+ ### Basic flagged rollup
26
+
27
+ This starter emits body rows and, when at least one total expression is non-BLANK, one flagged grand-total row:
30
28
 
31
29
  ```dax
32
30
  EVALUATE
33
31
  SUMMARIZECOLUMNS(
34
- 'Region'[Name],
32
+ ROLLUPADDISSUBTOTAL(
33
+ ROLLUPGROUP(
34
+ 'Region'[Name],
35
+ 'Product'[Category]
36
+ ),
37
+ "IsGrandTotal"
38
+ ),
35
39
  "Revenue", [Total Revenue],
36
40
  "Customer Count", DISTINCTCOUNT('Sales'[Customer Key])
37
41
  )
38
- ORDER BY 'Region'[Name]
42
+ ORDER BY 'Region'[Name], 'Product'[Category]
39
43
  ```
40
44
 
41
- **Summary query:**
45
+ ### Flagged-rollup scaffold
46
+
47
+ Replace each `<placeholder>` before running this query:
42
48
 
43
49
  ```dax
44
- EVALUATE
45
- ROW(
46
- "Revenue", [Total Revenue],
47
- "Customer Count", DISTINCTCOUNT('Sales'[Customer Key])
48
- )
50
+ DEFINE
51
+ // Application/page filter: constrains body and grand-total evaluation.
52
+ VAR _BodyFilter =
53
+ TREATAS({<body-filter-values>}, '<FilterDimension>'[<FilterColumn>])
54
+
55
+ // Maximum number of body rows to return.
56
+ VAR _MaxBodyRows = 500
57
+
58
+ VAR _RollupResult =
59
+ SUMMARIZECOLUMNS(
60
+ ROLLUPADDISSUBTOTAL(
61
+ ROLLUPGROUP(
62
+ '<Dimension1>'[<Column1>],
63
+ '<Dimension2>'[<Column2>]
64
+ ),
65
+ "IsGrandTotal"
66
+ ),
67
+ _BodyFilter,
68
+ "<MetricAlias>", [<Measure>]
69
+ )
70
+
71
+ VAR _ResultLimit =
72
+ _MaxBodyRows
73
+ + COUNTROWS(FILTER(_RollupResult, [IsGrandTotal] = TRUE()))
74
+
75
+ // Prioritize the total inside TOPN so payload limiting cannot discard it.
76
+ VAR _LimitedResult =
77
+ TOPN(
78
+ _ResultLimit,
79
+ _RollupResult,
80
+ [IsGrandTotal], DESC,
81
+ [<MetricAlias>], DESC,
82
+ '<Dimension1>'[<Column1>], ASC,
83
+ '<Dimension2>'[<Column2>], ASC
84
+ )
85
+
86
+ EVALUATE _LimitedResult
87
+ ORDER BY
88
+ [<MetricAlias>] DESC,
89
+ '<Dimension1>'[<Column1>] ASC,
90
+ '<Dimension2>'[<Column2>] ASC
49
91
  ```
50
92
 
51
- The `Customer Count` grand total must come from the summary query. Summing the per-region distinct counts would double-count customers that appear in multiple regions.
93
+ `_BodyFilter` is a regular `SUMMARIZECOLUMNS` filter, so it constrains both body and grand-total evaluation. Add one variable per application or page filter and pass each one as a `SUMMARIZECOLUMNS` filter argument.
94
+
95
+ DataGrid filtering is enabled by default. Extend this scaffold with [filter-aware grid totals](../../visual-interactivity/references/filter-aware-grid-totals.md) so local grid filters update the provided total without removing body rows needed for client-side filtering.
96
+
97
+ `_ResultLimit` adds one slot only when the grand-total row exists. `TOPN` ranks that row first internally so it survives the limit while returning no more than `_MaxBodyRows` body rows. Because `TOPN` does not guarantee output order, the final `ORDER BY` makes body-row order deterministic for testing and debugging; DataGrid still owns user-facing sorting. Include every body dimension after the metric in both sort lists to break ties deterministically.
52
98
 
53
- As with detail data, copy the semantic model's format strings into the summary factory's `columnMetadata`; `toDataTable()` places them on the `DataTable`:
99
+ When present, the row where `IsGrandTotal` is `TRUE` contains all grand-total column values. Each expression is evaluated again without the body grouping context; DAX does not add the body-row results. For example, the grand-total `Customer Count` is a distinct count across the regions in the current filter scope rather than a sum of their per-region distinct counts. If every grand-total expression evaluates to `BLANK` under the current filters, `SUMMARIZECOLUMNS` omits the row.
100
+
101
+ Copy the semantic model's format strings into the query factory's `columnMetadata`. Do not add metadata for the rollup flag because `toRollupDataTables()` removes it:
54
102
 
55
103
  ```typescript
56
104
  const columnMetadata: ColumnMetadataMap = {
105
+ "Region[Name]": { name: "RegionName", displayName: "Region" },
57
106
  "[Revenue]": { name: "Revenue", displayName: "Revenue", format: "$#,##0.00" },
58
107
  "[Customer Count]": { name: "Customer Count", displayName: "Customer Count", format: "#,##0" },
59
108
  };
60
109
  ```
61
110
 
62
- ## Rendering in VegaVisual (multi-DataTable)
111
+ Test the query with `npx fabric-app-semantic-model-tools run-dax-query --dax <file> --alias <name>`. Verify that the flag values are booleans, and copy the exact returned rollup flag name into the conversion options rather than assuming its serialized form.
63
112
 
64
- Use two hook calls and pass both tables as named datasets. The spec references each by name, so no TypeScript stitching is needed:
113
+ ## Rendering in DataGrid
65
114
 
66
- ```typescript
67
- // Two hook calls, two DataTables. Both factories target the same model,
68
- // so the two connection objects are equivalent — we destructure both
69
- // (factory signature requires it) but use one for both hooks.
70
- const { connection, query: detailQuery, columnMetadata: detailMeta, vegaLiteSpec } = revenueByRegion();
71
- const { connection: _summaryConn, query: totalQuery, columnMetadata: totalMeta } = revenueTotal();
72
-
73
- const detail = useSemanticModelQuery({ connection, query: detailQuery });
74
- const summary = useSemanticModelQuery({ connection, query: totalQuery });
75
-
76
- // After handling both hooks' loading and error states:
77
- const detailTable = toDataTable(detail.data.table, detailMeta);
78
- const summaryTable = toDataTable(summary.data.table, totalMeta);
79
- ```
115
+ DataGrid filtering is enabled by default. Follow [component wiring](../../visual-interactivity/references/filter-aware-grid-totals.md#component-wiring) to derive the rollup query from grid filter state, split its result with `toRollupDataTables()`, and connect `onFilterChange`.
80
116
 
81
- ```tsx
82
- <VegaVisual
83
- spec={vegaLiteSpec}
84
- data={{ detail: detailTable, summary: summaryTable }}
85
- theme={theme}
86
- />
87
- ```
117
+ `toRollupDataTables()` treats rows where all configured rollup flags are `FALSE` as body rows and rows where all flags are `TRUE` as the provided grand total. It rejects multiple grand-total rows and intermediate subtotal rows until DataGrid has an explicit subtotal API. The array-based flag contract leaves room to classify additional rollup levels when that support is added.
88
118
 
89
- The Vega-Lite spec (in the `.json` file) layers the two datasets:
90
-
91
- ```json
92
- {
93
- "layer": [
94
- {
95
- "data": { "name": "detail" },
96
- "mark": "bar",
97
- "encoding": {
98
- "x": { "field": "RegionName", "type": "nominal" },
99
- "y": { "field": "Revenue", "type": "quantitative" }
100
- }
101
- },
102
- {
103
- "data": { "name": "summary" },
104
- "mark": { "type": "rule", "color": "firebrick", "strokeDash": [4, 4] },
105
- "encoding": {
106
- "y": { "field": "Revenue", "type": "quantitative" }
107
- }
108
- }
109
- ]
110
- }
111
- ```
119
+ Always pass `grandTotalTable` through `grandTotals.data` so DataGrid treats it as authoritative; omitting `data` selects client-computed mode. Conversion failures are query-authoring errors and intentionally surface for malformed totals.
112
120
 
113
- This pattern works for: reference lines (average, target), cross-highlighting between datasets, annotations on top of detail data.
121
+ DataGrid owns grand-total rendering; do not append the total to body rows or use `cellRenderer` for it.
114
122
 
115
- ## Rendering in DataGrid (provided grand total)
123
+ ## Independent query organization and wiring
116
124
 
117
- Pass the detail and one-row summary tables separately. DataGrid owns grand-total rendering; do not append the summary to the body rows or use `cellRenderer` for the totals row.
125
+ Use separate queries when a consumer needs independently shaped result tables rather than classified rows from one shared result. Give each result its own `.dax` file and factory output (`connection`, `query`, and `columnMetadata`), and keep the shared Vega-Lite spec with the visualization factory. Use matching kebab-case file names, structural variants and barrel exports under frontend-relative `src/queries/<page>/`.
118
126
 
119
- DataGrid filtering is enabled by default. Derive the summary query from grid filter state so the provided total matches the visible rows:
127
+ Call each factory with the same shared inputs, then execute each returned query with its own hook call:
120
128
 
121
- ```tsx
122
- import type { CellValue } from "@microsoft/fabric-datagrid";
129
+ ```typescript
130
+ const detailConfig = revenueByRegion(params);
131
+ const summaryConfig = revenueSummary(params);
132
+
133
+ const detail = useSemanticModelQuery({
134
+ connection: detailConfig.connection,
135
+ query: detailConfig.query,
136
+ });
137
+ const summary = useSemanticModelQuery({
138
+ connection: summaryConfig.connection,
139
+ query: summaryConfig.query,
140
+ });
141
+ ```
142
+
143
+ Each `useSemanticModelQuery` invocation has its own loading and error state. Caching and request coalescing are template-specific; do not assume each hook owns a separate cache or causes a separate SDK round trip. When the tables form a matched set, replace the rendered set only after every request has succeeded and none is still loading; surface a failure from any request instead of combining fresh and stale results.
123
144
 
124
- const [gridFilters, setGridFilters] = useState<Record<string, CellValue[]>>({});
125
- const { query: totalQuery, columnMetadata: totalMeta } = revenueTotal({ filters: gridFilters });
126
- const summary = useSemanticModelQuery({ connection, query: totalQuery });
145
+ ## Rendering in VegaVisual
127
146
 
128
- if (summary.error) return <ErrorMessage message={summary.error.message} />;
147
+ Convert each successful result with its own metadata, then pass the `DataTable`s as named datasets:
129
148
 
130
- const filteredSummaryTable =
131
- !summary.isLoading &&
132
- summary.data?.status === "success"
133
- ? toDataTable(summary.data.table, totalMeta)
134
- : undefined;
149
+ ```typescript
150
+ const detailTable = toDataTable(detail.data.table, detailConfig.columnMetadata);
151
+ const summaryTable = toDataTable(summary.data.table, summaryConfig.columnMetadata);
135
152
  ```
136
153
 
137
154
  ```tsx
138
- <DataGrid
139
- data={detailTable}
140
- grandTotals={{ position: "bottom", data: filteredSummaryTable }}
141
- onFilterChange={(columnId, selectedValues) =>
142
- setGridFilters((current) => ({ ...current, [columnId]: selectedValues }))
143
- }
155
+ <VegaVisual
156
+ spec={detailConfig.vegaLiteSpec}
157
+ data={{ detail: detailTable, summary: summaryTable }}
144
158
  theme={theme}
145
159
  />
146
160
  ```
147
161
 
148
- Changing `gridFilters` re-queries the summary while DataGrid filters detail rows locally. The summary factory must translate each filterable column ID to its DAX predicate rather than interpolate column IDs directly. See the visuals skill's [DataGrid reference](../../visuals/references/data-grid-visual.md#grand-totals).
149
-
150
- ## Consistency Rule
151
-
152
- When splitting a visualization across multiple queries, all queries must share the same semantic contract:
162
+ See the visuals skill's [multi-data input](../../visuals/references/multi-data-input.md) reference for named dataset binding, overlays, reference lines, axis spines, and compatible field contracts. For a baseline plus selected-subset overlay, follow [Highlight queries](../../visual-interactivity/references/highlight-queries.md) for query generation and synchronization.
153
163
 
154
- - Same measures and application-level DAX filters; derive both queries from the same inputs
155
- - Local DataGrid filters remain enabled; pass `onFilterChange` selections to the summary factory so the provided total matches the visible rows
156
- - Summary metadata names match the detail table and include formats
157
- - Use `ROW(...)` for the summary so it returns exactly one row and preserves BLANK values; `SUMMARIZECOLUMNS` can return zero rows when all measures are BLANK
164
+ ## Consistency rules
158
165
 
159
- Each `useSemanticModelQuery` call is a separate SDK round trip, and every local filter change re-executes the summary query.
166
+ - Use one `EVALUATE` per DAX query; the Execute Queries transport supports one result table per query.
167
+ - For DataGrid grand totals, keep body and total expressions in one `SUMMARIZECOLUMNS` and split its flagged rows in TypeScript.
168
+ - Apply application filters to every row and local DataGrid filters through `ROLLUPADDISSUBTOTAL`'s `grandtotalFilter`; DataGrid continues filtering the complete body table locally.
169
+ - Keep output aliases and metadata names aligned; the body and grand-total `DataTable`s use the same format metadata.
170
+ - Validate that the query never returns multiple flagged grand-total rows.
171
+ - Pass the same model connection and shared filter parameters to every related query factory so tables do not drift.
172
+ - Keep shared field aliases, types, and encoded keys compatible across independently queried tables. Cross-highlight baseline and subset queries must also share grouping columns, measures, and row keys as described in [Highlight queries](../../visual-interactivity/references/highlight-queries.md).