@finos/legend-cube-builder 0.0.3 → 0.0.5

package/README.md

@@ -7,23 +7,30 @@ It holds the page (the canvas, the palette, the side-panel editors, the source p
 stores, the engine port with its `v1/` adapter, and the bundled model catalog. Legend Query hosts the page at
 `/query/cube`.
 
-**Status:** work in progress, version 0.0.x. This version has relational tables, Join, Sort, Filter, Restrict, Rename, Distinct, Drop, Limit and Slice, with one model, one
-runtime and one database per query. A cube is kept only through Export and Import of its spec, marked "(dev)", until
-the Cube store exists. A pasted model is kept in the cube. Nothing links to the page yet, there is no redo, and Cube sends no telemetry.
+**Status:** work in progress, version 0.0.x. This version has relational tables (from a model or a direct connection to H2
+or DuckDB), the access points of deployed data products and the data sets of deployed ingest definitions (both beta),
+Join, Sort, Group, Pivot (on the databases that run it), Filter, Restrict, Rename, Distinct, Drop, Limit, Slice, Concat, Compare Column Values (Difference),
+window functions (Apply Window Functions), computed columns (Extend Columns, whose expressions the engine checks) and
+rows kept by a Pure condition (Filter by Expression, which the engine checks too), with one model, one runtime and one database, data product project or producer deployment per query. A cube is kept only through Export and Import of its spec, marked "(dev)", until
+the Cube store exists. A pasted model is kept in the cube. The results grid works server-side by default: a header's
+sort and each drill-down level run on the engine, from the run on screen; local mode groups, pivots and filters the
+run's rows in the browser. The grid exports the whole query as CSV through the engine and the rows shown as Excel, lists
+the latest runs with their SQL, and formats number columns (a number, an ISO currency or a percent). Nothing links to the page yet, there is no redo, and Cube sends no telemetry.
 
 ## Layout
 
-| Folder                                | Holds                                                                                                                                                   |
-| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `src/components/`                     | the page, `CubeEditor`, and its parts: `canvas/`, `palette/`, `editors/`, `source-picker/`, `grid/`, `show-pure/`, `spec-transfer/`                     |
-| `src/stores/`                         | the MobX state of the page, its parts and the side panel; `editors/` (node drafts); `LocalModelCatalog` and `fixtures/` (the bundled model); `CubeHost` |
-| `src/graph-manager/`                  | the engine port (`CubeEngine.ts`) and `buildCubeEngine`                                                                                                 |
-| `src/graph-manager/protocol/pure/v1/` | the engine adapter, `V1_LegendCubeEngine`: the lambda serializer, the result and relation-type readers, the error mapping                               |
-| `src/__lib__/`                        | labels, help text, the command config and test ids                                                                                                      |
-| `src/__test-utils__/`                 | the fake engine and the helpers of the tests                                                                                                            |
-| `style/`                              | the stylesheet, built to `lib/index.css`                                                                                                                |
-
-It depends on `@finos/legend-cube`, legend-graph, legend-application, legend-art, legend-lego and legend-shared, never
+| Folder                                | Holds                                                                                                                                                    |
+| ------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `src/components/`                     | the page, `CubeEditor`, and its parts: `canvas/`, `palette/`, `editors/`, `source-picker/`, `grid/`, `show-pure/`, `spec-transfer/`                      |
+| `src/stores/`                         | the MobX state of the page, its parts and the node editor; `editors/` (node drafts); `LocalModelCatalog` and `fixtures/` (the bundled model); `CubeHost` |
+| `src/graph-manager/`                  | the engine port (`CubeEngine.ts`), the connection explorer, data product catalog and ingest catalog ports, and their builders                            |
+| `src/graph-manager/protocol/pure/v1/` | the engine adapter, `V1_LegendCubeEngine`: the lambda serializer, the result and relation-type readers, the error mapping                                |
+| `src/__lib__/`                        | labels, help text, the command config and test ids                                                                                                       |
+| `src/__test-utils__/`                 | the fake engine and the helpers of the tests                                                                                                             |
+| `style/`                              | the stylesheet, built to `lib/index.css`                                                                                                                 |
+
+It depends on `@finos/legend-cube`, legend-graph, legend-application, legend-art, legend-lego, legend-shared, and the
+depot, lakehouse and storage clients, never
 on an application package (`legend-application-*`) or on legend-query-builder.
 
 **The V1 rule:** `V1_*` protocol code lives under `src/graph-manager/protocol/pure/v1/`. The one product file outside
@@ -36,9 +43,20 @@ The engine-backed tests in `src/__tests__/` are an exception: they need both the
 
 `src/index.ts` exports:
 
-- `CubeEditor`, the page, and `CubeHost`, what a host gives it;
+- `CubeEditor`, the page, and `CubeHost`, what a host gives it; `CubeEntrySource`, a source a link asks the page to
+  start with (`initialSource`), and `formatCubeAccessPointEntryId` with its `CubeAccessPointEntry`, which write a
+  data product access point's link id ([hosting.md](./docs/hosting.md));
 - the engine port, everything in `CubeEngine.ts` (`CubeEngine`, `CubeEngineError`, `CubeEngineErrorKind`,
   `CubeResult`, `CubeModelOutline`, …), with `buildCubeEngine` and `CubeEngineConfig`;
+- the connection explorer port, everything in `CubeConnectionExplorer.ts`, with `buildCubeConnectionExplorer`;
+- data products: everything in `CubeDataProduct.ts` (the model kind, its runtime path, the environment types, the
+  warehouse rule) and `CubeDataProductCatalog.ts` (the catalog port and its classes), with
+  `buildCubeDataProductCatalog`, `buildCubeLakehouseEnvironment`, `CubeLakehouseServices`, `CubeLakehouseEnvironment`
+  and `getCubeRememberedWarehouse`;
+- ingest data sets: everything in `CubeIngest.ts` (the model kind, its runtime path, the saved settings) and
+  `CubeIngestCatalog.ts` (the catalog port and its classes), with `buildCubeIngestCatalog`;
+- published projects: everything in `CubeProject.ts` (the pointer model and its checks) and `CubeProjectCatalog.ts`
+  (the catalog port), with `buildCubeProjectCatalog`;
 - `LocalModelCatalog` and `BUNDLED_MODELS`;
 - `LEGEND_CUBE_COMMAND_CONFIG` and `LEGEND_CUBE_COMMAND_KEY`, the page's shortcuts;
 - `LEGEND_CUBE_TEST_ID`;
@@ -49,17 +67,38 @@ The engine-backed tests in `src/__tests__/` are an exception: they need both the
 The node-type registries (drafts, editors, help text and icons) are internal: there is no plugin API for node types
 yet.
 
-## New source kinds
+## Sources
+
+A cube reads one kind of source: relational tables (from a model, or from a database through a direct connection), the
+access points of deployed data products, or the data sets of deployed ingest definitions (both beta; PLAN §6.7, §6.8).
+The "Add a source" dialog has a tab per way to find a source, each implementing `CubeSourcePickerTab`
+(`src/stores/source-picker/`): Model, Database connection, Data product and Ingest. A host offers the last three by
+giving the page a `connectionExplorer`, a `dataProductCatalog` and an `ingestCatalog` (see [hosting](./docs/hosting.md)).
+
+A new kind of source needs, in the core, a `SourceDefinition` (node, codec, emitter and its query rules) and, in the
+builder, a tab, a Source panel editor, help text and an icon, and its branch in the engine's implementation, which
+builds the model each call runs on from the cube's saved model kind (`src/graph-manager/CubeDirectConnection.ts`,
+`CubeDataProduct.ts` and `CubeIngest.ts` are the examples).
+
+Tables of a published project's Databases come through the Project Database tab (PLAN §6.3), which extends the Sample Data tab: the
+cube saves the engine's pointer to the project at a released version, and the host's `projectCatalog` lists projects,
+versions and outlines from its depot.
+
+## Samples and examples
 
-Adding a kind of source (beyond relational tables) is not a registration yet. The builder is wired to relational
-tables in `CubeEditorState` (re-checking tables, opening the picker), `CubeSourcePickerState`, `CubeJoinDraft`,
-`CubeSourceEditor`, and the port's `resolveSchemas` and `CubeModelOutline`. New sources wait on the design of the
-sources dialog.
+The Sample Data tab bundles three datasets: Northwind (without Cube's test tables) and two made-up, seeded samples, Sports and Trades
+(`src/stores/fixtures/`), each an in-memory H2 database filled by its setup SQL. The Examples dialog (the header's
+Examples button) opens one of seven example cubes over them (`src/stores/CubeExamples.ts`) in place of the cube, then
+runs it, or starts a new cube on a dataset (PLAN §6.9). An
+example is built in code, so a new one needs no saved spec; add it to `CUBE_EXAMPLES`, and the engine test
+`CubeExamples.engine-roundtrip-test.ts` opens, types and runs it.
 
 ## Documentation
 
 - [Hosting the page](./docs/hosting.md): what a host gives the page, the shortcuts, the stylesheet, and the dev loop.
 - [Testing](./docs/testing.md): the test groups, the fake engine, and the tests against a real engine.
+- [Demo](./docs/demo/README.md): a walkthrough on the sample data, its [fast notes](./docs/demo/DEMO-FAST-NOTES.md), and
+  specs to import at each step.
 - [Adding an operation](./docs/adding-an-operation.md): the builder half of a new transform. The core half is in
   [`@finos/legend-cube`](../legend-cube/docs/adding-an-operation.md).
 - [The plan](../../docs/wip/legend-cube/PLAN.md) and [the known issues](../../docs/wip/legend-cube/ISSUES.md).

package/docs/adding-an-operation.md

@@ -5,21 +5,43 @@ node, its messages, emitter and codec, and its entry in the node registry. The p
 canvas read that registry, so the new type shows there with no change here.
 
 The builder adds four pieces: a draft, an editor and help text, in registries keyed by node type, and an icon, keyed by
-the icon name the node's definition gives. Join and Filter are the examples to follow.
+the icon name the node's definition gives. Join and Filter are the examples to follow; Group for an editor of rows
+(`CubeGroupEditor`, `CubeGroupDraft`), Concat for a two-input editor with fixes and one setting
+(`CubeConcatEditor`, `CubeConcatDraft`), Partition for an editor made of the shared rows
+(`CubePartitionEditor`, `CubePartitionDraft`), Difference for a two-input editor sharing Join's key rows
+(`CubeDifferenceEditor`, `CubeDifferenceDraft`), Extend for an editor that asks the engine before Apply
+(`CubeExtendEditor`, `CubeExtendDraft`), Filter by Expression for one condition the engine checks
+(`CubeExpressionFilterEditor`, `CubeExpressionFilterDraft`), and Pivot for an editor that loads values from the engine
+through `runCubeQuery` (`CubePivotEditor`, `CubePivotDraft`). A node type a database can't run is greyed, with the
+reason, in the palette and the Add menus (`getCubeAddDatabaseProblem` in `CubeAddPlacement.ts`): nothing to add for a
+new type but the core's rule.
 
 ## 1. A draft
 
-The side panel never edits the document: an editor edits a draft, and the panel's **Apply** stores the draft's node
-as one undo step (**Cancel** drops it).
+The node editor floats below its node (PLAN §11.8) and never edits the document: an editor edits a draft, and the
+editor's **Apply**, or closing it any way but **Cancel**, stores the draft's node as one undo step (**Cancel** drops
+it).
 
 - In `src/stores/editors/Cube<Type>Draft.ts`, extend `CubeNodeDraft<Type>`: observable fields for the settings being
   edited, actions that change them, and `build()`. `build()` returns `original` itself while nothing was edited, and
   otherwise a new node with the same id and `rest`. The panel also treats a node that saves the same as `original`
   as no change.
-- Register a factory in `CUBE_NODE_DRAFT_FACTORIES` (`src/stores/editors/CubeNodeDraftRegistry.ts`).
+- Register a factory in `CUBE_NODE_DRAFT_FACTORIES` (`src/stores/editors/CubeNodeDraftRegistry.ts`); it gets the
+  editor state too, for a draft that needs the engine.
+- A draft that must ask the engine before its node can be stored, as Extend's Validate does, says why in
+  `applyDisabledReason`: the panel then disables Apply with that reason as its title, and shows it in place of the
+  Problems list until it is `undefined`. Its engine work is a MobX `flow` on the draft; the node it builds then holds
+  the engine's answer (Extend's lambdas and typing), so it needs no more engine work once applied. While the reason
+  is set, Apply refuses the draft, and closing the panel or opening another node keeps it open with a notice
+  (`getEditorKeptOpenNotice`) rather than storing or dropping the edits.
+- A draft whose node the cube may replace underneath it with one that differs in nothing it edits, such as an Extend
+  typed in the background, overrides `follow(node)` to go on from the new node, keeping its rows; by default the panel
+  drops the edits with a notice (`CubeNodeEditorState.sync`).
 - A transform with nothing to set, such as Distinct, has no draft: it registers an editor that only describes it
   (`CubeDistinctEditor`) and is listed in `CUBE_NODE_TYPES_WITHOUT_SETTINGS`, which the registry test skips when it
-  asks for a factory. The panel then shows no Apply or Cancel (PLAN §7.4 item 2).
+  asks for a factory. The panel then shows no Apply or Cancel, and no Problems list (PLAN §7.4 item 2): every
+  transform that can be invalid has a draft. When such a transform gains a setting, as Concat did with Convert types,
+  give it a draft and take it off the list.
 
 ## 2. An editor
 
@@ -27,8 +49,28 @@ as one undo step (**Cancel** drops it).
   `editorState`, `draft`, `inputSchemas` (the inputs' schemas in port order; the panel shows the editor only once all
   are there) and `readOnly` (a cube saved by a newer version: show, don't edit).
 - It changes only the draft. An action on the document, such as Join's Swap Inputs, goes through
-  `editorState.nodeEditor`, which applies the draft first.
+  `editorState.nodeEditor`, which applies the draft first. A fix that adds nodes, as Join's "Rename them" and Concat's
+  "Rename them" and "Drop them" do, is a `can…` getter and an action there, made from the core's fix on the query with
+  the draft applied (`queryWithEdits`), as one undo step, the panel staying on the node. The editor shows the fix's
+  changes from the core's plan, given the draft's settings, so the editor and the store agree on what it offers.
 - `CubeColumnPicker` picks a column from a schema, and `CubeValueEditor` takes a value as a column's type wants it.
+- Join's key-pair rows are shared with Difference (`CubeJoinKeyRows`, over a `CubeJoinKeysDraft` from
+  `CubeJoinKeyPairs.ts`); `CubeColumnChecklist` takes a reason per column (`unpickableReason`), as Difference's
+  difference columns do.
+- A code editor is legend-lego's `CodeEditor` (language `pure`), with the error it should underline built from the
+  engine's location (`CubeEngineError.location`, as a legend-graph `CompilationError` with its `SourceInformation`),
+  as `CubeExtendEditor` does. Parse a text under a source id that names the row, so a parse error, and a typing error
+  on the text just parsed, point back to it. A stored lambda has no locations, and a plan has none either: the
+  Extend draft finds the row of a plan error by planning the columns' prefixes.
+- Rows that Group, Sort and Partition share: `CubeAggregationRowEditor` (column, function and output name, given its
+  function list, the `AggregationUse` and what a function that takes no column shows in the column's place, and a
+  field for a function's setting, Lag's and Lead's offset or NTile's bucket count, when the editor gives
+  `onSetting`; the row keeps a setting as typed text, parsed as Limit's size is),
+  `CubeSortRowEditor` (column, direction, move and remove; `getSortRowProblems`, `getTakenSortColumns`) and
+  `CubeColumnChecklist` (the input's columns, ticked, those that can't be compared shown but not tickable), with their
+  drafts' helpers in `CubeAggregationRows.ts` and `CubeSortRows.ts`. They are driven by callbacks, so a new editor
+  wires them to its own draft. Judge a row with the node Apply would store (`draft.build()`), as the Partition editor
+  does, so the row and the panel's Problems list agree.
   `isColumnDisabled` gives the reason a column can't be picked, shown after its type, as Sort does for a type that can't
   be sorted (`isSortableType`) and a column another row has; the column stays shown.
 - A whole-number setting (a size, a row index) is a `CubeIntegerField` over text the draft keeps as typed, read with
@@ -38,10 +80,14 @@ as one undo step (**Cancel** drops it).
   example.
 - Give each control a stable `aria-label` (numbered per row in a list, e.g. `Sort column 2`): the tests and the
   browser rehearsal find controls by it.
-- An editor must work in either host: today's side panel, or a floating editor anchored under the node, which M3
-  decides on (PLAN §11.4, §12.2 question 1). Don't rely on the panel's height: a list scrolls on its own (a
-  `max-height` with `overflow: auto`), column and direction pickers stay native `<select>` elements, as
-  `CubeColumnPicker` is, and no editor measures the panel or reads its size.
+- The editor shows in the floating node editor (PLAN §11.8): 432px wide, its body between 80px and a third of the
+  window, which scrolls. Fit that width (wrap long names with `break-words` or `break-all`), and give no list a
+  height cap or a scroller of its own: the body is the one scroller. Column and direction pickers stay native
+  `<select>` elements, as `CubeColumnPicker` is, and no editor measures the editor or reads its size.
+- Text a field stores on blur is kept when the editor closes: the editor's blur runs first. Text an editor keeps in
+  React state can register with `nodeEditor.addFlusher` to be applied on close, as the data product warehouse does.
+- A dropdown, picker or dialog opened from an editor and shown elsewhere (a portal) must be one of MUI's layers, or
+  hold the editor open with `nodeEditor.holdOpen()` while it is open: otherwise a press in it closes the editor.
 - A Tailwind class no other file uses yet, such as an arbitrary `grid-cols-[…]`, does nothing in the dev server until
   the deployment's Tailwind build runs again ([hosting.md](./hosting.md)); jsdom tests don't see it either way, so
   check a new layout in the browser.
@@ -56,6 +102,35 @@ text (§17.9) where it has one.
 
 Map the icon name of the node's definition to a legend-art icon in `NODE_ICONS` (`src/components/CubeNodeIcon.tsx`).
 
+## A node the engine types or checks
+
+Extend's types come from the engine (the core guide), and so does the word that a Filter by Expression's condition is
+one Boolean of a row (PLAN §11.11). `CubeEditorState` checks every such node that waits on `ERR_TYPING` or
+`ERR_CHECKING` through a reaction (`nodesToCheck`, `checkExpressions`), in one `typeLambdas` call, outside the undo
+history, through one path for both kinds (`src/stores/CubeEngineChecks.ts`): each kind gives a request, its lambdas
+and how to read the engine's answers into the node to store. The
+answer replaces the very nodes sent, in the cube shown and in the undo snapshots that give the engine the same input
+(`replaceOutsideHistory` takes a predicate per snapshot), and an answer for a node changed meanwhile is dropped. Each
+node is typed as a chain after its input's relation, one chain per first k columns (`src/stores/CubeExtendTyping.ts`),
+so a failure names its column. A condition is checked as its input's relation filtered with it, and typed alone as an
+extend column under a name no input column has, which names its type when it isn't one Boolean
+(`src/stores/CubeExpressionFilterCheck.ts`).
+
+A chain's types depend on what the engine sees of the input, which Cube's schema doesn't fully show (an Extend's
+columns are nullable to Cube, `[1]` to the engine after `toOne()`). So each typing records `upstream`, a digest of the
+input's emitted relation and the model (`getCubeExtendUpstream`), and the editor adds its own query rule to
+`analysis` (`createStaleCubeEngineCheckRule`): a node whose digest differs waits on `ERR_TYPING` or `ERR_CHECKING`
+and is checked again. Execute reads that validity as well as the emitter's `canEmit`, since the emitter doesn't know the rule. The
+digest covers the whole relation, filter values included, so any edit above an Extend has it typed again. A typing
+without a digest, as one loaded, is used as it is; on import every Extend is typed once more, as sources are checked
+again, which records one. When the engine can't be reached, a typing still current for its input is kept, with a
+warning. Every way out of `checkExpressions` leaves the node not stale, or the reaction would loop.
+
+The canvas shows a waiting node as resolving (`isTypingNode`), the header says so ("checking expressions"), and the
+panel leaves the waiting marker out of its problems. A new node type the engine types or checks joins
+`CubeEngineCheckedNode` and gives `buildCubeEngineCheckRequest` its request, so the reaction, the stale rule, the undo
+snapshots, the kept check offline and the import serve it too; its draft joins F10 in `CubeEditorState`.
+
 ## Warnings
 
 A warning is never an error: it doesn't make the node invalid or keep Execute from running. A warning worked out from
@@ -65,14 +140,25 @@ changed, is stored in `CubeEditorState.warnings`, by node key. `getNodeWarnings(
 node (`legend-cube__node--warning`) and lists them in its tooltip after its errors, and the panel shows each one as a
 `role="status"` line above the editor. A derived warning waits until the nodes it names have no errors of their own.
 
+Some warnings belong to one control, not to the node, and the editor shows them under it in the warning colour: Join's
+'type unknown' on a key whose column the model can't type, Concat's on any such column, and the Join and Filter
+editors' warning on a Date that Convert types made from dates and timestamps (`isDateOrTimestampType`), whose dates
+match timestamps only at midnight. An editor must not offer a change it warns about, as Concat's never offers Convert
+types for a column whose real type Cube doesn't know.
+
 ## Columns that change name
 
 The Join editor warns when a join key's column has no type in the model, tracing the column back to its table and the
 table's own name for it with `findColumnOrigins` (`src/stores/editors/CubeJoinDraft.ts`). It requires each node's
 output to have the column, maps a Rename's new name back to its old one, and follows a Join's same-named keys to the
-side the join keeps. Every other node is taken to pass its input's columns through under the same name. If a new
-transform's output columns aren't its inputs' under the same name (a computed column, say), teach `findColumnOrigins`
-how they map back, or the warning goes missing or shows on the wrong column.
+side the join keeps. A Group's keys and its Distinct Value, Min and Max outputs map back to their columns, and its
+counts, sums and averages to none, as a Partition's window functions do, its input columns passing through; a
+Concat's columns come from both inputs under the same name. Every other node is
+taken to pass its input's columns through under the same name: an Extend's input columns do, and its new columns,
+which the input lacks, come from none. A Difference's `x_1` and `x_2` come from `x` on their side, its difference
+from none. If a new transform's output columns aren't its inputs'
+under the same name (a computed column, say), teach `findColumnOrigins` how they map back, or the warning goes missing
+or shows on the wrong column.
 
 ## Tests
 
@@ -86,6 +172,19 @@ how they map back, or the warning goes missing or shows on the wrong column.
 - An engine test of the operation in `src/__tests__/LegendCubeOperations.engine-roundtrip-test.ts` (see
   [Testing](./testing.md)): its lambda as the engine parses the printed Pure, its typing against Cube's inferred
   schema, and what it returns, not its text.
+- A case of the new node type in `src/__tests__/CubeInferenceConformance.engine-roundtrip-test.ts`, whose coverage
+  test fails until there is one: the engine must type every node of the case as Cube infers it.
+- If its SQL can differ by database, its shapes at the end of `SHAPES` in
+  `src/__tests__/LegendCubeDialects.engine-roundtrip-test.ts` (earlier tests index the first ones by position), so the
+  checks every shape gets cover it, and a test of the facts its plans must keep on every database type. A shape some
+  database types can't plan at all goes in a list of its own over the types that can: windows are in
+  `WINDOW_SHAPES`, over `WINDOW_DATABASE_TYPES` (Spanner, Presto and Composite refuse any window), with the refusal
+  pinned.
+
+- A draft that waits for the engine is tested against the fake engine's mocks (`parseExpression`, `typeLambdas`,
+  `planLambda` in `FakeCubeEngine`), and an editor that changes a draft from the test wraps the change in `act`. An
+  editor with a code editor needs `MockedMonacoEditorInstance.getValue.mockReturnValue('')` before it renders: the
+  mocked editor holds no text of its own.
 
 The `v1/` adapter (`V1_CubeLambdaSerializer`) needs no change while the operation's emitter uses only IR and literal
 kinds it already writes. A new kind of IR node or literal needs its own case there, with a test in

package/docs/demo/DEMO-FAST-NOTES.md

@@ -0,0 +1,67 @@
+# Demo fast notes
+
+## Live build (Trades)
+
+1. **Examples → Trades → Start a new cube**: tick **TRADES**, **DESKS** → **Add 2**
+2. **Drop Join Another Input** (lands after TRADES = Left); drag **DESKS → Right**
+   - Join columns: **DESK_ID** = **ID** → Apply
+3. **Drop Group by Column**
+   - Group by: **DESK**, **REGION**
+   - Agg 1: **NOTIONAL_USD** · Sum · `TOTAL_NOTIONAL_USD`
+   - Agg 2: Count Rows · `TRADE_COUNT`
+   - Apply · **F9** → 6 rows
+4. **Drop Apply Window Functions**
+   - Partition: **REGION**
+   - Sort: **TOTAL_NOTIONAL_USD** · Descending
+   - Function: Rank · `RANK_IN_REGION`
+   - Apply · **F9** → Commodities 1, Rates 2, Equities Cash 3 (Americas)
+5. **Drop Extend Columns**
+   - Name: `NOTIONAL_MM`
+   - Expression: `x | $x.TOTAL_NOTIONAL_USD / 1000000` → Validate
+   - Apply · **F9** → Commodities 31.10
+6. **Drop Filter by Expression**
+   - `x | $x.NOTIONAL_MM > 25` → Validate
+   - Apply · **F9** → 4 rows
+7. **Show SQL** → Close · **Export** (fold the JSON)
+
+Duplicate-column error on a Join → open it → **Rename them**.
+
+**End spec:** [specs/trades-desks-5-final.cube.json](specs/trades-desks-5-final.cube.json) (Import → F9 → 4 rows).
+Steps in between: `specs/trades-desks-1-joined` … `-4-extended`.
+
+## Grid
+
+1. **Examples → Notional by desk and asset class**; right-click `join102` → **Select** → Execute (400 rows)
+2. Click **NOTIONAL_USD** header ×2
+3. Right-click REGION cell → **Drilldown by "REGION"**; DESK cell → **Drilldown by "DESK"**
+4. **Group rows…**: ids → Nothing, PRICE → Average
+5. Expand Americas → Commodities
+6. NOTIONAL_USD cell → **Format Column…**: Currency, 0 places
+7. **Runs** · Export ▸ **CSV (whole query)** · **Local**
+
+## Pivot (DuckDB)
+
+1. **Connect to a source → Direct Connection**: DuckDB, paste the setup SQL below
+   - Test connection → schema `demo` → **sales** → Add → **F9** (10 rows)
+2. **Drop Pivot by Column**
+   - Pivot column: **year** → Load values → All
+   - Rows: **country**
+   - Aggregation: Sum · **amount**
+   - Apply · **F9** → 4 rows (France 120 · 80 · 180)
+
+Setup SQL:
+
+```sql
+drop schema if exists demo cascade;
+create schema demo;
+create table demo.sales (id INTEGER PRIMARY KEY, country VARCHAR(10), product VARCHAR(10), year INTEGER, amount INTEGER, price DOUBLE);
+insert into demo.sales values (1, 'France', 'Tea', 2022, 120, 2.5), (2, 'France', 'Coffee', 2023, 80, 3.1), (3, 'France', 'Tea', 2024, 150, 2.7), (4, 'Germany', 'Tea', 2022, 90, 2.4), (5, 'Germany', 'Coffee', 2022, 60, 3.0), (6, 'Germany', 'Coffee', 2024, 110, 3.3), (7, 'Japan', 'Tea', 2023, 200, 2.9), (8, 'Japan', 'Tea', 2024, 40, 2.9), (9, 'France', 'Coffee', 2024, 30, 3.2), (10, 'Spain', 'Coffee', 2022, 70, 3.0);
+```
+
+Fallback: [specs/duckdb-sales-pivot.cube.json](specs/duckdb-sales-pivot.cube.json).
+
+## Function
+
+1. **Function** (palette) → Dataset Trades → **tradesAbove** · `1000000` → Add → **F9** → 44 rows
+
+**New** (header) clears the cube; Undo brings it back.

package/docs/demo/README.md

@@ -0,0 +1,102 @@
+# Legend Cube demo kit
+
+A walkthrough of Legend Cube on its sample data, and the specs to fall back on. Every number below is what the
+engine returned when these files were made (2026-10-13); the sample data is fixed, so they match on the day. The
+short version is [DEMO-FAST-NOTES.md](./DEMO-FAST-NOTES.md).
+
+| File                                      | What it is                                                                   |
+| ----------------------------------------- | ---------------------------------------------------------------------------- |
+| `specs/trades-desks-1-joined.cube.json`   | Act 2 after the Join: TRADES and DESKS joined on `DESK_ID` = `ID` (400 rows) |
+| `specs/trades-desks-2-grouped.cube.json`  | … then the Group by DESK and REGION (6 rows)                                 |
+| `specs/trades-desks-3-ranked.cube.json`   | … then the Rank within region (6 rows)                                       |
+| `specs/trades-desks-4-extended.cube.json` | … then the calculated column `NOTIONAL_MM` (6 rows)                          |
+| `specs/trades-desks-5-final.cube.json`    | … then Filter by Expression `NOTIONAL_MM > 25`: the finished cube (4 rows)   |
+| `specs/duckdb-sales-pivot.cube.json`      | Act 4's DuckDB cube with the Pivot, its setup SQL inside (4 rows)            |
+
+**Fallback:** if a live step goes wrong, **Import** the spec of the step before or after it and press **F9**. An
+imported cube opens without running.
+
+## Before the demo
+
+- A local engine, and the Query dev server (`yarn dev:query`) with Cube enabled (see [hosting](../hosting.md)).
+- Open each example once to warm the engine up.
+- Browser at about 1440×900, 100% zoom, a fresh `/query/cube` page.
+
+## 1. An example (about 1 min)
+
+1. **Examples → Top customers by orders**: it opens and runs, 10 rows, **Save-a-lot Markets** first with 31 orders.
+2. Click the **Group** step: its editor opens under it. **Escape** closes it.
+
+## 2. A live build on Trades (about 3 min)
+
+The question: which desk trades most, and how does each rank in its region?
+
+1. **Examples → Trades → Start a new cube**. Tick **TRADES** and **DESKS** (new: several tables at once), then
+   **Add 2**. TRADES, ticked first, is the step that runs.
+   - The source dialog's **← Examples** goes back to the examples.
+2. Drag **Join Another Input** onto the canvas: it lands after TRADES (Left). Drag DESKS' dot to the Join's lower
+   input (Right). The Join is red; its tooltip says _"Left join columns cannot be empty."_ Open it, **Add join
+   columns**: **DESK_ID** = **ID**, **Apply**. _(Fallback: `trades-desks-1-joined`.)_
+3. **Add Items ▾ → Group by Column**: tick **DESK** and **REGION**; aggregation 1 **NOTIONAL_USD**, **Sum**, named
+   **TOTAL_NOTIONAL_USD**; **Add aggregation**: **Count Rows** named **TRADE_COUNT**. **Apply**, **F9**: 6 desks.
+   _(`trades-desks-2-grouped`.)_
+4. **Add Items ▾ → Apply Window Functions**: partition by **REGION**; **Add sort column** **TOTAL_NOTIONAL_USD**
+   **Descending**; function **Rank** named **RANK_IN_REGION**. **Apply**, **F9**: Americas is Commodities 1, Rates 2,
+   Equities Cash 3; Europe is Equity Derivatives 1, Credit 2; Asia is FX 1. _(`trades-desks-3-ranked`.)_
+5. **Add Items ▾ → Extend Columns**: name **NOTIONAL_MM**, expression `x | $x.TOTAL_NOTIONAL_USD / 1000000`,
+   **Validate** (_"Float, can be empty"_), **Apply**, **F9**: Commodities 31,102,024.71 → **31.10**.
+   _(`trades-desks-4-extended`.)_
+6. **Add Items ▾ → Filter by Expression**: `x | $x.NOTIONAL_MM > 25`, **Validate**, **Apply**, **F9**: **4 desks**
+   (Commodities, Equity Derivatives, FX, Rates), their ranks unchanged. _(`trades-desks-5-final`.)_
+7. **Show SQL**: the formatted SQL in a dialog: the `rank() over (partition by …)` inside, the `> 25` filter outside.
+   **Copy SQL**, **Close**.
+8. **Export**: the spec as JSON, each object folding from the gutter.
+
+If two inputs share a column name that isn't a join key, the Join says _"Duplicate column names between inputs are not
+supported…"_: open it and click **Rename them**.
+
+## 3. The results grid (about 2 min)
+
+1. **Examples → Notional by desk and asset class**. Right-click the join step (`join102`) → **Select**, then
+   **Execute**: 400 trades. **Hide graph** for room.
+2. Click the **NOTIONAL_USD** header twice: sorted descending, on the engine (Runs goes up by 2).
+3. Right-click a REGION cell → **Drilldown by "REGION"**: Americas (189), Asia (68), Europe (143). Right-click a DESK
+   cell → **Drilldown by "DESK"**.
+4. **Group rows…**: set the id columns to **Nothing** and **PRICE** to **Average**; **Close**.
+5. Expand **Americas**, then **Commodities** (65): each level is its own query.
+6. Right-click a NOTIONAL_USD cell → **Format Column…**: **Currency**, decimal places **0**, **Apply**:
+   $81,410,352 for Americas.
+7. **Runs**: each run, sort and level fetched, with its SQL.
+8. Right-click → **Export ▸ CSV (whole query)**: `Notional_by_desk_and_asset_class.csv`, a header and 400 trades.
+9. **Local**: ag-grid's row groups, filters and the Columns panel with **Pivot Mode**. Back to **Server-side**.
+
+## 4. Pivot (about 1.5 min)
+
+1. On any sample cube (H2), point at **Pivot by Column** in the palette: greyed, _"Pivot is not supported on H2."_
+2. New page → **Connect to a source → Direct Connection**: Database **DuckDB**, paste the setup SQL below into
+   **Setup SQL**, **Test connection**, schema **demo**, pick **sales**, **Add**, **F9**: 10 sales.
+3. Palette **Pivot by Column**: pivot column **year**, **Load values** (2022, 2023, 2024), **All**, row column
+   **country**, aggregation **Sum** of **amount**. **Apply**, **F9**: a row per country, a column per year: France
+   120, 80, 180; Germany 150, –, 110; Japan –, 200, 40; Spain 70, –, –.
+4. _(Fallback: Import `duckdb-sales-pivot.cube.json`, F9.)_
+
+The DuckDB setup SQL:
+
+```sql
+drop schema if exists demo cascade;
+create schema demo;
+create table demo.sales (id INTEGER PRIMARY KEY, country VARCHAR(10), product VARCHAR(10), year INTEGER, amount INTEGER, price DOUBLE);
+insert into demo.sales values (1, 'France', 'Tea', 2022, 120, 2.5), (2, 'France', 'Coffee', 2023, 80, 3.1), (3, 'France', 'Tea', 2024, 150, 2.7), (4, 'Germany', 'Tea', 2022, 90, 2.4), (5, 'Germany', 'Coffee', 2022, 60, 3.0), (6, 'Germany', 'Coffee', 2024, 110, 3.3), (7, 'Japan', 'Tea', 2023, 200, 2.9), (8, 'Japan', 'Tea', 2024, 40, 2.9), (9, 'France', 'Coffee', 2024, 30, 3.2), (10, 'Spain', 'Coffee', 2022, 70, 3.0);
+```
+
+## 5. A function as a source (about 1 min)
+
+1. Palette **Function**: Dataset **Trades**, Read **Functions**: `tradesAbove(minNotionalUsd: Float)` and
+   `deskBook(desk, side?)`, which runs on its own runtime.
+2. Pick **tradesAbove**, value **1000000**, **Add**, **F9**: **44 trades** above a million dollars.
+
+## Not shown, and why
+
+- **Data products:** the data product servers don't run locally. The Data Product tab now takes several access points
+  at once, like tables.
+- **Services:** work in progress.

package/docs/demo/specs/duckdb-sales-pivot.cube.json

@@ -0,0 +1,110 @@
+{
+  "formatVersion": 1,
+  "context": {
+    "model": {
+      "_type": "cubeDirectConnection",
+      "connection": {
+        "_type": "RelationalDatabaseConnection",
+        "type": "DuckDB",
+        "databaseType": "DuckDB",
+        "datasourceSpecification": {
+          "_type": "duckDB",
+          "path": "",
+          "testDataSetupSqls": [
+            "drop schema if exists demo cascade",
+            "create schema demo",
+            "create table demo.sales (id INTEGER PRIMARY KEY, country VARCHAR(10), product VARCHAR(10), year INTEGER, amount INTEGER, price DOUBLE)",
+            "insert into demo.sales values (1, 'France', 'Tea', 2022, 120, 2.5), (2, 'France', 'Coffee', 2023, 80, 3.1), (3, 'France', 'Tea', 2024, 150, 2.7), (4, 'Germany', 'Tea', 2022, 90, 2.4), (5, 'Germany', 'Coffee', 2022, 60, 3.0), (6, 'Germany', 'Coffee', 2024, 110, 3.3), (7, 'Japan', 'Tea', 2023, 200, 2.9), (8, 'Japan', 'Tea', 2024, 40, 2.9), (9, 'France', 'Coffee', 2024, 30, 3.2), (10, 'Spain', 'Coffee', 2022, 70, 3.0)"
+          ]
+        },
+        "authenticationStrategy": {
+          "_type": "test"
+        }
+      }
+    },
+    "runtime": "cube::direct::Runtime"
+  },
+  "query": {
+    "selected": "pivot101",
+    "nodes": [
+      {
+        "kind": "relational",
+        "id": "relational101",
+        "database": "cube::direct::Database",
+        "schema": "\"demo\"",
+        "table": "\"sales\"",
+        "schemaSnapshot": [
+          {
+            "name": "id",
+            "type": {
+              "path": "meta::pure::precisePrimitives::Int"
+            },
+            "nullable": false
+          },
+          {
+            "name": "country",
+            "type": {
+              "path": "meta::pure::precisePrimitives::Varchar",
+              "params": [0]
+            },
+            "nullable": true
+          },
+          {
+            "name": "product",
+            "type": {
+              "path": "meta::pure::precisePrimitives::Varchar",
+              "params": [0]
+            },
+            "nullable": true
+          },
+          {
+            "name": "year",
+            "type": {
+              "path": "meta::pure::precisePrimitives::Int"
+            },
+            "nullable": true
+          },
+          {
+            "name": "amount",
+            "type": {
+              "path": "meta::pure::precisePrimitives::Int"
+            },
+            "nullable": true
+          },
+          {
+            "name": "price",
+            "type": {
+              "path": "meta::pure::precisePrimitives::Double"
+            },
+            "nullable": true
+          }
+        ]
+      },
+      {
+        "kind": "pivot",
+        "id": "pivot101",
+        "inputs": ["relational101"],
+        "rows": ["country"],
+        "column": "year",
+        "values": [
+          {
+            "kind": "integer",
+            "value": "2022"
+          },
+          {
+            "kind": "integer",
+            "value": "2023"
+          },
+          {
+            "kind": "integer",
+            "value": "2024"
+          }
+        ],
+        "aggregation": {
+          "column": "amount",
+          "function": "Sum"
+        }
+      }
+    ]
+  }
+}