@finos/legend-cube-builder 0.0.2 → 0.0.4

package/README.md

@@ -3,4 +3,96 @@
 The UI and the Legend engine adapter of Legend Cube, a canvas-based visual query builder. The domain model lives in the
 host-free [`@finos/legend-cube`](../legend-cube).
 
-Legend Query mounts Legend Cube at `/query/cube`.
+It holds the page (the canvas, the palette, the side-panel editors, the source picker and the results grid), its MobX
+stores, the engine port with its `v1/` adapter, and the bundled model catalog. Legend Query hosts the page at
+`/query/cube`.
+
+**Status:** work in progress, version 0.0.x. This version has relational tables (from a model or a direct connection to H2
+or DuckDB), the access points of deployed data products and the data sets of deployed ingest definitions (both beta),
+Join, Sort, Group, Filter, Restrict, Rename, Distinct, Drop, Limit, Slice, Concat, Compare Column Values (Difference),
+window functions (Apply Window Functions) and computed columns (Extend Columns, whose expressions the engine checks), with one model, one runtime and one database, data product project or producer deployment per query. A cube is kept only through Export and Import of its spec, marked "(dev)", until
+the Cube store exists. A pasted model is kept in the cube. Nothing links to the page yet, there is no redo, and Cube sends no telemetry.
+
+## Layout
+
+| Folder                                | Holds                                                                                                                                                    |
+| ------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `src/components/`                     | the page, `CubeEditor`, and its parts: `canvas/`, `palette/`, `editors/`, `source-picker/`, `grid/`, `show-pure/`, `spec-transfer/`                      |
+| `src/stores/`                         | the MobX state of the page, its parts and the node editor; `editors/` (node drafts); `LocalModelCatalog` and `fixtures/` (the bundled model); `CubeHost` |
+| `src/graph-manager/`                  | the engine port (`CubeEngine.ts`), the connection explorer, data product catalog and ingest catalog ports, and their builders                            |
+| `src/graph-manager/protocol/pure/v1/` | the engine adapter, `V1_LegendCubeEngine`: the lambda serializer, the result and relation-type readers, the error mapping                                |
+| `src/__lib__/`                        | labels, help text, the command config and test ids                                                                                                       |
+| `src/__test-utils__/`                 | the fake engine and the helpers of the tests                                                                                                             |
+| `style/`                              | the stylesheet, built to `lib/index.css`                                                                                                                 |
+
+It depends on `@finos/legend-cube`, legend-graph, legend-application, legend-art, legend-lego, legend-shared, and the
+depot, lakehouse and storage clients, never
+on an application package (`legend-application-*`) or on legend-query-builder.
+
+**The V1 rule:** `V1_*` protocol code lives under `src/graph-manager/protocol/pure/v1/`. The one product file outside
+it that names `V1_*` is the seam `src/graph-manager/protocol/pure/CubeEngineBuilder.ts`: `buildCubeEngine` is the only
+product code that makes a `V1_LegendCubeEngine`, so hosts never name a `V1_*` symbol. The repo's
+`enforce-module-import-hierarchy` lint rule forbids imports between `stores/` or `components/` and `v1/`, both ways.
+The engine-backed tests in `src/__tests__/` are an exception: they need both the adapter and `stores/`.
+
+## Public API
+
+`src/index.ts` exports:
+
+- `CubeEditor`, the page, and `CubeHost`, what a host gives it; `CubeEntrySource`, a source a link asks the page to
+  start with (`initialSource`), and `formatCubeAccessPointEntryId` with its `CubeAccessPointEntry`, which write a
+  data product access point's link id ([hosting.md](./docs/hosting.md));
+- the engine port, everything in `CubeEngine.ts` (`CubeEngine`, `CubeEngineError`, `CubeEngineErrorKind`,
+  `CubeResult`, `CubeModelOutline`, …), with `buildCubeEngine` and `CubeEngineConfig`;
+- the connection explorer port, everything in `CubeConnectionExplorer.ts`, with `buildCubeConnectionExplorer`;
+- data products: everything in `CubeDataProduct.ts` (the model kind, its runtime path, the environment types, the
+  warehouse rule) and `CubeDataProductCatalog.ts` (the catalog port and its classes), with
+  `buildCubeDataProductCatalog`, `buildCubeLakehouseEnvironment`, `CubeLakehouseServices`, `CubeLakehouseEnvironment`
+  and `getCubeRememberedWarehouse`;
+- ingest data sets: everything in `CubeIngest.ts` (the model kind, its runtime path, the saved settings) and
+  `CubeIngestCatalog.ts` (the catalog port and its classes), with `buildCubeIngestCatalog`;
+- published projects: everything in `CubeProject.ts` (the pointer model and its checks) and `CubeProjectCatalog.ts`
+  (the catalog port), with `buildCubeProjectCatalog`;
+- `LocalModelCatalog` and `BUNDLED_MODELS`;
+- `LEGEND_CUBE_COMMAND_CONFIG` and `LEGEND_CUBE_COMMAND_KEY`, the page's shortcuts;
+- `LEGEND_CUBE_TEST_ID`;
+- helpers that 0.0.2 published and nothing outside the package uses yet: `getRuntimesForDatabase`, `createTextModel`,
+  `BundledModel`, and the bundled model's `CUBE_NORTHWIND_MODEL`, `CUBE_NORTHWIND_DATABASE` and
+  `CUBE_NORTHWIND_RUNTIME`.
+
+The node-type registries (drafts, editors, help text and icons) are internal: there is no plugin API for node types
+yet.
+
+## Sources
+
+A cube reads one kind of source: relational tables (from a model, or from a database through a direct connection), the
+access points of deployed data products, or the data sets of deployed ingest definitions (both beta; PLAN §6.7, §6.8).
+The "Add a source" dialog has a tab per way to find a source, each implementing `CubeSourcePickerTab`
+(`src/stores/source-picker/`): Model, Database connection, Data product and Ingest. A host offers the last three by
+giving the page a `connectionExplorer`, a `dataProductCatalog` and an `ingestCatalog` (see [hosting](./docs/hosting.md)).
+
+A new kind of source needs, in the core, a `SourceDefinition` (node, codec, emitter and its query rules) and, in the
+builder, a tab, a Source panel editor, help text and an icon, and its branch in the engine's implementation, which
+builds the model each call runs on from the cube's saved model kind (`src/graph-manager/CubeDirectConnection.ts`,
+`CubeDataProduct.ts` and `CubeIngest.ts` are the examples).
+
+Tables of a published project's Databases come through the Project Database tab (PLAN §6.3), which extends the Sample Data tab: the
+cube saves the engine's pointer to the project at a released version, and the host's `projectCatalog` lists projects,
+versions and outlines from its depot.
+
+## Samples and examples
+
+The Sample Data tab bundles three datasets: Northwind (without Cube's test tables) and two made-up, seeded samples, Sports and Trades
+(`src/stores/fixtures/`), each an in-memory H2 database filled by its setup SQL. The Examples dialog (the header's
+Examples button) opens one of seven example cubes over them (`src/stores/CubeExamples.ts`) in place of the cube, then
+runs it, or starts a new cube on a dataset (PLAN §6.9). An
+example is built in code, so a new one needs no saved spec; add it to `CUBE_EXAMPLES`, and the engine test
+`CubeExamples.engine-roundtrip-test.ts` opens, types and runs it.
+
+## Documentation
+
+- [Hosting the page](./docs/hosting.md): what a host gives the page, the shortcuts, the stylesheet, and the dev loop.
+- [Testing](./docs/testing.md): the test groups, the fake engine, and the tests against a real engine.
+- [Adding an operation](./docs/adding-an-operation.md): the builder half of a new transform. The core half is in
+  [`@finos/legend-cube`](../legend-cube/docs/adding-an-operation.md).
+- [The plan](../../docs/wip/legend-cube/PLAN.md) and [the known issues](../../docs/wip/legend-cube/ISSUES.md).

package/docs/adding-an-operation.md

@@ -0,0 +1,181 @@
+# Adding an operation: the builder half
+
+Start with the core half, in [`@finos/legend-cube`'s guide](../../legend-cube/docs/adding-an-operation.md): the
+node, its messages, emitter and codec, and its entry in the node registry. The palette, the context menu and the
+canvas read that registry, so the new type shows there with no change here.
+
+The builder adds four pieces: a draft, an editor and help text, in registries keyed by node type, and an icon, keyed by
+the icon name the node's definition gives. Join and Filter are the examples to follow; Group for an editor of rows
+(`CubeGroupEditor`, `CubeGroupDraft`), Concat for a two-input editor with fixes and one setting
+(`CubeConcatEditor`, `CubeConcatDraft`), Partition for an editor made of the shared rows
+(`CubePartitionEditor`, `CubePartitionDraft`), Difference for a two-input editor sharing Join's key rows
+(`CubeDifferenceEditor`, `CubeDifferenceDraft`), and Extend for an editor that asks the engine before Apply
+(`CubeExtendEditor`, `CubeExtendDraft`).
+
+## 1. A draft
+
+The node editor floats below its node (PLAN §11.8) and never edits the document: an editor edits a draft, and the
+editor's **Apply**, or closing it any way but **Cancel**, stores the draft's node as one undo step (**Cancel** drops
+it).
+
+- In `src/stores/editors/Cube<Type>Draft.ts`, extend `CubeNodeDraft<Type>`: observable fields for the settings being
+  edited, actions that change them, and `build()`. `build()` returns `original` itself while nothing was edited, and
+  otherwise a new node with the same id and `rest`. The panel also treats a node that saves the same as `original`
+  as no change.
+- Register a factory in `CUBE_NODE_DRAFT_FACTORIES` (`src/stores/editors/CubeNodeDraftRegistry.ts`); it gets the
+  editor state too, for a draft that needs the engine.
+- A draft that must ask the engine before its node can be stored, as Extend's Validate does, says why in
+  `applyDisabledReason`: the panel then disables Apply with that reason as its title, and shows it in place of the
+  Problems list until it is `undefined`. Its engine work is a MobX `flow` on the draft; the node it builds then holds
+  the engine's answer (Extend's lambdas and typing), so it needs no more engine work once applied. While the reason
+  is set, Apply refuses the draft, and closing the panel or opening another node keeps it open with a notice
+  (`getEditorKeptOpenNotice`) rather than storing or dropping the edits.
+- A draft whose node the cube may replace underneath it with one that differs in nothing it edits, such as an Extend
+  typed in the background, overrides `follow(node)` to go on from the new node, keeping its rows; by default the panel
+  drops the edits with a notice (`CubeNodeEditorState.sync`).
+- A transform with nothing to set, such as Distinct, has no draft: it registers an editor that only describes it
+  (`CubeDistinctEditor`) and is listed in `CUBE_NODE_TYPES_WITHOUT_SETTINGS`, which the registry test skips when it
+  asks for a factory. The panel then shows no Apply or Cancel, and no Problems list (PLAN §7.4 item 2): every
+  transform that can be invalid has a draft. When such a transform gains a setting, as Concat did with Convert types,
+  give it a draft and take it off the list.
+
+## 2. An editor
+
+- In `src/components/editors/Cube<Type>Editor.tsx`, an `observer` component taking `CubeNodeEditorProps`:
+  `editorState`, `draft`, `inputSchemas` (the inputs' schemas in port order; the panel shows the editor only once all
+  are there) and `readOnly` (a cube saved by a newer version: show, don't edit).
+- It changes only the draft. An action on the document, such as Join's Swap Inputs, goes through
+  `editorState.nodeEditor`, which applies the draft first. A fix that adds nodes, as Join's "Rename them" and Concat's
+  "Rename them" and "Drop them" do, is a `can…` getter and an action there, made from the core's fix on the query with
+  the draft applied (`queryWithEdits`), as one undo step, the panel staying on the node. The editor shows the fix's
+  changes from the core's plan, given the draft's settings, so the editor and the store agree on what it offers.
+- `CubeColumnPicker` picks a column from a schema, and `CubeValueEditor` takes a value as a column's type wants it.
+- Join's key-pair rows are shared with Difference (`CubeJoinKeyRows`, over a `CubeJoinKeysDraft` from
+  `CubeJoinKeyPairs.ts`); `CubeColumnChecklist` takes a reason per column (`unpickableReason`), as Difference's
+  difference columns do.
+- A code editor is legend-lego's `CodeEditor` (language `pure`), with the error it should underline built from the
+  engine's location (`CubeEngineError.location`, as a legend-graph `CompilationError` with its `SourceInformation`),
+  as `CubeExtendEditor` does. Parse a text under a source id that names the row, so a parse error, and a typing error
+  on the text just parsed, point back to it. A stored lambda has no locations, and a plan has none either: the
+  Extend draft finds the row of a plan error by planning the columns' prefixes.
+- Rows that Group, Sort and Partition share: `CubeAggregationRowEditor` (column, function and output name, given its
+  function list, the `AggregationUse` and what a function that takes no column shows in the column's place, and a
+  field for a function's setting, Lag's and Lead's offset or NTile's bucket count, when the editor gives
+  `onSetting`; the row keeps a setting as typed text, parsed as Limit's size is),
+  `CubeSortRowEditor` (column, direction, move and remove; `getSortRowProblems`, `getTakenSortColumns`) and
+  `CubeColumnChecklist` (the input's columns, ticked, those that can't be compared shown but not tickable), with their
+  drafts' helpers in `CubeAggregationRows.ts` and `CubeSortRows.ts`. They are driven by callbacks, so a new editor
+  wires them to its own draft. Judge a row with the node Apply would store (`draft.build()`), as the Partition editor
+  does, so the row and the panel's Problems list agree.
+  `isColumnDisabled` gives the reason a column can't be picked, shown after its type, as Sort does for a type that can't
+  be sorted (`isSortableType`) and a column another row has; the column stays shown.
+- A whole-number setting (a size, a row index) is a `CubeIntegerField` over text the draft keeps as typed, read with
+  `parseWholeNumberText` (`src/stores/editors/CubeIntegerText.ts`): an optional sign and digits, nothing else, and empty
+  text gives `undefined`, which the node reports. Never `Number()` the raw text (it reads `''` as 0, `0x10` as 16 and
+  `1e3` as 1000) and never fall back to a default. `CubeRowCountDraft` and `CubeRowCountEditor` (Limit's) are the
+  example.
+- Give each control a stable `aria-label` (numbered per row in a list, e.g. `Sort column 2`): the tests and the
+  browser rehearsal find controls by it.
+- The editor shows in the floating node editor (PLAN §11.8): 432px wide, its body between 80px and a third of the
+  window, which scrolls. Fit that width (wrap long names with `break-words` or `break-all`), and give no list a
+  height cap or a scroller of its own: the body is the one scroller. Column and direction pickers stay native
+  `<select>` elements, as `CubeColumnPicker` is, and no editor measures the editor or reads its size.
+- Text a field stores on blur is kept when the editor closes: the editor's blur runs first. Text an editor keeps in
+  React state can register with `nodeEditor.addFlusher` to be applied on close, as the data product warehouse does.
+- A dropdown, picker or dialog opened from an editor and shown elsewhere (a portal) must be one of MUI's layers, or
+  hold the editor open with `nodeEditor.holdOpen()` while it is open: otherwise a press in it closes the editor.
+- A Tailwind class no other file uses yet, such as an arbitrary `grid-cols-[…]`, does nothing in the dev server until
+  the deployment's Tailwind build runs again ([hosting.md](./hosting.md)); jsdom tests don't see it either way, so
+  check a new layout in the browser.
+- Register it in `CUBE_NODE_EDITORS` (`src/components/editors/CubeNodeEditorRegistry.ts`).
+
+## 3. Help text
+
+An entry in `CUBE_NODE_HELP_TEXT` (`src/__lib__/LegendCubeHelpText.ts`), shown in the panel's header. Use the spec's
+text (§17.9) where it has one.
+
+## 4. An icon
+
+Map the icon name of the node's definition to a legend-art icon in `NODE_ICONS` (`src/components/CubeNodeIcon.tsx`).
+
+## A node the engine types
+
+Extend's types come from the engine (the core guide). `CubeEditorState` types every Extend that waits on `ERR_TYPING`
+through a reaction (`extendsToType`, `retypeExtends`), in one `typeLambdas` call, outside the undo history: the
+answer replaces the very nodes sent, in the cube shown and in the undo snapshots that give the engine the same input
+(`replaceOutsideHistory` takes a predicate per snapshot), and an answer for a node changed meanwhile is dropped. Each
+node is typed as a chain after its input's relation, one chain per first k columns (`src/stores/CubeExtendTyping.ts`),
+so a failure names its column.
+
+A chain's types depend on what the engine sees of the input, which Cube's schema doesn't fully show (an Extend's
+columns are nullable to Cube, `[1]` to the engine after `toOne()`). So each typing records `upstream`, a digest of the
+input's emitted relation and the model (`getCubeExtendUpstream`), and the editor adds its own query rule to
+`analysis` (`createStaleCubeExtendTypingRule`): an Extend whose digest differs waits on `ERR_TYPING` and is typed
+again. Execute reads that validity as well as the emitter's `canEmit`, since the emitter doesn't know the rule. The
+digest covers the whole relation, filter values included, so any edit above an Extend has it typed again. A typing
+without a digest, as one loaded, is used as it is; on import every Extend is typed once more, as sources are checked
+again, which records one. When the engine can't be reached, a typing still current for its input is kept, with a
+warning. Every way out of `retypeExtends` leaves the node not stale, or the reaction would loop.
+
+The canvas shows a waiting node as resolving (`isTypingNode`), the header says so, and the panel leaves `ERR_TYPING`
+out of its problems. A new node type the engine types needs the same: today the reaction, the rule and the typing
+are Extend's.
+
+## Warnings
+
+A warning is never an error: it doesn't make the node invalid or keep Execute from running. A warning worked out from
+the query, as the Sort warning is (`findLostSortOrders` in the core), goes in `CubeEditorState.derivedWarnings`, by
+node id, so it follows every edit and undo and is never stored; one about the outside world, such as a table that
+changed, is stored in `CubeEditorState.warnings`, by node key. `getNodeWarnings(node)` gives both: the canvas marks the
+node (`legend-cube__node--warning`) and lists them in its tooltip after its errors, and the panel shows each one as a
+`role="status"` line above the editor. A derived warning waits until the nodes it names have no errors of their own.
+
+Some warnings belong to one control, not to the node, and the editor shows them under it in the warning colour: Join's
+'type unknown' on a key whose column the model can't type, Concat's on any such column, and the Join and Filter
+editors' warning on a Date that Convert types made from dates and timestamps (`isDateOrTimestampType`), whose dates
+match timestamps only at midnight. An editor must not offer a change it warns about, as Concat's never offers Convert
+types for a column whose real type Cube doesn't know.
+
+## Columns that change name
+
+The Join editor warns when a join key's column has no type in the model, tracing the column back to its table and the
+table's own name for it with `findColumnOrigins` (`src/stores/editors/CubeJoinDraft.ts`). It requires each node's
+output to have the column, maps a Rename's new name back to its old one, and follows a Join's same-named keys to the
+side the join keeps. A Group's keys and its Distinct Value, Min and Max outputs map back to their columns, and its
+counts, sums and averages to none, as a Partition's window functions do, its input columns passing through; a
+Concat's columns come from both inputs under the same name. Every other node is
+taken to pass its input's columns through under the same name: an Extend's input columns do, and its new columns,
+which the input lacks, come from none. A Difference's `x_1` and `x_2` come from `x` on their side, its difference
+from none. If a new transform's output columns aren't its inputs'
+under the same name (a computed column, say), teach `findColumnOrigins` how they map back, or the warning goes missing
+or shows on the wrong column.
+
+## Tests
+
+- `src/components/editors/__tests__/CubeNodeEditorRegistry.test.ts` fails for a type missing a piece: it checks every
+  registered type for help text, an icon and an editor, and that each transform's draft gives back its node until
+  edited. Add the new help text to its exact list.
+- Add the new item, in the spec's menu order, to the lists in `src/components/palette/__tests__/CubePalette.test.tsx`
+  and to `TRANSFORMS` in `src/components/canvas/__tests__/CubeCanvasContextMenu.test.tsx`.
+- A draft test (`src/stores/editors/__tests__/`) and an editor test (`src/components/editors/__tests__/`), as Join's
+  and Filter's are.
+- An engine test of the operation in `src/__tests__/LegendCubeOperations.engine-roundtrip-test.ts` (see
+  [Testing](./testing.md)): its lambda as the engine parses the printed Pure, its typing against Cube's inferred
+  schema, and what it returns, not its text.
+- A case of the new node type in `src/__tests__/CubeInferenceConformance.engine-roundtrip-test.ts`, whose coverage
+  test fails until there is one: the engine must type every node of the case as Cube infers it.
+- If its SQL can differ by database, its shapes at the end of `SHAPES` in
+  `src/__tests__/LegendCubeDialects.engine-roundtrip-test.ts` (earlier tests index the first ones by position), so the
+  checks every shape gets cover it, and a test of the facts its plans must keep on every database type. A shape some
+  database types can't plan at all goes in a list of its own over the types that can: windows are in
+  `WINDOW_SHAPES`, over `WINDOW_DATABASE_TYPES` (Spanner, Presto and Composite refuse any window), with the refusal
+  pinned.
+
+- A draft that waits for the engine is tested against the fake engine's mocks (`parseExpression`, `typeLambdas`,
+  `planLambda` in `FakeCubeEngine`), and an editor that changes a draft from the test wraps the change in `act`. An
+  editor with a code editor needs `MockedMonacoEditorInstance.getValue.mockReturnValue('')` before it renders: the
+  mocked editor holds no text of its own.
+
+The `v1/` adapter (`V1_CubeLambdaSerializer`) needs no change while the operation's emitter uses only IR and literal
+kinds it already writes. A new kind of IR node or literal needs its own case there, with a test in
+`src/graph-manager/protocol/pure/v1/__tests__/`.

package/docs/hosting.md

@@ -0,0 +1,157 @@
+# Hosting the Cube page
+
+A Legend application hosts the page by giving it a `CubeHost` and contributing its shortcuts. Legend Query is the
+host today: `LegendQueryCubeHost` (`src/stores/cube/`), and the lazy route and page (`src/components/cube/`) of
+`@finos/legend-application-query`.
+
+## The host
+
+```ts
+interface CubeHost {
+  readonly applicationStore: GenericLegendApplicationStore;
+  readonly engine: CubeEngine;
+  readonly modelCatalog: LocalModelCatalog;
+  readonly connectionExplorer?: CubeConnectionExplorer | undefined;
+  readonly dataProductCatalog?: CubeDataProductCatalog | undefined;
+  readonly ingestCatalog?: CubeIngestCatalog | undefined;
+  readonly projectCatalog?: CubeProjectCatalog | undefined;
+}
+```
+
+- **The application store** gives the page its commands, user data, clipboard, theme, notifications and alerts, and
+  its telemetry service, which Cube doesn't use yet: it sends no events. Cube keeps no auth or config of its own.
+- **The engine:** `buildCubeEngine(config, tracerService)` builds Legend's implementation of the `CubeEngine` port.
+  `config` is the engine client's configuration (`CubeEngineConfig`); Legend Query passes its own engine server URL
+  and options, so Cube talks to the engine its query editor uses.
+- **The models:** `new LocalModelCatalog(engine)`. It offers `BUNDLED_MODELS` (the Northwind, Sports and Trades datasets of the Sample Data tab)
+  and caches each model's outline: its tables, for the source picker, and its runtimes' connections with their
+  database types, which Drop, Slice, Limit and Distinct need on some databases (PLAN §11.4). Users can also paste a Pure
+  model.
+- **Direct database connections (optional):** `buildCubeConnectionExplorer(config, tracerService)` builds the explorer
+  that reads the database behind a connection through the engine's schema exploration
+  (`/pure/v1/utilities/database/schemaExploration`), with its own client configured as the engine's. Without one,
+  the source picker offers no database connection. H2 and DuckDB only for now, and only secret-free authentication: a
+  cube saves its connection, so it never holds a password or token.
+- **Data products (optional, beta):** `buildCubeDataProductCatalog(services)` lists the deployed data products of a
+  class from the lakehouse's lite list and reads a product's access points from its deployed artifact and definition in
+  the depot. Pass `buildCubeLakehouseEnvironment(services)` to `buildCubeEngine` as `lakehouseEnvironment`, so data
+  product cubes run in the viewer's lakehouse environment. `services` holds the host's lakehouse contract and depot
+  clients, its token getter and the viewer's id; Legend Query builds them only when its `lakehouse` is configured. With
+  the optional `marketplaceServerClient`, the catalog's `search` runs the marketplace's Lakehouse Access full-text
+  search instead, one page of 100 matches, never reading the lite list, and its optional `isCutShort` says whether an
+  answer leaves out matches; Legend Query passes a client only when `marketplace.serverUrl` is also set. The optional
+  `getMarketplaceLink` builds a product's page in the host's marketplace, for the preview, the Source panel and the
+  "Request access" links; without it, no link shows. The catalog's `getAccess` reads the viewer's access to each access
+  point group from their contracts, and the optional `enterpriseStereotype` names the stereotype marking groups open to
+  everyone (Legend Query reads `options.dataProductConfig.publicStereotype`, as Studio and Marketplace do).
+
+- **Ingest data sets (optional, beta):** `buildCubeIngestCatalog(config, tracerService, services)` reads the deployed
+  ingest definitions as Data Cube's producer source does: the lakehouse platform's ingest environment named by the
+  viewer's environment in the class, its producer deployments, a deployment's definitions deployed from SDLC, and a
+  definition's grammar, which the engine Cube uses parses. It needs `services.platformServerClient` and
+  `services.ingestServerClient` (an ingest client with no server of its own: each call names one), and gives none
+  without them. Pass it to `buildCubeEngine` as `ingestCatalog` too, so ingest cubes type and run. Legend Query builds
+  both clients only when `lakehouse.platformUrl` is set.
+
+- **Published projects (optional):** `buildCubeProjectCatalog(depotServerClient)` lists the depot's projects, their
+  released versions, and a version's own Databases with the runtimes of it and its dependencies, for the source
+  dialog's Project tab (PLAN §6.3). It reads nothing until the tab opens. Project cubes save the engine's pointer, so
+  the engine must fetch from the same depot. Legend Query builds it from `depot.url`.
+
+Legend Query mounts the Cube page at `/query/cube` only where its config turns on non-production features,
+since Cube is still a proof of concept: `"extensions": { "core": { "NonProductionFeatureFlag": true } }`. Without
+it, as in production, the address shows nothing. To see it locally, add it to the deployment's
+`dev/config.json`, which is ignored by git. `yarn setup` writes that file without the flag, since turning it on also
+changes the rest of Query (its data space selector lists data products too), which Query's end-to-end tests check.
+
+In Legend Query's config file, data products need `lakehouse.url` and `depot.url`; ingest data sets need
+`lakehouse.url` and `lakehouse.platformUrl`, the key Data Cube and Marketplace use for the platform. Two keys are optional:
+`marketplace.serverUrl`, for search on the marketplace server, takes the value Legend Marketplace's own config gives
+`marketplace.url`; `extensions.core.dataProductConfig.publicStereotype`, for the "Enterprise access" badge, is the
+stereotype Studio and Marketplace use for groups open to everyone.
+
+```json
+{
+  "lakehouse": {
+    "url": "<the lakehouse contract server>",
+    "platformUrl": "<the lakehouse platform>"
+  },
+  "marketplace": {
+    "url": "<Legend Marketplace, for links>",
+    "productionParallelUrl": "<its production-parallel deployment>",
+    "serverUrl": "<the marketplace server>"
+  },
+  "extensions": {
+    "core": {
+      "dataProductConfig": {
+        "publicStereotype": {
+          "profile": "<profile path>",
+          "stereotype": "<stereotype>"
+        }
+      }
+    }
+  }
+}
+```
+
+Render the page with `<CubeEditor host={host} />`, and pass `initialDocument` to open a given cube, or
+`initialSource` to start an empty cube with one data product access point, as a link to Legend Query's Cube page does:
+`?sourceType=dataProductAccessPoint&sourceId=<id>`. The id is `<class>/<data product id>/<deployment id>/<access point
+group>/<access point>`, each part URI-encoded (`formatCubeAccessPointEntryId` writes it), where `<class>` is
+`PRODUCTION` or `PRODUCTION_PARALLEL` (`CubeDataProductEnvironmentType`), and the link encodes the
+whole id once more as the parameter's value, as any query value is (`encodeURIComponent(id)`). Legend Query reads the
+two parameters and takes them out of the address, keeping the others; a `?queryId=` wins and stays, for when Cube
+saves queries (M8). The page's state
+lives as long as the page; Legend Query makes a new host on each visit.
+
+**The bundled model, and H2 connections, run only on an engine that allows LocalH2.** It sets up Northwind in an
+in-memory H2 database through its connection's `testDataSetupSqls`. On an engine that forbids LocalH2, its queries don't
+run.
+
+## Shortcuts
+
+A Legend application binds keys only through its plugins. The host returns the page's commands from one of its
+plugins:
+
+```ts
+override getExtraKeyedCommandConfigEntries(): KeyedCommandConfigEntry[] {
+  return collectKeyedCommandConfigEntriesFromConfig(LEGEND_CUBE_COMMAND_CONFIG);
+}
+```
+
+The page registers the commands while it is open: F9 runs the query, and Ctrl+Z or Cmd+Z undoes.
+
+## The stylesheet
+
+- Import `@finos/legend-cube-builder/lib/index.css` in the host's stylesheet (Legend Query does it in
+  `legend-application-query-bootstrap/style/index.scss`). The page's root element has the class `legend-cube`.
+- The page uses Tailwind classes. The host's Tailwind build must scan the builder's sources: Legend Query's deployment
+  scans `../legend-*/src/**/*.tsx`. So Tailwind classes go in `.tsx` files only.
+- React Flow's stylesheet comes with the canvas. Load the page lazily, as Legend Query does, to keep it and the page's
+  libraries out of the host's main bundle.
+
+## The dev loop
+
+1. Start an engine on `localhost:6300` that allows LocalH2: legend-engine's `org.finos.legend.engine.server.Server`
+   from an IDE, or the repo's docker compose
+   (`fixtures/legend-docker-setup/grammar-test-setup/grammar-test-setup-docker-compose.yml`). Check it with
+   `curl -s localhost:6300/api/server/v1/info`.
+2. Run `yarn build` once: `yarn dev:ts` doesn't build the stylesheets.
+3. Run `yarn dev:ts` and `yarn dev:query`, and open `http://localhost:9001/query/cube`.
+4. Run `yarn workspace @finos/legend-cube-builder dev:sass` to rebuild the stylesheet as you change it (the root
+   `yarn dev:ts` already rebuilds the TypeScript).
+
+If Tailwind classes seem to do nothing, the deployment's Tailwind watcher may have stopped: it exits when
+`yarn dev:query` runs without a terminal. Restart it in a terminal of its own with
+`yarn workspace @finos/legend-application-query-deployment dev:tailwindcss`, or rebuild once with `build:tailwindcss`
+(again after each new class).
+
+**A second checkout.** To run a second copy next to the first (another git worktree, say), give it its own port and
+skip the watchers you don't need: in that checkout, run `yarn install`, `yarn build` and
+`yarn workspace @finos/legend-application-query-deployment setup` once, then
+`yarn workspace @finos/legend-application-query-deployment build:tailwindcss` and
+`yarn workspace @finos/legend-application-query-deployment dev:webpack --port 9002`, and open
+`http://localhost:9002/query/cube`. The dev server serves the packages' built `lib/`, so rebuild what you change
+(`yarn workspace @finos/legend-cube-builder build:ts`, and `@finos/legend-cube`'s) before looking, or run `yarn dev:ts`.
+
+The manual check of the whole page is the plan's Part B (PLAN §11.2).

package/docs/testing.md

@@ -0,0 +1,188 @@
+# Testing Legend Cube's builder
+
+The builder has two test groups:
+
+- **`core`** (the default group): unit tests and jsdom page tests against a fake engine. No engine needed:
+  `TEST_GROUP=core yarn workspace @finos/legend-cube-builder test`.
+- **`engine-roundtrip`**: tests against a real engine on `localhost:6300`, in files named
+  `*.engine-roundtrip-test.ts`: `TEST_GROUP=engine-roundtrip yarn workspace @finos/legend-cube-builder test`.
+
+A plain `yarn workspace @finos/legend-cube-builder test` runs both, so it fails without an engine. Add a path pattern to
+run some files, e.g. `TEST_GROUP=core yarn workspace @finos/legend-cube-builder test source-picker`. The core
+(`@finos/legend-cube`) has no engine tests.
+
+## Page tests, against a fake engine
+
+- `const { host, fake } = TEST__createCubeHost()` gives a host over `createFakeCubeEngine()`'s engine, and `fake`,
+  its Jest mocks, answering from `FAKE_NORTHWIND_OUTLINE` and `FAKE_NORTHWIND_SCHEMAS`. Override an answer per test,
+  e.g. `fake.execute.mockRejectedValueOnce(…)`, or pass answers to `TEST__createCubeHost(answers)`.
+- `TEST__renderInCubeApplication(<CubeEditor host={host} />, host.applicationStore, LEGEND_CUBE_TEST_ID.EDITOR)`
+  renders the page in a Legend application and waits for it.
+- A test of one part of the page makes the state itself, `new CubeEditorState(host, document)`, and renders the part
+  with it, e.g. `<CubeCanvas editorState={editorState} />`, which floats the node editor from the canvas when a node
+  is clicked (as `CubeJoinEditor.test.tsx` does). `TEST__importDocument(editorState, document)` then imports a cube and waits
+  for its tables to be typed again, so nothing changes after the test. A `<CubeEditor>` test gets the page's state by
+  spying on `CubeEditorState.prototype.registerCommands`, which the page calls once, and reading the spy's
+  `mock.contexts[0]` (as `CubeCommands.test.tsx`'s `renderPage` does); it imports a cube through the Import dialog.
+- `TEST__findCanvasNode(id)`, `TEST__getCanvasNodes()` and `TEST__getCanvasNodeTooltip(node)` read the canvas; the
+  tooltip's text is the node's `aria-description`.
+- `TEST__openAddItems()` and `TEST__chooseAddItem(label)` use the header's Add Items, e.g. to open the source dialog on
+  a table tab (`'Relational Database Table'`); the empty canvas's link opens it with no tab chosen.
+- `src/__test-utils__/CubeNorthwindTestQueries.ts` has the Northwind tables' columns and ready-made queries, e.g.
+  `sliceQuery()`.
+
+What jsdom can't do:
+
+- **Fit the canvas.** jsdom measures nothing, so React Flow never reports the nodes measured and the refit never runs.
+  Check fitting in a browser (PLAN §11.2 Part B).
+- **Show Chrome's own behaviour**, e.g. its date input sending React no Enter key.
+- **Print a failing comparison of two queries.** A query holds `BigInt`s, which Jest can't print, so a failing
+  `toBe` between queries crashes the worker. Compare them as booleans: `expect(a === b).toBe(true)`.
+
+The floating node editor (PLAN §11.8):
+
+- **A press outside it** closes it on `pointerdown`, which jsdom 20 has no event class for: dispatch
+  `new MouseEvent('pointerdown', { bubbles: true, button: 0 })`, as `useCubeNodeEditorDismiss.test.tsx` does
+  (`fireEvent.pointerDown` sends no button).
+- **Its placement** can be tested once the canvas's rectangle and the window's size are stubbed, as
+  `CubeNodeEditorPopper.test.tsx` does; flipping and scrolling are checked in a browser.
+- **A tooltip test** lets MUI's 800 ms memory of an open tooltip lapse after each test, or a retry (`jest.retryTimes`,
+  set for every package) passes on the shorter `enterNextDelay`, as `CubeCanvas.test.tsx`'s tooltip tests do.
+
+Drag and drop goes through react-dnd's HTML5 backend: fire `dragStart`, `dragEnter`, `dragOver`, `drop` and
+`dragEnd` on the elements.
+
+**Proving a test.** To show a new test fails without the code it covers, change the code in a copy of the package
+outside the checkout, never in place: a running `yarn dev:ts` rebuilds `lib/` from the changed file, and the dev server
+would serve it.
+
+## Engine tests
+
+- **The engine:** a local legend-engine, or the repo's docker compose
+  (`fixtures/legend-docker-setup/grammar-test-setup/grammar-test-setup-docker-compose.yml`), on `localhost:6300`. CI
+  runs the group on that docker image.
+- **No `fetch` in Jest:** the repo's Jest setup blocks it. Engine tests reach the engine through axios: the
+  `CUBE_ENGINE_TEST__*` helpers (`src/__test-utils__/CubeEngineTestSupport.ts`) for direct calls, and
+  `V1_createEngineBackedCubeEngine()` (`src/graph-manager/protocol/pure/v1/__test-utils__/`). The latter returns
+  `{ engine, calls }`: a real `V1_LegendCubeEngine` whose client calls are spied onto those helpers (and legend-graph's
+  `ENGINE_TEST_SUPPORT__grammarToJSON_model`), and the spies. Make it in `beforeEach` or in the test: the repo's Jest
+  config restores mocks before each test, so an engine made in `beforeAll` would call the real client, and the
+  blocked `fetch`.
+- **Assert meaning, not text.** The CI engine image changes with every engine merge, so assert row counts, the set of
+  keys returned, types and nullability, not SQL or Pure formatting, and never an engine answer that is a known
+  defect.
+- **Rows come back in the database's order** unless a Sort reaches the node that runs, and H2's order changes between
+  queries, so compare counts and sets, never positions, without a Sort. A Concat's rows come in no order either.
+  Facts of the Cube Northwind data a test may rely on: ORDERS has 830 orders, `ORDER_ID` running from 10248 to 11077
+  with no gap, in 21 ship countries; CUSTOMERS has 91 rows and SUPPLIERS 29; `CUBETEST.ALLTYPES` has 3 rows, ID 3
+  empty but for its key. Better still, check a result against its tables run alone, as the Concat tests do. Avoid
+  comparing or sorting on its 32-bit `REAL` columns (`FREIGHT`, `UNIT_PRICE`, `DISCOUNT`), where exact equality
+  silently matches nothing.
+- `src/__tests__/LegendCubeNorthwind.engine-roundtrip-test.ts` is the slice's automated acceptance (PLAN §11.2 Part
+  A). It logs the engine's commit.
+- `src/__tests__/LegendCubeOperations.engine-roundtrip-test.ts` checks each operation: its lambda as the engine parses
+  the printed Pure, its typing against Cube's inferred schema, and its rows. Its helpers are in
+  `src/__test-utils__/CubeOperationsTestUtils.ts`; they take any `CubeEngine` and import no `V1_*` class.
+- `src/__tests__/CubeInferenceConformance.engine-roundtrip-test.ts` holds Cube's inference to the engine's (PLAN
+  §11.5): it types every node of every case in one batch and compares names, positions, precise types with their
+  parameters, and nullability exactly (`TEST__typingDifferences`). Where the engine misreports nullability (an outer
+  join's padded columns, the FULL merged key, a Difference's keys named apart, an Extend's columns after `toOne()`), a
+  case lists the columns in `widerNullable`, and Cube must say nullable
+  there. Every registered node type needs a case, or the coverage test fails, but a data product's access point
+  (`NOT_TYPED_BY_THE_ENGINE`: the open-source engine reads no data product; `CubeDataProduct.engine-roundtrip-test.ts`
+  types its stand-in). A case whose Concat converts types names the types it converts (`converted`), so a case that
+  silently stops converting fails. A case's every node must be valid. The saved-spec samples are typed the same way,
+  node by node, but one-way (`CubeSpecCorpus.engine-roundtrip-test.ts`).
+- **Sort descending in order tests.** H2 scans ORDERS in ascending `ORDER_ID` order, so a test that sorts ascending,
+  or numbers rows by `ORDER_ID`, passes without the sort; the operations tests sort descending, with a control test
+  pinning H2's own order.
+- **Database workarounds** (PLAN §11.4) are checked two ways. The operations tests write them for `SqlServer` and run
+  them on the fixture's H2, which takes both forms, so their rows can be compared with the native forms'.
+  `src/__tests__/LegendCubeDialects.engine-roundtrip-test.ts` plans them, never running anything, on a test-only copy
+  of the fixture model with a static connection per database type (`CUBE_ENGINE_TEST__generatePlanSql`), and checks
+  facts about the SQL: row numbers in the Sort's order, no `limit m,n`, no `top N distinct`; for Group, its reduces,
+  HAVING and the GROUP BY target after two Renames (an engine issue, pinned); for Concat, one UNION ALL, each input's
+  Sort and Limit in its own subquery, and no SQL cast when it converts types. A check that should catch
+  a missing workaround must fail without it: run it once with the workaround off before relying on it.
+- **Windows** (PLAN §11.6) are planned in `WINDOW_SHAPES` over `WINDOW_DATABASE_TYPES`, since Spanner, Presto and
+  Composite refuse any window (pinned): an OVER clause with no frame, `count(col)` and `count(1)` for the counts,
+  `rank()`, `dense_rank()` and `row_number()`, M5b's `ntile`, `percent_rank`, `cume_dist`, `lag`, `lead` and
+  `first_value`, Last's over the reversed sort (ClickHouse's own `lagInFrame` and `leadInFrame` over a whole-partition
+  frame, DuckDB's `first`), with the reversed sort putting empty values at the other end on every type, a Filter after a Partition as a `WITH n_…` and a `WHERE` outside the
+  window, never QUALIFY, and the capture's ORDER BY at the root. Run, they are checked three ways: the operations
+  tests ("Partition on the engine"), with values worked out on the fixture (rows tied on the sort share a running
+  value and a rank); `LegendCubeDirectConnectionOperations.engine-roundtrip-test.ts` on H2 and DuckDB, a second
+  database that runs them, windowed Distinct Count included; and `CubeWindowComposition.engine-roundtrip-test.ts`,
+  which runs pairs and triples of nodes with a Partition against a small JavaScript reference with SQL's null rules
+  and default frame (and M5b's functions, refusing a tie for the ones that depend on a row's place), checks every Partition is written in the array form, and shows (its negative control) that the
+  single form, unbound, gives wrong rows. The core's `PartitionEmitter.test.ts` pins the emitted text.
+  `CubeWindowIsolation.engine-roundtrip-test.ts` checks the lets themselves through the adapter, with a test-only
+  window: they run, type, show in Show Pure, and put an error inside a let on its window.
+- **Difference and Extend** (PLAN §11.7). The operations tests run a Difference against a reference computed from its
+  inputs' own rows, every numeric family included; the plan-only test pins its join (a native `full outer join` on
+  every type but H2, where the engine emulates it) and its `coalesce(…, 0) - coalesce(…, 0)`, never a cast, and an
+  Extend's columns written as their expressions; both run on H2 and DuckDB through the direct connection.
+  `CubeExpressions.engine-roundtrip-test.ts` covers Extend's engine calls: an expression parsed with its digits and
+  locations, typed as a chain after its input over the cube's model (its own enumerations and functions too), the
+  failing column found and located, a plan that fails, and a run. A test that needs a typed Extend without the engine
+  builds one with `TEST__typedExtend` (`src/__test-utils__/CubeExpressionTestUtils.ts`), whose types the conformance
+  cases check against the engine.
+- `src/__tests__/CubeExamples.engine-roundtrip-test.ts` compiles the Sports and Trades sample models and opens every
+  example cube (PLAN §6.9) as the Examples dialog does: it types its tables, checks every node types as Cube infers it
+  (Sum outputs nullable to Cube only, PLAN §5.7) and runs it, counting rows.
+- `src/__tests__/CubeNorthwindRelationTypes.json` records the engine's relation type for every table of the bundled
+  model. If the test comparing with it fails, the engine's typing changed: check the change, then record the file
+  again by hand.
+
+### Direct connections
+
+- `TEST__createCubeHost()` also gives `connections`, a fake connection explorer
+  (`src/__test-utils__/FakeCubeConnectionExplorer.ts`) whose `listSchemas` answers `CUBE_DIRECT` and whose `listTables`
+  answers `FAKE_DIRECT_TABLES`, each a Jest mock.
+- On the engine, `V1_createEngineBackedCubeEngine()` also spies `buildDatabase` (schema exploration), and
+  `V1_createEngineBackedCubeConnectionExplorer()` gives a real explorer. `directH2Connection(schema)` and
+  `directDuckDBConnection(schema)` (`src/__test-utils__/CubeDirectConnectionFixtures.ts`) build small databases from
+  setup SQL.
+- **Setup SQL runs on every pooled connection**, so a setup must first drop what it creates. Each engine test file
+  still uses its own schema names, so files never depend on each other's data.
+- **Never assert an engine answer that is a known defect.** Schema exploration types H2's and DuckDB's `DECIMAL(10,2)`
+  as `Numeric(0,0)` and DuckDB's `VARCHAR(5)` as `Varchar(0)`; only the core's DuckDB sample
+  (`packages/legend-cube/src/spec/__tests__/fixtures/direct/`) holds those types, and no engine test reads it.
+
+### Data products and ingest data sets
+
+- **Fakes.** `TEST__createCubeHost()` also gives `dataProducts`, a fake data product catalog
+  (`src/__test-utils__/FakeCubeDataProductCatalog.ts`). An ingest catalog is opt-in: build one per test with
+  `createFakeCubeIngestCatalog()` (`src/__test-utils__/FakeCubeIngestCatalog.ts`) and pass its `catalog` as the host's
+  `ingestCatalog`; without it, Cube offers no Ingest tab.
+- **The open-source engine reads neither.** It has no data product or ingest definition to read, can't parse an ingest
+  definition, and can't type or run a `#I{...}#` accessor. So the engine tests stand each source in for with a Pure
+  function declaring the relation type the source would have, over an H2 table:
+  `src/__tests__/CubeDataProduct.engine-roundtrip-test.ts` for access points, and
+  `src/__tests__/CubeIngest.engine-roundtrip-test.ts` for ingest data sets. The ingest one swaps each accessor Cube
+  emits for its stand-in, then checks the nodes downstream (a Filter on `LAKE_OUT_ID`, a Join of two data sets, a
+  Group) against the engine's typing, and runs them.
+- **Testable locally:** reading a definition's data sets and types, the URN parser, the saved model, the source
+  dialog, the panels and the re-checks, all with fakes; downstream lambdas, on the stand-ins.
+- **Not testable locally:** the ingest servers' answers, the parse of a deployed definition's grammar, compiling and
+  running `#I{...}#`, the environment name and whether a warehouse is accepted. Those are checked in an internal
+  deployment (PLAN §11.2 Part B2).
+
+The engine tests in `src/__tests__/` import `V1_*` classes there, as an exception to the V1 rule: they need both the
+`v1/` adapter and `stores/`, which the import-hierarchy lint rule keeps apart.
+
+### Depot databases
+
+- **Fakes.** `createFakeCubeProjectCatalog()` (`src/__test-utils__/FakeCubeProjectCatalog.ts`) gives a catalog to pass
+  as the host's `projectCatalog`; without one, Cube offers no Project tab. Its `com.example:sales` holds the fake
+  engine's Northwind Database, so the fake engine types its tables.
+- **CI:** `src/__tests__/CubeDepotSamples.engine-roundtrip-test.ts` compiles the mock depot's committed sample
+  projects and types every table of their own Databases. The engine fetches a pointer's project from a depot, which CI
+  doesn't run, so no CI test sends a pointer.
+- **By hand (`cube-local`):** with the engine on :6300 and `yarn dev:mock-depot-server` on :6200,
+  `TEST_GROUP=cube-local yarn workspace @finos/legend-cube-builder test` runs the real catalog, the engine's pointer
+  fetch and the Project tab end to end (`src/__tests__/CubeDepot.cube-local-test.ts`). Its depot client sends its GETs
+  through axios, since the repo's Jest setup blocks `fetch`.
+- **Changing the sample projects:** edit `fixtures/legend-mock-server/scripts/generate-cube-depot.mjs` and run it with
+  the engine up (`node scripts/generate-cube-depot.mjs`). The engine caches a pointer's project per version, so restart
+  it after changing a version it has fetched.