@qoretechnologies/reqraft 0.10.54 → 0.10.55-pr.115.g24a94bf

package/.claude/CLAUDE.md

@@ -167,6 +167,7 @@ yarn build:test         # Type-check without emit
 
 - `yarn precheck` runs: lint → test → build:test:prod
 - `pre-push` hook enforces: `build:test:prod`, `lint`, `test`
+- CI (`.github/workflows/tests.yml`) runs precheck, build and the full story suite for PRs into `develop` **and** for every push to `develop`. GitHub skips `pull_request` workflows while a PR has merge conflicts, so a conflicting PR gets no test runs until it is rebased or merged up
 - Branch naming: always start with the issue number, e.g. `feature/49_pooled-connections`
 
 ### Versioning — every PR to `develop` MUST bump the version
@@ -201,6 +202,26 @@ yarn build:test         # Type-check without emit
 - Stories are co-located alongside source: `src/hooks/useFetch/useFetch.stories.tsx`
 - Storybook port: **6008**
 - Story + interaction tests run on **Vitest browser mode** (Playwright/Chromium): `yarn test:stories [file-substring]`. Run `yarn install-playwright` once first.
+- **Story network — mocks must follow the instance, and stories must not share connections.**
+  - Build every mock URL with `storyApiUrl(path)` / `storySocketUrl(path)` from
+    `src/stories/storyNetwork.ts`, never a hard-coded `https://hq.qoretechnologies.com:8092/…`.
+    Both mock layers match on host (storybook-addon-mock keys on `host + pathname`,
+    mock-socket on the URL minus its query), so a hard-coded host silently stops matching
+    when `REACT_APP_QORUS_INSTANCE` is overridden and the request goes to the real network.
+  - Requests components make on their own (FormEngine's `system/qorus-type-info`, the
+    expression catalogue `useExpressions` fetches whenever functions are allowed, the
+    reconnect probe `system/pid`) are mocked for every story via `GLOBAL_STORY_MOCK_DATA`
+    (`parameters.mockAddonConfigs.globalMockData`). Add new ones there, not per story file.
+  - A mock `response` must be an object, array or function — a bare number or string fails
+    the addon's validation and the entry is silently never matched (`TStoryMockResponse`
+    enforces this at compile time). The addon compares query-parameter *keys* only, so
+    `system?action=a&context=x` also answers `system?action=b&context=y`.
+  - `.storybook/preview.tsx` resets the shared LSP connections and the expression render
+    client before every story. Do not add per-file `ReqraftWebSocketsManager.closeAll()`
+    calls for LSP isolation.
+  - CI reaches the live hq instance with a valid token, so a story that leaks a real
+    request can pass in CI and hang locally. When a story passes in CI but times out on
+    your machine, look for an unmocked request before anything else.
 - **Visual regression via Qlip** (not Chromatic): the story run captures snapshots through the `qlip` Vitest plugin. Upload is gated on `QLIP_UPLOAD_TOKEN` (`vitest.config.ts`), so **local runs capture but never upload** — PNGs land in `qlip/screenshots/<ts>/stories/auto/<StoryId>.png` (gitignored); read the PNG to verify a visual change. CI (`.github/workflows/tests.yml`) sets the token and uploads a build for dashboard review. The full visual-change flow + the `qlip` MCP review skill live in the imported `stacks/frontend/FRONTEND.md`.
 
 ## Code Patterns & Conventions
@@ -260,6 +281,10 @@ yarn build:test         # Type-check without emit
 | `query()` returning stale data | Pass `cache: false`; POST/DELETE bypass cache automatically |
 | Heartbeat noise in logs | Filter `ev.data === 'pong'` — already done inside `useReqraftWebSocket` |
 | Storage path collisions | Use `includeAppPrefix: true` in `useReqraftStorage` to namespace per app |
+| Story shows FormEngine's loading skeleton / "Connecting to language server…" forever | An unmocked request or a connection inherited from an earlier story — see "Story network" above |
+| "No Preview — Sorry, but you either have no stories…" in a story failure | Not a registration failure: addon-vitest injects Storybook's hidden preview body into every test page, and a failed query's DOM dump prints it. Read the assertion above the dump |
+| A story run stops reporting and never times out (no failure, no output) | The page's main thread is blocked, so the in-page 30s test timer cannot fire — usually a synchronous render loop inside `act()`. Find the story with `--reporter=verbose` (the last ✓ is the one before it), then give Chromium `launchOptions: { args: ['--remote-debugging-port=9333'] }` in `vitest.config.ts` locally, enable `Debugger` over CDP BEFORE the freeze, and `Debugger.pause` once `Runtime.evaluate` stops answering to get the stack. Revert the config afterwards |
+| `yarn test:stories` hangs before any test runs (the port message is incidental) | Seen when the checkout path contains `+` — e.g. an auto-named `.claude/worktrees/fix+name` — while the same tree at a `+`-free path runs normally. Use a path without `+` |
 
 ## File Reference
 

package/.tasks/INDEX.md

@@ -58,6 +58,7 @@ needs phase 1.
 | [RENDER_EXPRESSION_LSP](./RENDER_EXPRESSION_LSP.md) | done pending user verify, branch `feature/render-expression-lsp` — server deployed + live-verified end-to-end 2026-06-10 (final gate 201/201 storybook, 268/268 jest, browser proof of server rendering) | **8th.** Server "Explain" over the LSP: new `dpql/renderExpression` in `QorusLspWebSocketHandler.qc` + `useRenderExpression` swap. Closes EXPRESSION_BUILDER_REPORT Phase D. Design: `design/RENDER_EXPRESSION_TRANSPORT.md`. |
 | [FIELD_STACK_REPORT](./FIELD_STACK_REPORT.md) | done pending user verify (uncommitted) — final gate 234/234 storybook (incl. 37 ported Template/Auto plays), 268/268 jest | **9th.** Verbatim re-port of the IDE's `Field/template.tsx` + `Field/auto.tsx` (the operand layer under the ported builder) + their IDE stories, replacing the from-scratch TemplateField/AutoFormField; closes the `isFunction` operand-flow gap (behavioral difference #4 in EXPRESSION_BUILDER_REPORT). Supersedes the from-scratch implementation from REQRAFT_AUTO_FIELD (its stories/assertions are kept and reconciled). |
 | [BRANCH_REMEDIATION](./BRANCH_REMEDIATION.md) | done pending user verify (uncommitted) — jest 268→356, plays 241→255, lint/tsc clean | **10th.** Burn-down of the 2026-06-10 full-branch audit: tests/stories/API/comment-voice remediation + behavior fixes (Select modal-click collapse, edge-triggered Url, ReqraftLspClient rename, CI auth wiring). Ledger: gitignored `SWEEP.local.md`. |
+| [STORY_NETWORK_ISOLATION](./STORY_NETWORK_ISOLATION.md) | done pending user verify, folded into `fix/per-field-templates` (PR #115), not pushed | **Standalone.** ExpressionField stories failed off-network on `develop` (passed in CI only because CI reaches the live instance): LSP connections leaking across stories, an unhandled socket give-up in `LspSharedConnection`, unmocked background requests, and mock URLs hard-coded to the hq host. |
 | [TEMPLATE_TOKEN_GRAMMAR](./TEMPLATE_TOKEN_GRAMMAR.md) | committed `0ef167c`, in PR (branch `bugfix/template-token-grammar`) — user-verified 2026-08-27; 13/13 unit, 21/21 template plays, qlip PNGs read | **11th (bugfix).** Braced `$data:{…}` state-output refs rehydrate as raw text: canonical token grammar + strict whole-token check + TemplateField renders whole-token values as the picker chip. Twin fix applied in qorus-ide the same day; ships as a normal per-PR patch release (`0.10.37`). |
 | [EXPRESSION_FIELD_SEAMS](./EXPRESSION_FIELD_SEAMS.md) | committed `f5fb2ee`, pushed (branch `bugfix/116_expression-field-seams`) — user-verified 2026-09-07; PR not yet opened | **12th (bugfix).** The `extraActions` seam stopped one hop short of the `ExpressionField` shell hosts mount, and the builder's operand `TemplateField` mounts dropped it again — qorus-ide lost its AI-assist button on every expression card when it adopted the shell ([#116](https://github.com/qoretechnologies/toolkit-react/issues/116)). Forward it through both hops; pin with a nested-operand story; ships as a per-PR patch (`0.10.51`; re-based onto `4860e6e` after #114 published 0.10.50). |
 | [EXPORT_EXPRESSION_BUILDER](./EXPORT_EXPRESSION_BUILDER.md) | committed `0eea9e7`, pushed, PR open (branch `feature/119_export-expression-builder`) | **13th (surface fix).** The barrel exported the `ExpressionField` shell but not the bare `ExpressionBuilder`, so a host mirroring the library's root/nested rule had to deep-import `dist/…/builder` ([#119](https://github.com/qoretechnologies/toolkit-react/issues/119)). Export + a barrel test; ships as `0.10.53`. |

package/.tasks/STORY_NETWORK_ISOLATION.md

@@ -0,0 +1,98 @@
+# Story network isolation — ExpressionField stories fail off-network
+
+**Status:** done pending user verify — folded into `fix/per-field-templates`
+(PR #115) so every open reqraft fix ships in one PR; not yet pushed.
+Originally built and verified on `develop` (stories 479/479, unit 930/930,
+off-network), then re-verified on the combined branch.
+
+## Symptom
+
+`yarn test:stories ExpressionField` failed 6 of 8 on `develop` locally, while the
+same file passed in CI (run 34250300001, 2026-09-08). Siblings failed the same
+way: FormEngineRemote 11/11, AutoFormField 1, SchemaDefinitionField 2,
+useWebsocket 3, TemplateField 1.
+
+## Root causes
+
+1. **An LSP connection outlived the story that opened it.** `useRenderExpression`
+   holds the endpoint's `LspSharedConnection` for the page lifetime. `Default`
+   (no mock LSP) dialled the real instance; `Empty` / `Text Mode` then joined that
+   connection's never-settling handshake, so their mock LSP was never dialled
+   ("Connecting to language server…" forever). Each passes when run alone.
+2. **The shared connection had no way out of a dead socket** (product bug). No
+   `onReconnectFailed` handling: a first handshake whose socket gave up stayed
+   pending forever for every later client; one that had completed kept reporting
+   the dead socket as ready. `close()` also abandoned an in-flight handshake, and
+   `release()` could evict a connection that had replaced its own after a reset.
+3. **Requests components make on their own were unmocked.** FormEngine gates on
+   `system/qorus-type-info`; TemplateField's "Use Expression" waits on the
+   expression catalogue; every `ReqraftWebSocket` reconnect awaits a `system/pid`
+   probe. Against an unreachable instance none settles — the form stays a
+   skeleton, the menu entry never appears, the socket never reconnects or gives
+   up. CI's live instance answered all three, which is why CI stayed green.
+4. **Mock URLs hard-coded the hq host.** Both mock layers match on host, so
+   overriding `REACT_APP_QORUS_INSTANCE` silently disabled every mock.
+
+## Fix
+
+- `src/utils/lspClient.ts` — give-up / close settle the handshake and forget the
+  socket; retry paths guarded by identity; `release()` only evicts itself.
+  4 new unit tests + the disconnect-mid-handshake test now asserts the rejection.
+- `src/stories/storyNetwork.ts` — `STORY_QORUS_INSTANCE`, `storyApiUrl`,
+  `storySocketUrl`, `GLOBAL_STORY_MOCK_DATA` (types catalogue, empty expression
+  catalogue, reconnect probe),
+  `TStoryMockResponse` (the addon silently ignores non-object responses).
+- `.storybook/preview.tsx` — `mockAddonConfigs.globalMockData`, per-story LSP
+  reset in a project `beforeEach`; `.storybook/vitest.setup.ts` mirrors the
+  addon's global + story merge.
+- All mock URLs moved to the helpers; FormEngine's duplicate types mock and
+  DpqlEditor's ad-hoc `closeAll()` removed.
+
+### Reconciled with `fix/per-field-templates`
+
+- That branch had already met the leak from the fixture side:
+  `createMockLspServer` reset the shared connections on create and close. The
+  project-level `beforeEach` covers the same ground plus stories that never
+  create a mock server (and the render client), so the fixture resets were
+  dropped in favour of the one mechanism.
+- Its `query()` strips a path's leading slash and uses a `noApiPrefix` url
+  verbatim; `buildReqraftApiUrl` follows those rules, so the doubled-slash mock
+  spellings are gone.
+- **A page-freezing render loop in the row menu** (from `a21640b`), reachable
+  only once FormEngine rendered offline: `FormEngine › Compact Expressions`
+  froze the tab and hung the whole suite with no output. The row kept ONE
+  `{key, items}` slot, and an expression's operands — separate TemplateFields
+  in the same row — published different keys into it, each overwrite
+  re-rendering the row and both operands. The same slot also kept an unmounted
+  editor's handlers when a reopened row offered the same key. Fixed in
+  `rowMenuContext.ts` (per-publisher registration via `useId`, withdrawn on
+  unmount, clicks delegated to the latest handler) and `TemplateField` (claims
+  the channel and hides it from what it renders, so operands keep their own
+  menu). Unit tests reproduce the loop and both stale-handler cases.
+- `ExpressionField › Field Menu Stays In The Toolbar` had gone stale on that
+  branch: `a21640b` moved a row field's menu into the row's ⋮ (one menu per
+  control) after the story was written. The story now asserts that single
+  menu — the row's `.options-readfirst-more`, with no `.template-more` beside
+  it — still sits level with the toolbar.
+
+## Not causes (checked)
+
+- "No Preview" in failure output: addon-vitest injects Storybook's hidden preview
+  body into every test page; a failed query's DOM dump prints it.
+- A `+` in the checkout path hangs `yarn test:stories` before any test runs
+  (mechanism not traced); unrelated to the stories.
+
+## Follow-ups (not done)
+
+- `users?action=current` is still unmocked for most stories — harmless today
+  (`waitForStorage={false}`), but CI renders a real user while local runs render
+  none. Candidate for a default mock once permission-gated snapshots are checked.
+- Unmocked requests still pass through to the network. Failing them loudly for
+  non-live stories would make this class impossible rather than documented.
+- `ReqraftWebSocketsManager.closeAll()` deletes entries by `socket.url` rather than
+  the pool key, and closing a CONNECTING socket reports code 1006, which schedules
+  a reconnect. Not exercised by this fix.
+- ~~`tests.yml` runs only on `pull_request`~~ — done on this branch: it now also
+  runs on pushes to `develop` (the push trigger had been dropped in 2021 with no
+  recorded reason). Conflicting PRs still get no `pull_request` runs until they
+  merge up.

package/README.md

@@ -148,7 +148,7 @@ client.onNotification('qonsole/sessionStateChanged', (params) => { /* … */ });
 client.disconnect();
 ```
 
-Clients on the same LSP endpoint share ONE underlying WebSocket — each `ReqraftLspClient` is a per-document facade over the shared connection, multiplexed by document URI (the server keys language sessions per document), so N editors cost one socket. Auto-reconnect re-opens every document on the shared socket, 15s request timeout, request/response correlation by `id`, pending requests rejected on close. Document URIs should be opaque and client-generated — per the Qonsole LSP contract they must not contain session tokens, usernames, sandbox identifiers, or other secrets that end up in server logs.
+Clients on the same LSP endpoint share ONE underlying WebSocket — each `ReqraftLspClient` is a per-document facade over the shared connection, multiplexed by document URI (the server keys language sessions per document), so N editors cost one socket. Auto-reconnect re-opens every document on the shared socket, 15s request timeout, request/response correlation by `id`, pending requests rejected on close. A `connect()` never outlives its socket: once the socket exhausts `maxReconnectTries`, a handshake still waiting rejects, and the next `connect()` — from that client or any other — dials again, so an editor opened after an outage connects once the server is back, without a page reload. Document URIs should be opaque and client-generated — per the Qonsole LSP contract they must not contain session tokens, usernames, sandbox identifiers, or other secrets that end up in server logs.
 
 ## Community
 

package/design/FORM_ENGINE_EXPRESSIONS.md

@@ -124,6 +124,116 @@ templates/return-type, and bind a provider only when the field supplies
 one. Worst case, text mode still parses/serializes correctly with no
 provider; only completions are reduced.
 
+## Detecting an expression typed as plain text
+
+*Added 2026-09-06. As-built.*
+
+An author who never opens the expression view still types DPQL into the
+ordinary editor. `TemplateField` notices, and what it does next depends on
+the field's own type.
+
+**A successful parse means nothing.** This is the finding the whole design
+rests on, and it is counter-intuitive enough to be worth stating plainly.
+Verified against the live LSP: `dpql/parse` returns `success: true` for
+`hello`, for `42`, for `2026-09-06` and for `@a`. A bare literal is a valid
+DPQL program, so "does it parse?" cannot separate an expression from a
+value. What separates them is the shape of the AST that comes back:
+
+| typed text | `exp` | verdict |
+|---|---|---|
+| `hello`, `42`, `2026-09-06`, `@a` | `value` | a literal |
+| `$local:count` | `template` | a template |
+| `@a > 5`, `"a" + "b"`, `toInt("5")`, `@n LIKE "a%"` | `>`, `+`, `toInt`, `like` | an expression |
+
+Text that is not DPQL at all is safe on its own: `hello world`, `x@y.com`,
+`a - b test` and `{"a":1}` all fail to parse outright.
+
+**Three stages, in `helpers/dpqlDetection.ts`:**
+
+1. `mightBeDpqlExpression` — a cheap lexical pass whose ONLY job is to keep
+   a server round-trip off every keystroke. It decides nothing. `-` and `/`
+   require surrounding spaces so a date or a path never triggers a probe.
+2. `dpql/parse` on the shared probe document (below), debounced 400 ms.
+3. `classifyTypedText` — `switch` when the text could not be a literal of
+   the field's declared type (the form's own `validateField` answers that),
+   `offer` when it could, `none` otherwise.
+
+**Why the asymmetry.** Rewriting the string `a + b` into a concatenation on
+a `string` field would destroy a value the author meant, so a type that
+accepts any text is only ever OFFERED the switch. Text that could not be a
+literal there was already an error the moment it was typed, so switching
+costs nothing and explains the error. Every switch carries an Undo that
+restores the exact text, and a declined text is never asked about again.
+
+**Reachability.** The auto-switch only applies where the editor accepts free
+text — a constrained `string` (identifier / `validation_regex`), `binary`,
+`hash`. `int` and `float` render `input[type=number]`, which drops `@`, `>`
+and spaces before any handler runs, so detection there is unreachable by
+design of the editor and the ⋮ menu remains the way in. Pinned by a test.
+
+**Template precedence.** `isValueTemplate` is deliberately loose (starts
+with `$`, contains `:`), so `$local:count + 1` was being swallowed into
+template mode with its arithmetic as dead text. A value that is not one
+complete template token and carries an operator now stays in the plain
+editor, where detection can reach it. A token standing alone
+(`$local:count`, `$data:{1.field}`) still flips, unchanged.
+
+**The probe document** (`components/dpqlEditor/dpqlProbe.ts`). One shared,
+lazily-opened DPQL document for the whole app, reference-counted by
+`useDpqlProbe` and closed when the last field using it unmounts. It is
+cheap: `ReqraftLspClient` multiplexes every document over ONE socket per
+endpoint and `initialize` runs once for it, so a probe is one extra
+`didOpen` plus one server-side session — not a connection. It is still
+shared rather than per-field, because a form with forty expression-capable
+options would otherwise open forty sessions to ask the same question.
+
+The document binds NO provider context; verified live, a context-less
+`didOpen` still builds a session that `dpql/parse` answers on. It sends no
+`target_type` either: detection asks whether the text is an expression, not
+whether it is the right type — a mismatch is something to show the author
+inside the editor, never a reason to refuse to open it. A probe never
+throws; an unreachable instance simply leaves the typed text alone.
+
+## Return-type checking
+
+*Added 2026-09-06. As-built.*
+
+`dpql/parse` has always been able to answer "can this expression's result
+satisfy type X?" — it takes a `target_type` and returns `inferred_type`,
+`type_compatible`, `auto_coercible` and a `suggested_fix`. Nothing in the
+front end ever sent one, so it never answered.
+
+`ExpressionField` now sends the field's own type with every parse, and
+shows what comes back. Three outcomes, not two:
+
+| the expression | shown |
+|---|---|
+| already fits | nothing |
+| converts, and the conversion is TOTAL (a number used as text) | nothing |
+| converts, but the conversion may FAIL (text used as a number) | a warning, with the conversion offered |
+| cannot convert at all | a danger message, with the conversion offered |
+
+The middle row is why the server grew `coercion_may_fail`: `canAutoCoerce()`
+admits both kinds, and reporting a conversion that cannot fail is what
+teaches authors to ignore the reports that matter.
+
+`auto` and `any` are never sent as a target — they accept anything, so the
+answer would always be "compatible" and the round trip would buy nothing. A
+field offering a CHOICE of return types sends nothing either, rather than
+asking about one of them arbitrarily.
+
+**What the field cannot decide.** Whether text really denotes a number
+depends on the text, which does not exist until the expression runs. So the
+warning is a warning, not a refusal, and the value is checked again
+server-side at run time — see `design/test-interface-object.md` in the qorus
+repo. For that to be possible the FormEngine's stored envelope carries the
+field's `type` beside `is_expression` and the AST, which is the type the
+author was shown.
+
+The same analysis lives in two server handlers (the LSP's `dpql/parse` and
+the Creator WS equivalent); both were changed together. Deduplicating them
+is a separate cleanup.
+
 ## Out of scope
 
 - `server_expression_handling` server-function-vs-qorus-function

package/design/FORM_ENGINE_FIELD_MIGRATION.md

@@ -129,6 +129,158 @@ Create `src/components/form/fields/auto/AutoFormField.tsx`:
   collapsing to `'long-string'`. **This is the one behaviour change for
   existing consumers** — see Migration impact.
 
+**Revised 2026-09-14 — an untyped field is typed into, never asked for a
+type.** No field shows the type picker's *"Please select data type"*
+anymore. It asked a question about storage ("is this Text or a Number?")
+that an author writing a value often cannot answer, before they could write
+anything, and it replaced a typable field with a pick-only one.
+
+- **`AutoFormField`**, for a type that is still `auto` / `any`, renders the
+  long-string editor and stores what is typed as untyped — exactly what
+  template mode stores. The picker appears only where there is a real
+  choice: the schema allows several concrete types (`allowed_types`), or
+  the field already holds a value of a concrete type.
+- **An explicit type** is chosen from the field's ⋮ menu — the per-type
+  entries under *Set Custom Value* (see `engine/typeChoices.ts`).
+- **`TemplateField`** opens an empty untyped field that may hold a custom
+  value in template mode, whose editor is typable and offers the templates.
+  It decides that from the field's props, not from whether the template list
+  has arrived: on a cold load the type-filtered list does not exist at
+  mount, and a landing chosen from it put every such field on the picker.
+- A field that may NOT hold a custom value (FormEngine passes
+  `allowCustomValues={false}` for `any`) still opens on its template menu —
+  a pick-only control because the schema made it one, not a type question.
+- Emptying that editor leaves the field in template mode, where an empty
+  field lands. Switching to custom mode mounted a different editor in place
+  of the one being typed in, so the last backspace lost the cursor.
+
+Guarded by `__tests__/untypedFieldNeverAsksForAType.test.tsx` and the
+`AutoFormField` / `TemplateField` stories (`Empty`, `ViaFormEngine`,
+`AutoComponent`, `EmptyAnyOpensOnTemplates`).
+
+**Revised 2026-09-15 — template mode draws references as named chips.**
+Template mode's typable editor (a string or untyped field that may also hold
+a custom value) is `RichTextFormField` with `valueFormat: 'text'`, not a
+textarea. The field still stores the plain string; each template reference in
+it (`TEMPLATE_TOKEN_SOURCE`) is drawn as a chip, so a chosen template reads as
+the name it was chosen by — in the Visual builder's operands and in every
+string field that takes templates. A textarea spelled it `$local:name`.
+
+- The chip's label is `templateChipLabel`: the catalogue's name for the
+  reference, else the reference as the DPQL Text view labels it
+  (`local: name`), so one reference reads alike in both views. Its hover and
+  colour are the collapsed row's (`ReadOnlyTemplateTag`).
+- The editor keeps its own document and rebuilds it only when the stored
+  string changes from outside: rebuilding from the echo of what was typed
+  would hand the editor a new tree per keystroke, which moves the cursor.
+- `string` holds one line: Enter adds none, and a pasted break is flattened.
+- A value that is exactly one braced reference (`$data:{…}`, machine-written)
+  keeps its pick-only chip selector, unchanged.
+- Only a value that STARTS with a reference enters template mode, so
+  `AutoFormField`'s text editor — `string`, `long-string` and untyped — is the
+  same chip editor whenever the field offers templates: text like
+  `Interface $local:id failed` reads with its references named too. `data`
+  and `binary` keep the textarea (encoded content), as does a field with no
+  templates to offer. Guarded by `__tests__/textFieldDrawsTemplateChips.test.tsx`
+  and the `TemplateField` stories *Template References In Text* (+ *On Phone*).
+
+This reverses the 2026-08-27 choice (`85ba6ec`) to keep plain tokens as raw
+text: that review rejected a *pick-only* chip because it could not be typed
+into, and this editor is both. Conversion lives in `helpers/templateText.ts`
+(`__tests__/templateText.test.ts`, `__tests__/richTextTextValue.test.tsx`).
+
+**Revised 2026-09-16 — a field offers a template list only when it has one,
+and template mode has a way out.** Two rules about the menu, both reported
+from the stories:
+
+- **No empty menu.** `TemplateField` hands its editor `undefined` — not an
+  empty list — when there is nothing to offer: the field's schema does not
+  allow templates (a `FormEngine` option without `supports_templates`), or the
+  type filter emptied the catalogue. An editor GIVEN a list draws the control
+  that opens it, so a list with no items opened a menu onto nothing. Guarded
+  by `__tests__/noEmptyTemplateMenu.test.tsx`.
+- **A way back to a custom value.** Template mode's chip editor draws no `×`
+  of its own, so a field that entered template mode from the ⋮ ("Use Template")
+  offered nothing to leave it with. The ⋮ now carries "Use Custom Value"
+  whenever the field is in template mode and accepts custom values; leaving
+  clears the value only when it holds a reference (which would put the field
+  straight back into template mode) and keeps anything the author typed.
+- **A lone group in the ⋮ opens itself.** Two of the menu's groups are
+  collapsed sections (`MenuActionsSection`, `CustomMenuItems`), so a menu
+  holding nothing but one of them asked for a click to reach the only thing on
+  offer — on an untyped field, "Set Custom Value" hiding the data types behind
+  it. It now starts expanded when it is the menu's only group; it keeps its
+  label, so the rows still say what they are, and can still be collapsed. Where
+  the menu holds more than one group the sections stay shut: there the click is
+  the choice. The same rule the template list follows for a lone category, and
+  the row menu already publishes these rows flat. Guarded by
+  `__tests__/loneMenuSectionOpensItself.test.tsx` (the section's own contract)
+  and the story *Auto · Via Form Engine Menu Opens Its Only Group* (the rule, in
+  a real browser — the menu lives in a popover jsdom never opens).
+- **No document toolbar on a form field.** `RichTextFormField` asked Reqore
+  for undo and redo but not styling, and Reqore draws that bar as a panel as
+  wide as the tag list: an empty 600px box under the field holding two greyed
+  icons — they are disabled until there is history to walk. It is off for every
+  form use now; both still work from the keyboard, where every other text
+  field's do. The `richtext` type (the "Text" row of the data-type submenu) is
+  where this was reported. Guarded by
+  `__tests__/richTextFieldHasNoToolbar.test.tsx`.
+
+**Revised 2026-09-16 — a field says what it takes, and an untyped one is
+checked as the type it was given.** Two rules from the same report (picking
+"Binary" and being told nothing about the value):
+
+- **A binary field names its encodings.** Its placeholder reads
+  `Base64 — or 0x-prefixed hex, or a data:…;base64 URL`, which is what the
+  server actually decodes: Qorus `lib/misc.ql`
+  (`_priv_parse_ui_hash_value_intern`) reads base64 unless the value starts
+  `0x` or `data:<mime>;base64,`, symmetric with the `toBase64()` it encodes
+  with (`Classes/UserApi.qc`). Bare hex is the trap — it is not rejected, it is
+  read as base64 and corrupts. `validateField`'s `binary` branch already
+  accepted exactly these three spellings; the placeholder is that rule said
+  before the fact instead of after. A caller's own placeholder still wins, and
+  `data` is left alone — it is not base64. Guarded by
+  `__tests__/binaryFieldSaysWhatItTakes.test.tsx`.
+- **An untyped option is validated as the type its VALUE carries.** `auto` and
+  `any` declare no type: the author picks one and it is recorded beside the
+  value. `getOptionFieldMessages` read the schema first, so a binary value was
+  validated as `auto` — which auto-detects from the value and accepts whatever
+  is there — while the form's own check prefers the stored type
+  (`getOptionFieldStorageType`). The two disagreed, so the form said "a field
+  is not valid and requires attention" while the field it meant gave no reason.
+  The messages now use the same precedence; a schema that names a concrete type
+  still wins, because the schema is what the field accepts.
+  `isUntypedOptionType` (`helpers/optionUiTypes.ts`) is the shared spelling.
+  Guarded by `__tests__/untypedFieldReportsItsChosenType.test.ts`.
+
+**Revised 2026-09-16 (audit pass) — three rules the review pulled out of the
+above.**
+
+- **Whatever the field's own ⋮ offers, its ROW menu offers too.** A field
+  inside a compact row draws no menu of its own and publishes into the row's
+  (`rowMenuContext`). "Use Custom Value" was added to the drawn menu only, so
+  on the surface most options are actually edited from, template mode was still
+  a one-way door. The published list now carries it on the same condition, with
+  its own token in `publishedKey` — that key is what gates re-publication, so a
+  row without one never re-publishes when template mode flips. Guarded by
+  `__tests__/rowMenuOffersTheWayOutOfTemplateMode.test.tsx`.
+- **`RichTextFormField` takes `tags` and does not forward it.** The editor's
+  `tags` is its template catalogue; a form field's `tags` is the chips to show.
+  Two different things under one name, and `{...rest}` sat after the computed
+  value, so a field spreading its props erased the template list — patched at
+  two call sites with `omit(rest, 'tags')` and latent at two more. `rest` is now
+  spread FIRST, `tags` is declared as the field's chips and dropped, and the
+  patches are gone.
+- **One list of the untyped spellings.** `any`/`auto` was hand-rolled in eight
+  places. `UNTYPED_OPTION_TYPES` / `isUntypedOptionType` in
+  `helpers/optionUiTypes.ts` is now the one, and `engine/typeChoices`
+  re-exports it as `BuiltInAnyLikeTypes` for its existing callers. It lives in
+  helpers because a helper cannot import from a component.
+
+Note that a `FormEngine` option only offers templates when it declares
+`supports_templates` — a field with no ⋮ template entry is that option's
+schema speaking, not a defect.
+
 ### Out of scope for phase 1
 
 - The `type-depends-on` / `requestFieldData` cross-field type
@@ -178,7 +330,9 @@ qorus-ide coupling and copies over near-verbatim. The real work is the
    in `CatalogLeafForm`. `catalogLeafToFieldSchema` already emits the
    right shape; the adapter wires `value`/`onChange`. This is where the
    phase-1 `auto` field pays off — leaves typed `auto` now render a
-   real type-picker instead of a textarea.
+   real type-picker instead of a textarea. (Revised 2026-09-14: an empty
+   `auto` leaf is a typable editor again, with the type on its ⋮ menu — see
+   phase 1.)
 3. Port `CatalogNodeEditor` (group/map/list dispatch + drawer editors),
    swapping qorus-ide's `QorusTable` for `ReqoreTable` (or a Reqraft
    table if one exists) and `Hint` for a Reqraft equivalent or nothing.

package/design/RENDER_EXPRESSION_TRANSPORT.md

@@ -78,13 +78,79 @@ errors: -32602 missing/invalid expression; -32803 render failure;
         -32601 on servers predating the method
 ```
 
-## Fallback behaviour (reqraft)
-
-`useRenderExpression` keeps the public signature `{ render, serverRendering }`:
-
-- Server-first; on LSP unreachable (bounded connection wait + cooldown) or
-  `-32601` (older server — remembered for the page lifetime), `render`
-  falls back to the client-side approximation (`renderExpressionToText`)
-  and `serverRendering` reports `false`.
-- Consumers (`builder/renderTemplate.tsx`, `ExpressionField.tsx`) are
-  untouched.
+## Waiting, not approximating (reqraft)
+
+`useRenderExpression()` returns `{ render, renderRich }`; both resolve the
+server's rendering or `null`:
+
+- A render **waits for the connection** — there is no timer racing it. The
+  connection has its own reconnect schedule and rejects once it gives up
+  (see below), so a render never outlives a dead socket.
+- `null` means the server cannot render: the connection gave up, or the server
+  predates the method (`-32601`, remembered for the page lifetime).
+- There is **no client-side stand-in**. A Qorus form does not work without
+  Qorus, so the approximation only ever appeared while the page's first
+  connection was still opening, marked "Approximate" on the first Explain of
+  every page. `renderExpressionToText` survives only as the synchronous summary
+  a collapsed row prints (`readFirst.ts`), and is never shown in place of a
+  server rendering.
+
+## How an expression is shown when it is not being edited
+
+Every read-only expression surface draws through **`DpqlRendering`**
+(`src/components/form/expressions/DpqlRendering.tsx`): the Explain panel, the
+Text view's Preview, the conversion a type-fit message suggests, and a collapsed
+expression row. It is a read-only `DpqlEditor` with diagnostics and hover off,
+so every one of them reads the same — monospace, coloured by the language
+server's semantic tokens (the canonical tokenizer; there is no client-side
+highlighter), template references as chips.
+
+- `ExpressionRendering` renders an AST: it waits in the form's one waiting shape
+  (`FormFieldsSkeleton`, `data-wait="expression-rendering"`), then shows the
+  server's rendering, or says "This expression could not be rendered." when the
+  server cannot.
+- The Preview appears only once the server's rendering exists and differs from
+  the typed text; it has no waiting state of its own.
+- A surface with a template catalogue passes it (`templates`): a reference the
+  catalogue knows reads as the name it was chosen by, with the entry's
+  description as its hover — the Discord assistant's Save Reply row reads
+  `trim(Choices[0].message.content)`. One it does not know keeps its path
+  (`data: dc_ai_reply.choices[0].message.content`). A collapsed row passes its
+  field's own templates, else the form's; the builder's Explain passes its own.
+- A reference's path takes every dot except one that starts a method call, so
+  the server's `$local:str.endsWith($local:p, true)` chips `$local:str`, not
+  `str.endsWith`.
+- A new read-only expression surface uses `DpqlRendering`, never its own
+  `<code>`, rich-text field or chip splitter.
+
+## Lifetime of the shared render socket
+
+The module-level render client never releases its connection: the first
+expression rendered on a page holds the endpoint's `LspSharedConnection` for
+the rest of the page. Two consequences are designed for:
+
+- **A socket that gives up is forgotten.** When the socket exhausts its
+  reconnect attempts, the connection fails any handshake still waiting and
+  drops the socket; the next `connect()` from any client on that endpoint
+  dials afresh. Without this, a first handshake that never completed held
+  every later editor on "Connecting to language server…" for the life of the
+  page, and one that had completed kept reporting the dead socket as ready.
+- **Storybook resets it per story.** Stories run in one page, so the
+  connection an earlier story opened would otherwise serve a later story —
+  including one that starts its own mock LSP, which is then never dialled.
+  `.storybook/preview.tsx` runs `_resetRenderExpressionTransportForTests()`
+  and `_resetSharedLspConnectionsForTests()` in a project-level `beforeEach`.
+- **Every story has a language server.** Because renderings wait rather than
+  approximate, the same `beforeEach` starts the mock DPQL server
+  (`dpqlMockLsp.ts`) for every story that is not `live`. Its answers follow the
+  real server's (`dpqlMockLanguage.ts`, pinned in
+  `__tests__/dpqlMockLanguage.test.ts`); a story that needs other answers starts
+  its own server on the URL, which replaces the default.
+- **The rendering is spelled by the catalogue.** The server renders a function
+  by name (`concat("a", $local:x)`), an expression with a `render_template` by
+  that template (`$arg[0].startsWith($arg[1], $arg[2])`, a missing argument
+  read as its `default_value`), and any other operator as its arguments joined
+  by its symbol. A two-argument word comparison serializes infix
+  (`"test" startsWith "t"`) and otherwise as a call by symbol. The story server
+  reads `mockExpressions` by the same rules, which is why that fixture carries
+  the live `render_template`s and groups.

package/design/SMART_EDITOR_UX.md

@@ -253,6 +253,27 @@ Concrete plumbing:
   `SELECT * FROM users WHERE name = "Alice"` is replaced with a
   valid DPQL expression (e.g. `1 == 1` or `2026 - 2024 == 2`).
 
+**The server holds the text the editor shows** *(revised 2026-09-14)*. The
+server colours and diagnoses its own copy of the document, so two things must
+stay true, and neither did for a value set from outside rather than typed:
+
+- **Every value reaches the server.** `SmartEditor` passes each `value` to
+  `session.didChange`, which sends only text the server does not already hold,
+  and holds text set before the document opened for `didOpen`. Before, only
+  typing was sent and `didOpen` carried the text at mount, so a Preview that
+  follows the author was coloured for text it no longer showed.
+- **Offsets use the tree the editor holds.** Slate normalises a value set from
+  outside — a leading chip gains an empty text leaf before it — without
+  changing its text, so `value` does not change. `SmartEditor` adopts every
+  tree the editor reports as the one token, diagnostic and signature offsets
+  are computed against. Before, it kept the pre-normalisation tree and every
+  offset pointed one leaf off: a Text view seeded with
+  `"$local:name" == "John"` was never highlighted.
+
+Guarded by `__tests__/smartEditor/sessionKeepsServerText.test.ts` and the
+`ExpressionField` story *Via Form Engine Text Mode*, which asserts the seeded
+text is coloured.
+
 **Surface area.**
 `src/utils/lspClient.ts` (legend capture, ~10 lines),
 `src/utils/lspClient.types.ts` (new `ILspSemanticTokensLegend` /

package/dist/components/dpqlEditor/DpqlEditor.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"DpqlEditor.d.ts","sourceRoot":"","sources":["../../../src/components/dpqlEditor/DpqlEditor.tsx"],"names":[],"mappings":";AA2BA,OAAO,EAAE,gBAAgB,EAAE,cAAc,EAAE,MAAM,SAAS,CAAC;AAG3D,YAAY,EAAE,gBAAgB,EAAE,cAAc,EAAE,CAAC;AASjD,eAAO,MAAM,UAAU,6GA4KtB,CAAC"}
+{"version":3,"file":"DpqlEditor.d.ts","sourceRoot":"","sources":["../../../src/components/dpqlEditor/DpqlEditor.tsx"],"names":[],"mappings":";AA2BA,OAAO,EAAE,gBAAgB,EAAE,cAAc,EAAE,MAAM,SAAS,CAAC;AAG3D,YAAY,EAAE,gBAAgB,EAAE,cAAc,EAAE,CAAC;AASjD,eAAO,MAAM,UAAU,6GA+KtB,CAAC"}

package/dist/components/dpqlEditor/DpqlEditor.js

@@ -75,7 +75,7 @@ var useDpqlSession_1 = require("./useDpqlSession");
 // of completion is handled by typing inside an already-open token.
 var DPQL_TRIGGERS = new Set(['@', '$', '.', ':']);
 exports.DpqlEditor = (0, react_1.forwardRef)(function (_a, ref) {
-    var value = _a.value, onChange = _a.onChange, provider = _a.provider, recordType = _a.recordType, options = _a.options, actionCode = _a.actionCode, height = _a.height, _b = _a.readOnly, readOnly = _b === void 0 ? false : _b, _c = _a.showDiagnostics, showDiagnostics = _c === void 0 ? true : _c, _d = _a.enableHover, enableHover = _d === void 0 ? true : _d, _e = _a.templateTagsUseIntent, templateTagsUseIntent = _e === void 0 ? false : _e, onBlur = _a.onBlur, _f = _a.useServerParse, useServerParse = _f === void 0 ? false : _f, _g = _a.alertPayloadContext, alertPayloadContext = _g === void 0 ? false : _g, fsmContext = _a.fsmContext;
+    var value = _a.value, onChange = _a.onChange, provider = _a.provider, recordType = _a.recordType, options = _a.options, actionCode = _a.actionCode, height = _a.height, _b = _a.readOnly, readOnly = _b === void 0 ? false : _b, _c = _a.showDiagnostics, showDiagnostics = _c === void 0 ? true : _c, _d = _a.enableHover, enableHover = _d === void 0 ? true : _d, _e = _a.templateTagsUseIntent, templateTagsUseIntent = _e === void 0 ? false : _e, loadingIndicator = _a.loadingIndicator, templates = _a.templates, onBlur = _a.onBlur, _f = _a.useServerParse, useServerParse = _f === void 0 ? false : _f, _g = _a.alertPayloadContext, alertPayloadContext = _g === void 0 ? false : _g, fsmContext = _a.fsmContext;
     var dpql = (0, useDpqlSession_1.useDpqlSession)({
         provider: provider,
         recordType: recordType,
@@ -180,7 +180,7 @@ exports.DpqlEditor = (0, react_1.forwardRef)(function (_a, ref) {
                 return dpqlHelpers_1.dpqlSlateConverter.toSlateNodes(text);
             } });
     }, [useServerParse, serverParsedFor, serverParsedNodes]);
-    var tagRenderer = (0, react_1.useMemo)(function () { return (0, dpqlTags_1.makeDpqlTagRenderer)(dpql.fieldMeta, { templateTagsUseIntent: templateTagsUseIntent }); }, [dpql.fieldMeta, templateTagsUseIntent]);
+    var tagRenderer = (0, react_1.useMemo)(function () { return (0, dpqlTags_1.makeDpqlTagRenderer)(dpql.fieldMeta, { templateTagsUseIntent: templateTagsUseIntent, templates: templates }); }, [dpql.fieldMeta, templateTagsUseIntent, templates]);
     (0, react_1.useImperativeHandle)(ref, function () { return ({
         format: function () { return __awaiter(void 0, void 0, void 0, function () {
             var formatted;
@@ -200,7 +200,7 @@ exports.DpqlEditor = (0, react_1.forwardRef)(function (_a, ref) {
         parse: dpql.parse,
         serialize: dpql.serialize,
     }); }, [dpql, value, onChange]);
-    return ((0, jsx_runtime_1.jsx)(SmartEditor_1.SmartEditor, { session: dpql.session, value: value, onChange: onChange, tagRenderer: tagRenderer, triggerCharacters: DPQL_TRIGGERS, converter: converter, completionInserter: dpqlInserter_1.dpqlCompletionInserter, height: height, readOnly: readOnly, showDiagnostics: showDiagnostics, enableHover: enableHover, onBlur: onBlur, isLoading: isParsing }));
+    return ((0, jsx_runtime_1.jsx)(SmartEditor_1.SmartEditor, { session: dpql.session, value: value, onChange: onChange, tagRenderer: tagRenderer, triggerCharacters: DPQL_TRIGGERS, converter: converter, completionInserter: dpqlInserter_1.dpqlCompletionInserter, height: height, readOnly: readOnly, showDiagnostics: showDiagnostics, enableHover: enableHover, onBlur: onBlur, isLoading: isParsing, loadingIndicator: loadingIndicator }));
 });
 exports.DpqlEditor.displayName = 'DpqlEditor';
 //# sourceMappingURL=DpqlEditor.js.map