@qoretechnologies/reqraft 0.8.12-canary.20260609082134.0 → 0.10.0

package/.tasks/EXPRESSION_BUILDER_REPORT.md

@@ -0,0 +1,149 @@
+# TASK — Expression Builder verbatim re-port
+
+**Status:** Phases A + B + C DONE & verified (uncommitted), including the
+follow-ups (shell-chrome polish, the `validations` `'expression'` case — both
+ticked in Phase C below; the `'expression'` validator gained jest coverage
+2026-06-11). The ported, IDE-matching builder is live in `ExpressionField` /
+FormEngine. Phase D (server render-expression) + E (IDE consumes reqraft) are
+out of scope.
+Design:
+`design/EXPRESSION_BUILDER_REPORT_STRATEGY.md`. Supersedes the from-scratch
+visual builder in `.tasks/FORM_ENGINE_EXPRESSIONS.md` (plumbing retained).
+
+Method (Filip): copy IDE components **and their tests** verbatim → get them
+1:1 + green → then improve. Compare to qorus-ide Storybook **:6007** between
+phases. **No commits until user verifies.**
+
+## Surface area
+
+| Area | reqraft path | IDE source |
+|---|---|---|
+| Type-compat helpers | `src/helpers/expressions.ts` (NEW) | `qorus-ide/src/helpers/expressions.ts` |
+| `areQorusTypesCompatible` | `src/helpers/…` | `qorus-ide/src/helpers/functions.tsx` |
+| `validateField` export | `src/helpers/validations.ts` | (export the existing `_validateField`) |
+| Select flat/intent | `src/components/form/fields/select/Select.tsx` | `qorus-ide` Field/select |
+| TemplateField expr path | `src/components/form/fields/template/TemplateField.tsx` | `qorus-ide` Field/template |
+| ExpressionBuilder | `src/components/form/expressions/builder/**` (NEW) | `qorus-ide/src/components/ExpressionBuilder/**` |
+| Builder tests | `…/builder/*.stories.tsx` | `qorus-ide/src/stories/Components/ExpressionBuilder.stories.tsx` |
+
+## Phase A — Field-stack parity (leaf) — DONE & verified (2026-06-10)
+
+- [x] `src/helpers/expressions.ts` — verbatim port of getArgumentType,
+      argumentMatchesType (from IDE expressions.ts) + areQorusTypesCompatible
+      (from IDE functions.tsx); imports adapted (getQorusTypes/defaultQorusTypes
+      from reqraft `useQorusTypes`). Typechecks.
+- [x] `validateField` — already exported from validations.ts (no change needed)
+- [x] `SelectFormField` — `flat` + `intent` now forwarded (additive; defaults
+      preserve current look). This is the fix Filip named: "make it flat and
+      small and it looks the same." The chip can now go borderless.
+- [x] **GATE:** jest 267/267, full storybook **188/189** (pre-existing flaky
+      only). No regression from the shared Select change.
+- Note: `TemplateField`'s `allowFunctions`/`isFunction` mechanism moved to
+  Phase B — it's circularly dependent with the ExpressionBuilder (TemplateField
+  renders the builder; the builder renders TemplateField), so they land
+  together. Phase B builds in a NEW dir so the current build stays green.
+
+## Phase B — Verbatim ExpressionBuilder + tests — DONE & verified (2026-06-10)
+
+- [x] 9 IDE files copied verbatim into `src/components/form/expressions/builder/`
+      (index.tsx ~1260 LOC + item/argumentWrapper/argumentLabel/argumentDetail/
+      argumentInfo + the two confirm modals + renderTemplate). JSX/markup/props
+      preserved; only imports/hooks/types/seams changed.
+- [x] Seams: AiAssistanceAction **removed**; `renderTemplate` rewritten to use
+      reqraft `useRenderExpression` (client-side, no Creator-WS);
+      `useRegisterHintView` **stubbed**.
+- [x] `TemplateField` — additive `allowFunctions`/`isFunction`; when set, renders
+      the ported builder. Absent → unchanged (field migration safe).
+- [x] Adaptations: `Select` `defaultItems`→`items`; `Select.onChange` single-arg;
+      catalogue passed as `override` (replaces fetch) **and threaded through the
+      group recursion** so offline/CI is deterministic (badge = mock count, not
+      the live 121). `auto`/`Select`/`TemplateField` import aliases.
+- [x] Ported `ExpressionBuilder.stories.tsx` (the IDE play tests) — offline via
+      `mockExpressions`. **10/10 pass.**
+- [x] **GATE:** renders **1:1 vs :6007** (verified by screenshot — chips, underline
+      fields, operator words, group nesting, type-pickers, collapse/Explain all
+      match). eslint clean, jest 267/267, full suite **198/199** (pre-existing
+      flaky only). Old from-scratch builder left intact; build green.
+- Open follow-ups (Phase C / refinements): wire into `ExpressionField`;
+      `validations` has no `'expression'` type case yet (per-expression validity
+      tag always "valid"); `useRenderExpression` is the client-side approximation.
+
+## Phase C — Re-wrap reqraft features — DONE & verified (2026-06-10)
+
+- [x] `ExpressionField` Visual mode now renders the **ported** builder
+      (`./builder`); Text = `DpqlEditor`; Explain/Preview from the shell.
+      Adapted: `onChange (value, remove?)`, `returnType` cast, and
+      **`localTemplates={{ items: [] }}`** (the builder's `filterTemplatesFunc`
+      does `templates.items.filter(...)` — undefined templates crashed it;
+      caught via the error boundary during verify and fixed).
+- [x] From-scratch `ExpressionBuilder.tsx` + its stories **retired** (deleted);
+      no remaining references.
+- [x] **GATE:** FormEngine field → ExpressionField → ported builder verified
+      (ViaFormEngine + the toggle path). jest 267/267, full suite **193/193**
+      (the flaky `OnValidityChange` even passed). HEAD unchanged.
+- [x] **Shell cleanup (done 2026-06-10):** Visual mode now renders ONLY the
+      ported builder (it has its own Explain). The shell's Explain + Preview +
+      Explanation are scoped to **Text mode** (where the DpqlEditor has none).
+      ExpressionField play tests updated to assert the builder rendered.
+- [x] **Danger-border fix (done 2026-06-10):** three parts —
+      (1) ported the IDE's `validations` **`'expression'` case** (reads the
+      catalogue from `field.expressions`); (2) `FormEngine.isOptionValid` now
+      passes `isFunction` from the field value's `is_expression` (the per-field
+      red intent used the base-type check); (3) `ExpressionField` defaults
+      `returnType` to `'auto'` so a valid expression card renders `muted` not
+      `danger`. Verified: the FormEngine expression field is no longer red.
+      eslint clean, jest 267/267, full suite 192/193 (pre-existing flaky only).
+- [x] **Oversized chips/pickers fix (2026-06-10):** root cause — the IDE's
+      `Select` spreads `{...rest}` into its rendered ReqoreButton/Dropdown/Tag,
+      so the builder's verbatim `size='tiny'` reaches ReQore; reqraft's
+      `SelectFormField` accepted `size` in its interface but **never forwarded
+      it** (same class of bug as `flat`/`intent`). Forwarded `size` to all four
+      render branches (bound as `componentSize` — plain `size` would shadow
+      lodash's `size(items)` badge count). Chips now tiny, type pickers
+      compact, operand rows no longer wrap — density matches the IDE.
+      jest 267/267, full suite 192/193 (pre-existing flaky only).
+- Known seam: reqraft's `SelectFormField` still doesn't rest-spread like the
+      IDE's — `flat`/`intent`/`size` are forwarded explicitly; any other
+      pass-through prop the verbatim IDE markup relies on needs the same
+      treatment if a gap shows up.
+- [x] **Type-picker parity (2026-06-10):** `AutoFormField` now matches the
+      IDE's AutoField — unresolved type shows **`auto`** as the selected picker
+      label (not a "Select a type" placeholder; `auto` is not a pickable item
+      so the badge count stays identical to the IDE's 9/14), and the hint is
+      the IDE's orange minimal `ReqoreTag` **"Please select data type"**
+      (was a grey info tag). The 3 AutoFormField story assertions updated.
+- [x] **AI-assist slot (2026-06-10):** the IDE's purple hover button on each
+      expression card is `AiAssistanceAction` → `AiButton` (sends the selected
+      expression's context to the IDE's AI chat). It was one of the three
+      deliberate port seams — `AiButton` depends on IDE-only infrastructure
+      (AI context, SystemStore, QonsoleStore, Creator-WS) that reqraft doesn't
+      have. Added **`extraActions?: IReqorePanelAction[]`** to the ported
+      builder (threaded through the group recursion, prepended exactly where
+      the IDE renders the AI action) so the IDE injects its AiButton at
+      Phase E. jest 267/267, full suite 192/193 (pre-existing flaky only).
+- [x] **`extraActions` upgraded to a factory (2026-06-10):** a static array
+      could not reproduce the IDE's button — the IDE builds
+      `AiAssistanceAction({ context: selectedExpression })` *inline per card*.
+      The seam now also accepts
+      `(ctx: { selectedExpression?, value? }) => IReqorePanelAction[]`,
+      resolved inside each `Expression` with that card's selected schema —
+      so Phase E restores the AI button 1:1, per-card context included.
+      Proven by the new `WithInjectedExtraActions` story (hover → injected
+      action present; 11/11 ported play tests). Full suite **194/194**
+      (even the flaky `OnValidityChange` passed), jest 267/267.
+- Ops note: the Storybook dev server reliably wedges on HMR after edits to
+      these files (symptom: play tests fail in ~2ms with
+      `ReferenceError: __test is not defined`, preview stuck on spinner).
+      A clean restart fixes it every time — restart before trusting failures.
+
+## Phase D — Server (tracked, not in reqraft)
+
+- [x] `render-expression` over LSP/REST; swap `useRenderExpression` —
+      picked up 2026-06-10 in `.tasks/RENDER_EXPRESSION_LSP.md` (branch
+      `feature/render-expression-lsp`): new LSP `dpql/renderExpression`
+      (server code written, deploy pending) + server-first hook with
+      client-side fallback. Design: `design/RENDER_EXPRESSION_TRANSPORT.md`.
+
+## Phase E — Dedupe (not now)
+
+- [ ] IDE consumes reqraft's builder

package/.tasks/FIELD_STACK_REPORT.md

@@ -0,0 +1,234 @@
+# TASK — TemplateField + auto verbatim re-port
+
+**Status:** done pending user verify (uncommitted), branch
+`feature/dpql-editor` — stacked on the uncommitted
+EXPRESSION_BUILDER_REPORT batch (user chose to proceed without
+committing the baseline first, 2026-06-10). Full storybook suite
+**232/232**, jest 268/268, `yarn precheck` clean. Click-through in
+`VERIFY.local.md`.
+
+Continuation of the field-stack re-faithful-ization started in
+[EXPRESSION_BUILDER_REPORT](./EXPRESSION_BUILDER_REPORT.md) Phase A.
+Design rationale: `design/EXPRESSION_BUILDER_REPORT_STRATEGY.md`
+(Filip's method: copy IDE components **and their tests** verbatim →
+1:1 + green → then improve). reqraft's `TemplateField` and
+`AutoFormField` are from-scratch reimplementations — the operand layer
+under the ported ExpressionBuilder — and this task replaces them with
+verbatim ports of `qorus-ide/src/components/Field/template.tsx` (774
+LOC) and `Field/auto.tsx` (1017 LOC).
+
+**No commits, no pushes. STOP for user browser verification at the end.**
+
+## Surface area
+
+| Area | reqraft path | IDE source |
+|---|---|---|
+| TemplateField | `src/components/form/fields/template/TemplateField.tsx` (REPLACED) | `Field/template.tsx` |
+| Auto field | `src/components/form/fields/auto/AutoFormField.tsx` (REPLACED) | `Field/auto.tsx` |
+| AllowedValues | `src/components/form/fields/allowed-values/AllowedValues.tsx` (NEW) | `components/AllowedValues/index.tsx` |
+| `getValueOrDefaultValue` | `src/helpers/validations.ts` (ADD export) | `helpers/validations.ts:1353` |
+| FormEngine expression model | `src/components/form/engine/FormEngine.tsx` | `Field/systemOptions.tsx` (4-arg `handleValueChange`) |
+| Field dispatch | `src/components/form/fields/Field.tsx` (auto/any call site) | — |
+| Template stories | `…/template/TemplateField.stories.tsx` (NEW) + fixtures | `stories/Fields/Template.stories.tsx` |
+| Auto stories | `…/auto/AutoFormField.stories.tsx` (extended) | `stories/Fields/Auto.stories.tsx` |
+
+## Seam map (replace, don't copy — same classes as the builder port)
+
+**template.tsx:**
+- `useGetAppActionData` (app-action badge on the template dropdown) —
+  IDE-only app catalogue; **dropped** (badge omitted).
+- `SaveValueButton` (`canSaveValue` menu item) — depends on IDE
+  saved-values/modal infra; **dropped**. `menuItems` seam remains for
+  consumers needing extra menu entries.
+- `useExpressions(allow, {url, expressions})` → reqraft
+  `useExpressions({allow, expressionsUrl, extraExpressions})`;
+  `.value/.valueForSelect` dep keys → `.expressions`.
+- `ExpressionBuilder` import → `../../expressions/builder` (the ported
+  builder). `effectiveIsFunction` renders the **bare builder** like the
+  IDE — the Phase-1 `supports_expressions`/`is_expression`-prop toggle
+  mechanism (which rendered `ExpressionField`) is **removed**; the IDE
+  model wins (accepted drawback #2 in the strategy doc).
+- `TemplatesListProps` defined locally (IDE keeps it in `Field/richText`).
+- ComponentMap leafs (`LongStringField`, `Number`, `BooleanField`,
+  `DateField`, `RichTextField`, `FileField`) — local 2-arg
+  (`onChange(name, value)`) wrappers around reqraft's leaf fields.
+- `isValueTemplate`/`getTemplateKey`/`getTemplateValue` — re-exported
+  from `helpers/templates` (IDE defines them inline).
+
+**auto.tsx:**
+- **Dropped IDE-only types** (fall through to the verbatim `default:`
+  "Unknown type!" tag, recoverable via `componentOverrides`):
+  `processor-mappings`, `tool-catalog`, `test-cases`, `active-windows`,
+  `collection-documents`, `data-provider` (ConnectorField), the
+  InterfaceSelector family (`mapper`…`value-map`), `connection` (+
+  `renderConnectionManagement` removed), `option_hash`, `code-editor`
+  (SmartEditor needs an LSP `session` — too heavy for this seam).
+- **`componentOverrides?: Record<string, React.FC>`** — one additive
+  seam prop (same class as the builder's `extraActions`): checked
+  before the switch so the IDE can re-inject its own editors at
+  Phase E (dedupe).
+- `fetchData('/dataprovider/arg_schemas/…')` → reqraft `query()`
+  (same `{data, ok, error}` shape).
+- `FieldAllowedValues`/`CheckGroup` → NEW port (see below).
+- `Options` (hash + arg_schema) → `FormEngine` (it IS the ported Options).
+- Leaf fields → reqraft equivalents with inline `onChange` adaptation
+  per call site (reqraft leafs are single-arg `onChange(value)`; the
+  builder port set this precedent): LongString, RichText, Boolean
+  (`checked` prop), Date, Number, ByteSize, Url, Select, MultiSelect,
+  Color, File, Object(hash/list), RadioGroup (enum), ArrayAutoField
+  (renderItem = `TemplateField component={auto}` — mirrors IDE
+  arrayAuto internals), SchemaDefinitionEditor, DpqlEditor.
+- `Loader` → minimal `ReqoreSpinner` (same stand-in as schema-definition's).
+- `IField` (FieldWrapper) → folded into `IAutoFieldProps` locally
+  (`requestFieldData` kept — drives `type-depends-on`).
+- `console.log` debug calls removed; `useWhyDidYouUpdate` kept
+  (reqraft has the hook).
+- `allowSaving`/`showSavedValues` props **kept in the interface** but
+  inert (saved-values infra is IDE-only) — documented as a seam.
+
+**AllowedValues port seams:** `useSavedValues` → empty list (prop kept,
+inert); `useSubscriptionEvents` (CONNECTION_DELETED) → dropped;
+`ConnectionManagement` per-item actions → dropped (actions = []);
+`Select` → `SelectFormField` (`items`, single-arg onChange adapter).
+
+## Consumer reconciliation (the swap's blast radius)
+
+- `Field.tsx` auto/any case → new API (`defaultType`, 2-arg onChange
+  adapter).
+- `FormEngine` → IDE Options model: `allowFunctions` from
+  `supports_expressions`, `isFunction={other.is_expression}`,
+  `isDefaultFunction` from `default_view`, 4-arg `handleValueChange`
+  where the **4th arg** (not a value wrapper) drives `is_expression`.
+- `builder/index.tsx` — call sites already verbatim-IDE-shaped
+  (`component={auto}`, 4-arg onChange); stale `as any` casts removed
+  where signatures now match.
+- `ExpressionField` stays exported with its stories (the NET-NEW DPQL
+  text mode shell) but is no longer rendered by TemplateField.
+  **Follow-up decision for verification:** whether FormEngine-level
+  expression fields should regain the Text/DPQL mode via ExpressionField
+  chrome, or stay IDE-verbatim (bare builder).
+
+## Phases
+
+### Phase 1 — verbatim port + ported tests — DONE (2026-06-10)
+
+- [x] `getValueOrDefaultValue` exported from validations
+- [x] AllowedValues ported
+- [x] auto.tsx ported verbatim (seams above)
+- [x] template.tsx ported verbatim (seams above)
+- [x] Field.tsx + FormEngine + builder reconciled
+- [x] IDE Template stories ported (offline: `templates.json` +
+      `multiLevelTemplates.json` fixtures copied; `_tests*` helpers
+      inlined per the builder-stories pattern)
+- [x] IDE Auto stories ported (offline; saved-values/connection
+      stories adapted or dropped per seams)
+- [x] **Drift fixes surfaced by the ported tests** (the whole point of
+      porting the IDE tests):
+      - `filterTemplatesByType` — reqraft used naive badge equality;
+        replaced with the IDE verbatim (`areQorusTypesCompatible`
+        compat + top-level groups always kept). String fields now see
+        ALL templates (string accepts `any`), int sees int, etc.
+      - `NumberFormField` — gained the IDE's templates-dropdown wrap
+        (focus-opened template list); plain input unchanged without
+        templates.
+      - IDE `arrayAuto` ported verbatim as `ArrayAuto`
+        (`fields/array/ArrayAuto.tsx`) — panel-per-item with
+        `TemplateField component={auto}` per item + "Add new item"
+        button; the ported auto uses it. reqraft's compact
+        `ArrayAutoField` (renderItem seam) stays for `Field.tsx`.
+- [x] GATE: ported play tests **37/37**; 1:1 screenshots vs :6007
+      (TemplateWithFunctionValue, nested-function, type-picker 14,
+      ListWithElementType, allowed values — differences are
+      mock-vs-live catalogue content only). jest 268/268, eslint
+      clean, `build:test:prod` clean.
+
+### Phase 2 — regression reconciliation — DONE (2026-06-10)
+
+- [x] jest suite green (268/268)
+- [x] full storybook suite green — **230/230, including the usually
+      flaky `OnValidityChange`** (the 14/9 type-picker badges and
+      `auto` + "Please select data type" assertions kept)
+- [x] Reconciliation fixes:
+      - auto gained a `long-string` alias in the string case (reqraft
+        FormField vocabulary; the only reqraft-only `ui_type` in real
+        use — evidence: story grep).
+      - `OnValidityChange` play now awaits the textarea (the IDE-style
+        auto resolves its type in a mount effect, one tick after
+        validity is first reported) — also de-flakes the known flake.
+      - Ported builder: `?.` hardening on `filterTemplatesFunc`'s
+        `items.filter` — with no templates at all (FormEngine,
+        templates off) the IDE original crashes on `{}.items.filter`
+        (latent IDE bug; it always has fetched templates).
+      - `ExpressionField` stays exported with its stories (DPQL text
+        mode); FormEngine now routes expressions through the IDE
+        model (bare builder via TemplateField `isFunction`) — its
+        `ViaFormEngine` story passes through the new path.
+
+### Phase 3 — small-fields audit — DONE (2026-06-10)
+
+- [x] `byte-size` — already structurally verbatim (Number + unit
+      Select); aligned the unit list to the IDE's exact `KiB`/`MiB`
+      (GiB/TiB were a FORM_FIELD_EXTRAS addition — trimmed for 1:1,
+      can be re-added post-parity) and unfixed the unit select so it
+      stretches like the IDE's. The IDE's field-level `useMount`
+      default-value push stays out (defaults are FormEngine-level in
+      reqraft). IDE ByteSize stories are smoke-only — no play tests
+      to port.
+- [x] `url` — **replaced verbatim**: protocol Select + `://` + address
+      (with the IDE's `getProtocol`/`getAddress` splitting). The
+      FORM_FIELD_EXTRAS single-input version is superseded per this
+      batch's mandate. Seams: the `qorus_instance`-gated remote
+      protocols fetch dropped (`protocols` prop kept, IDE default
+      list); the `://` label rendered as the muted paragraph the IDE's
+      StringField produces. `Field.stories` Url play updated to the
+      new structure. No IDE play tests exist for url.
+- [x] `multi-select` — portable IDE behaviours brought over: selected
+      values missing from the allowed list still render as chips, `*`
+      wildcard collapses the selection + disables other items,
+      wrap/tooltip item props, `showNoItemsMessage={false}`,
+      `selectorProps={{useTargetWidth:false}}`, `selectedItemSize`.
+      The IDE's editor dialog / FieldEnhancer `reference` machinery is
+      app-coupled — not ported. IDE MultiSelect stories are smoke-only.
+- [x] `schema-definition` — drift-checked; the Options→FormEngine
+      adaptation stays. Ported the two applicable IDE play tests:
+      **Sandboxed** (untrusted caller → no Advanced tab) and
+      **WithValidationBanner** (name mismatch banner + "Sync name").
+- [x] GATE: full storybook suite **232/232**, jest 268/268, eslint +
+      `build:test:prod` clean.
+
+### Follow-up — top-level ExpressionField re-wrap — DONE (2026-06-11)
+
+Resolves the "DPQL text mode unrouted" open decision (user picked
+re-wrap). One additive seam, default = IDE-verbatim:
+
+- [x] `TemplateField` gained **`allowTextExpressions`** — when set, the
+      expression branch renders the `ExpressionField` shell (Visual =
+      the same ported builder, Text = the DPQL editor) instead of the
+      IDE's bare builder. `handleExpressionChange` serves both (the
+      shell now emits the identical `(value, remove)` contract).
+- [x] `FormEngine` sets it on its TemplateField (opt out per form via
+      `templateFieldProps={{ allowTextExpressions: false }}`). Nested
+      operands (builder arguments, ArrayAuto items) never receive the
+      prop — they stay IDE-verbatim bare builder, so the ported
+      Template/Auto stories remain 1:1 vs :6007.
+- [x] `ExpressionField` upgrades: `onChange` threads the builder's
+      `remove` (exit-expression-mode now works from the shell);
+      `localTemplates` + `serverHandled` pass through to the builder.
+- [x] **Live stories** (mock LSP): `ViaFormEngineTextMode` (serialize
+      direction — switching to Text seeds the editor from the stored
+      AST) and `ViaFormEngineTextTyping` (parse direction — typing
+      DPQL lands in the form value with `is_expression` via the 4-arg
+      flow). `ViaFormEngine` extended to assert the Visual/Text toggle.
+      Play-test note: synthetic keystrokes only land in an EMPTY Slate
+      document — hence the typing story starts from an empty
+      expression; the seeded story asserts content, not keystrokes.
+- [x] GATE: full storybook suite **234/234 twice in a row**, jest
+      268/268, eslint + `build:test:prod` clean. Mock-fidelity note:
+      entering Text mode re-parses the serialized text (parse-on-edit
+      design); the mock's echo-parse makes the preview look degraded —
+      the real `dpql/parse` round-trips the AST losslessly.
+
+### STOP — user verifies in browser before commit
+
+`VERIFY.local.md` gets the story-by-story click-through; reqraft
+storybook on :6011 side-by-side with qorus-ide :6007.