@solidxai/core 0.1.13-beta.2 → 0.1.13-beta.4

package/docs/workflow/control-flow-node-addition-sop.md

@@ -0,0 +1,324 @@
+# Control Flow Node Addition SOP
+
+This SOP describes the places that must be reviewed when adding a new workflow
+control-flow node type. Control nodes are higher risk than task nodes because
+they own child execution semantics and are interpreted by the runtime, topology
+canvas, validation layer, expression autocomplete, execution logs, and metadata
+sync.
+
+Use this document when adding nodes such as `parallel`, `sequential`, `if`,
+`switch`, `forEach`, `loopUntil`, retry groups, fan-out/fan-in controls, human
+approval gates, or other nodes with child slots.
+
+## Core Rule
+
+A control node must define three contracts together:
+
+- Runtime contract: how children execute and what gets written to
+  `context.outputs`.
+- Authoring contract: which child slots exist and how they are represented in
+  YAML.
+- Reference contract: how downstream nodes can refer to outputs produced by the
+  control node and its children.
+
+Do not add a control node by only registering a provider class. If the node owns
+children, every section below needs a deliberate yes/no decision.
+
+## Backend Files To Review
+
+- `src/services/workflow/nodes/<node>.node.ts`
+  Add the `WorkflowNodeProvider` metadata and the `WorkflowNodeHandler`
+  implementation.
+
+- `src/services/workflow/workflow-runtime.service.ts`
+  Confirm `context.runNodes(...)` receives the correct context overrides:
+  `outputs`, `item`, `index`, `parentNodeId`, and `parentStepExecutionKey`.
+
+- `src/services/workflow/workflow-expression.service.ts`
+  Confirm expressions can address the data shape exposed by the new control
+  node. Most additions should not require changes here if they use plain object
+  paths.
+
+- `src/services/workflow/workflow-definition-validator.service.ts`
+  Confirm the YAML shape is accepted and invalid child-slot shapes are rejected.
+
+- `src/types/workflow-dsl.types.ts`
+  Update comments and types when introducing a new child collection key,
+  execution context concept, or output convention.
+
+- `src/services/workflow/workflow-node-registry.service.ts`
+  Confirm provider discovery/registration picks up the new node class.
+
+## Frontend Files To Review
+
+- `solid-core-ui/src/components/workflow/WorkflowFlowCanvas.tsx`
+  Confirm the topology visual editor can render the child slots, add-node
+  insertion affordances, branch labels, join lines, and nested layout.
+
+- `solid-core-ui/src/components/workflow/WorkflowFlowCanvas.css`
+  Add styling only if the new control mode needs a distinct visual treatment.
+
+- `solid-core-ui/src/components/workflow/WorkflowNodeSchemaEditor.tsx`
+  Confirm the node editor displays child slot summaries and runtime fields
+  correctly.
+
+- `solid-core-ui/src/routes/pages/admin/core/WorkflowDefinitionEditorPage.tsx`
+  Review validation, node update/remove traversal, first-node selection,
+  expression suggestion collection, and any YAML normalization logic.
+
+- `solid-core-ui/src/types/workflow-node.ts`
+  Update UI-facing metadata types when adding new slot layout modes, field
+  metadata, or output metadata.
+
+## Provider Metadata Checklist
+
+Every control node provider should define:
+
+```ts
+@WorkflowNodeProvider({
+  type: 'newControlType',
+  kind: 'control',
+  category: 'control-flow',
+  label: 'New Control Type',
+  description: '...',
+  authoring: {
+    defaultConfiguration: {},
+    configurationFields: [],
+    childSlots: [
+      {
+        key: 'tasks',
+        label: 'Tasks',
+        kind: 'sequence',
+        layout: 'sequential',
+        required: true,
+      },
+    ],
+    outputs: [],
+    supportsName: true,
+    supportsDescription: true,
+  },
+  runtime: {
+    emitsLogs: true,
+    emitsArtifacts: false,
+    deterministicOutputs: false,
+    executionMode: 'task',
+  },
+})
+```
+
+The important fields are:
+
+- `kind: 'control'`: tells UI and validators this node may own children.
+- `authoring.childSlots`: tells topology which child collections exist.
+- `childSlots[].key`: must match the YAML property used by the runtime.
+- `childSlots[].kind`: use `sequence` for arrays and `case-collection` for
+  object-of-arrays.
+- `childSlots[].layout`: use `sequential` or `parallel` unless a new layout mode
+  has been fully implemented in the canvas.
+- `authoring.outputs`: describes the control node's output object for
+  expression autocomplete and docs.
+
+## YAML Shape Contract
+
+Prefer existing child keys unless the new control mode truly needs a new shape:
+
+- `tasks`: generic ordered or parallel child sequence.
+- `then`: truthy branch.
+- `else`: falsey branch.
+- `defaults`: fallback branch.
+- `cases`: named object-of-arrays for switch-like controls.
+
+Example:
+
+```yaml
+- id: gate
+  kind: control
+  type: if
+  configuration:
+    condition: "{{ inputs.enabled }}"
+  then:
+    - id: success_log
+      kind: task
+      type: log.write
+      configuration:
+        message: Enabled
+  else:
+    - id: skip_log
+      kind: task
+      type: log.write
+      configuration:
+        message: Skipped
+```
+
+If a new child key is introduced, update all traversal helpers. At the time of
+writing these include validation, find/update/remove node helpers, topology
+layout, and expression suggestion collection.
+
+## Runtime Semantics
+
+Control handlers should execute children through `context.runNodes(...)`.
+
+Sequential example:
+
+```ts
+const outputs = {};
+await context.runNodes(context.node.tasks ?? [], { outputs });
+return { output: { outputs } };
+```
+
+Parallel example:
+
+```ts
+const inheritedOutputs = context.outputs ?? {};
+const outputs = { ...inheritedOutputs };
+await context.runNodes([task], { outputs });
+```
+
+Use inherited outputs when child nodes need to read values produced before the
+control node. Return only the branch-specific outputs when the control node
+should avoid promoting branch outputs globally.
+
+Be explicit about these decisions:
+
+- Are child outputs promoted into parent `context.outputs`?
+- Are child outputs grouped under the control node output?
+- Can sibling branches read each other's outputs?
+- Can children read outputs produced before the control node?
+- Does a failed child fail the whole workflow or become structured output?
+- Does `onError: continue` behave at the child level, control level, or both?
+
+## Output Reference Contract
+
+Downstream expression examples must be documented in the provider metadata or
+node docs.
+
+For a task node:
+
+```yaml
+{{ outputs.login.accessToken }}
+```
+
+For a parallel node that groups branch outputs:
+
+```yaml
+{{ outputs.parallel.tasks.0.outputs.get.data }}
+{{ outputs.parallel.tasks.1.outputs.list.status }}
+```
+
+For a loop-like node, choose and document one stable shape, for example:
+
+```yaml
+{{ outputs.loop.items.0.outputs.transform.data }}
+{{ outputs.loop.count }}
+```
+
+Avoid undocumented magic promotion such as making every branch child globally
+available as `outputs.childId` unless collision behavior is also defined.
+
+## Topology Canvas Checklist
+
+Review `WorkflowFlowCanvas.tsx` whenever child-slot semantics change.
+
+Confirm:
+
+- The node is detected as a control node through `kind` or `childSlots`.
+- Every child slot renders with the intended layout.
+- Add-node buttons insert into the correct slot and index.
+- Branch labels are readable and stable under nesting.
+- Edges enter the control node, connect to children, and rejoin correctly.
+- Empty slots still render an insertion target.
+- Nested controls do not overlap on desktop or narrow widths.
+- Delete/reorder operations preserve the YAML shape.
+
+If adding a new `childSlots[].layout` value beyond `sequential` or `parallel`,
+first update UI metadata types and canvas layout algorithms. Do not ship a new
+layout string that the canvas silently treats as sequential.
+
+## Expression Autocomplete Checklist
+
+Review `WorkflowDefinitionEditorPage.tsx` expression suggestion logic whenever
+runtime output semantics change.
+
+Confirm:
+
+- Inputs and variables remain available everywhere.
+- A node can see outputs from definitely earlier nodes.
+- A branch can see parent/prior outputs when runtime permits it.
+- Sibling branch outputs are hidden when branches run concurrently.
+- Downstream nodes see the grouped output shape of the control node.
+- The suggestions match actual runtime output paths.
+
+Expression autocomplete is advisory, but wrong suggestions are worse than
+missing suggestions because they produce YAML that validates yet fails at
+runtime.
+
+## Validation Checklist
+
+Validation should reject malformed control structures early:
+
+- `tasks`, `then`, `else`, and `defaults` must be arrays when present.
+- `cases` must be an object whose values are arrays.
+- Required child slots must exist and satisfy `minItems` when enforced.
+- Unsupported child keys should produce a clear message.
+- Node ids must stay unique across nested children.
+- Node `kind` must match registered provider metadata.
+- Required configuration fields must still be validated inside control nodes.
+
+## Execution History Checklist
+
+Confirm execution detail screens make sense after a run:
+
+- Parent control step has a useful output payload.
+- Child steps have `parentNodeId` and `parentStepExecutionKey`.
+- Logs emitted inside children appear under the correct step.
+- Timeline order is understandable for parallel/concurrent branches.
+- Failed child behavior is visible in step status and parent output.
+
+## Metadata And Samples
+
+When the new control mode is public, add or update sample YAML in the module
+metadata that owns workflow samples. Include:
+
+- Minimal happy-path sample.
+- Nested control sample if nesting is expected.
+- Example showing how a downstream node reads the control output.
+- Example showing error behavior if the control has custom failure semantics.
+
+If workflow definitions are stored through metadata sync, verify the generated
+YAML and metadata JSON preserve the new child structure without normalization
+loss.
+
+## Testing Checklist
+
+Run at least:
+
+- Backend build: `npm run build` in `solid-core-module`.
+- Frontend build: `npm run build` in `solid-core-ui`.
+- Consuming app build when testing through an app package copy.
+
+Manual workflow tests:
+
+- Create a workflow with the new control node from topology.
+- Add, edit, delete, and reorder child nodes.
+- Save, reload, and verify YAML is unchanged except expected formatting.
+- Execute happy path.
+- Execute failure path.
+- Inspect execution detail, logs, step output, and final output.
+- Use `Ctrl+Space` in a downstream expression field and verify suggestions
+  match real output paths.
+
+## Minimal Regression Review
+
+Before merging, compare behavior against existing controls:
+
+- `parallel`
+- `sequential`
+- `if`
+- `switch`
+- `forEach`
+
+The new node should not require special cases in shared UI/runtime code unless
+its semantics are genuinely different. Prefer expressing behavior through
+provider metadata first, then add narrow code changes only where metadata cannot
+describe the behavior.
+

package/docs/workflow/data-engineering-pipeline-example-notes.md

@@ -0,0 +1,330 @@
+# Data Engineering Pipeline Example Notes
+
+## Purpose
+
+This document captures a concrete workflow example and the accompanying UI notes derived from the Kestra reference screens. It is intended to serve as a working design reference for the SolidX workflow-definition list view, detail view, and editor experience.
+
+---
+
+## Example Workflow YAML
+
+```yaml
+id: data-engineering-pipeline
+namespace: tutorial
+labels:
+  name: Data Engineering Pipeline
+
+inputs:
+  - id: columns_to_keep
+    type: ARRAY
+    itemType: STRING
+    defaults:
+      - brand
+      - price
+
+tasks:
+  - id: extract
+    type: io.kestra.plugin.core.http.Download
+    description: Download the raw products JSON data.
+    uri: https://dummyjson.com/products
+
+  - id: transform
+    type: io.kestra.plugin.scripts.python.Script
+    description: Filter the product fields and write a cleaned JSON file.
+    containerImage: python:3.11-alpine
+    inputFiles:
+      data.json: "{{ outputs.extract.uri }}"
+    outputFiles:
+      - "*.json"
+    script: |
+      import json
+      import os
+
+      with open("data.json", "r") as file:
+          data = json.load(file)
+
+      filtered_data = [
+          {column: product.get(column, "N/A") for column in {{ inputs.columns_to_keep }}}
+          for product in data["products"]
+      ]
+
+      with open("products.json", "w") as file:
+          json.dump(filtered_data, file, indent=4)
+
+  - id: query
+    type: io.kestra.plugin.jdbc.duckdb.Queries
+    description: Aggregate the cleaned data with DuckDB and store the result.
+    inputFiles:
+      products.json: "{{ outputs.transform.outputFiles['products.json'] }}"
+    sql: |
+      INSTALL json;
+      LOAD json;
+      SELECT brand, round(avg(price), 2) as avg_price
+      FROM read_json_auto('{{ workingDir }}/products.json')
+      GROUP BY brand
+      ORDER BY avg_price DESC;
+    fetchType: STORE
+
+description: |
+  **Use case:** Compact data engineering pipeline that filters product data and aggregates it by brand.
+  **Highlights:**
+  - Download product JSON from a public API.
+  - Run a Python transform that keeps only the requested columns and emits a cleaned file.
+  - Query the cleaned data with DuckDB to compute average prices per brand and store the result for download in the UI.
+```
+
+---
+
+## What This Example Represents
+
+This workflow is a good V1 reference because it is:
+
+- Simple enough to understand quickly.
+- Sequential, so it maps well to the current implicit-edge model.
+- Realistic, because it uses HTTP download, transformation, and data querying.
+- Rich enough to exercise inputs, outputs, descriptions, code editing, and topology rendering.
+
+The visual topology implied by this workflow is:
+
+```text
+Start
+  |
+  v
+extract
+  |
+  v
+transform
+  |
+  v
+query
+  |
+  v
+End
+```
+
+---
+
+## Detail Page Notes
+
+The reference detail experience suggests a workflow page shaped around tabs, not a single long form.
+
+### Tabs Observed
+
+- `Overview`
+- `Topology`
+- `Executions`
+- `Edit`
+- `Revisions`
+- `Triggers`
+- `Logs`
+- `Metrics`
+- `Dependencies`
+- `Concurrency`
+- `Audit Logs`
+
+### SolidX Interpretation
+
+For SolidX workflow-definition UX, the current recommended prioritization is:
+
+- Keep: `Overview`
+- Keep: `Topology`
+- Keep: `Executions`
+- Keep: `Edit`
+- Keep: `Triggers`
+- Keep: `Logs`
+- Keep: `Metrics`
+- Ignore for now: `Revisions`
+- Ignore for now: `Concurrency`
+- Ignore for now: `Audit Logs`
+
+---
+
+## Overview Notes
+
+The overview page should change meaningfully once executions exist.
+
+### Before Executions
+
+The page can show an empty-state hero with a primary call to action:
+
+- execute workflow
+- guide the user toward the first run
+
+### After Executions
+
+The page can evolve into a summary dashboard showing:
+
+- success ratio
+- failed ratio
+- in-progress count
+- pending count
+- total executions
+- execution trend / chart
+- description panel
+
+This means the overview page should not be treated as static documentation. It should become an operational summary once runtime data exists.
+
+---
+
+## Topology Notes
+
+Topology is read-only in the reference detail page.
+
+Recommended SolidX interpretation:
+
+- `Topology` tab should be a clean read-only visualization.
+- It should render the workflow structure without edit affordances.
+- It can reuse the same layout logic as the editor flow view, but without mutation controls.
+
+---
+
+## Executions Notes
+
+The `Executions` tab should present a list of all executions for the workflow.
+
+Important characteristics from the reference:
+
+- list-based presentation
+- filtering/search support
+- empty state when no executions exist
+
+For SolidX:
+
+- use existing retrieve APIs
+- support server-side pagination, sorting, and search where possible
+- keep the visuals close to SolidX list/table patterns
+
+---
+
+## Edit Tab Notes
+
+This is the most important area for the current UI work.
+
+### Desired Structure
+
+- `Edit` is a top-level detail tab.
+- Inside `Edit`, only `Code` and `Flow` should be primary editing surfaces.
+- `Docs` should open in a side panel, not as a main third panel in the base layout.
+
+### Editor Layout
+
+- left: code editor
+- right: flow editor
+- docs: contextual side panel that opens on demand
+
+### Editing Mode
+
+- flow uses React Flow
+- modal editing for nodes
+- docs are contextual to the selected node type
+- save remains manual
+
+---
+
+## Flow Canvas Notes
+
+The reference behavior to preserve:
+
+- `+` insertion controls between nodes
+- per-node actions:
+  - edit
+  - delete
+  - view docs
+- directional flow should be visible
+- zoom in / zoom out controls should exist
+- users may drag nodes to rearrange the graph visually
+- dragging is for visual arrangement only, not for defining semantic connections via drag-and-drop wiring
+
+### Important Modeling Note
+
+Workflow structure is still governed by the workflow DSL contract.
+
+That means:
+
+- node order remains driven by workflow structure
+- edges are implicit from the DSL
+- the canvas is a visual/editor representation of the workflow, not the source of truth for execution semantics
+
+---
+
+## Triggers Notes
+
+Triggers are wanted, but not the first editing priority.
+
+Current note:
+
+- keep a `Triggers` tab in the long-term detail experience
+- rendering and editing of triggers can come after the core editor is stabilized
+
+---
+
+## Logs Notes
+
+Logs are wanted.
+
+Recommended direction:
+
+- separate `Logs` tab
+- searchable / filterable runtime log list
+- use workflow or execution context to scope logs
+
+---
+
+## Metrics Notes
+
+Metrics are wanted, but depend on runtime persistence.
+
+Important note:
+
+- metrics are only meaningful once we persist them into backing tables or another queryable runtime store
+
+So for now:
+
+- keep `Metrics` in the design vocabulary
+- treat it as a pending feature unless the persistence model is implemented
+
+---
+
+## Pending Feature Notes
+
+The following are intentionally deferred or partially deferred:
+
+- metrics persistence and reporting
+- trigger authoring experience
+- richer overview analytics
+- revisions
+- concurrency controls
+- audit-log specific workflow views
+- blueprints for this area as part of a later phase
+
+---
+
+## Screenshot References
+
+These screenshots informed the notes above:
+
+- `/Users/harishpatel/Desktop/Screenshot 2026-07-10 at 16.58.43.png`
+- `/Users/harishpatel/Desktop/Screenshot 2026-07-10 at 16.58.48.png`
+- `/Users/harishpatel/Desktop/Screenshot 2026-07-10 at 16.58.53.png`
+- `/Users/harishpatel/Desktop/Screenshot 2026-07-10 at 16.58.59.png`
+- `/Users/harishpatel/Desktop/Screenshot 2026-07-10 at 16.59.07.png`
+- `/Users/harishpatel/Desktop/Screenshot 2026-07-10 at 16.59.16.png`
+- `/Users/harishpatel/Desktop/Screenshot 2026-07-10 at 16.59.22.png`
+- `/Users/harishpatel/Desktop/Screenshot 2026-07-10 at 16.59.28.png`
+- `/Users/harishpatel/Desktop/Screenshot 2026-07-10 at 17.01.42.png`
+- `/Users/harishpatel/Desktop/Screenshot 2026-07-10 at 17.01.47.png`
+
+---
+
+## Recommended Next UI Build Order
+
+Based on this example, the clean next implementation sequence is:
+
+1. Finish the bespoke workflow-definition list view.
+2. Tighten the workflow-definition detail shell with tabs.
+3. Refine the `Edit` tab into:
+   - left code editor
+   - right React Flow editor
+   - contextual docs side panel
+4. Add modal-based node configuration using schema-driven forms.
+5. Add runtime tabs for executions, logs, and later metrics.