@solidxai/core 0.1.13-beta.3 → 0.1.13-beta.4

package/docs/workflow/workflow-module-metadata-dsl.md

@@ -0,0 +1,302 @@
+# Workflow Builder Metadata / DSL Shape
+
+## Purpose
+
+This document captures the current YAML representation for workflows in the `workflow-builder` module.
+
+The design goals are:
+
+- human-shareable
+- seedable
+- import/export friendly
+- expressive enough for a visual builder
+- structured enough for validation
+
+## Design Principles
+
+- YAML is the canonical persisted format in the `workflowDefinition.definitionYaml` field
+- The YAML must parse into a structured object with a top-level `nodes` array
+- Metadata, inputs, variables, nodes, and triggers should be explicit sections where needed
+- The format must support both runnable steps and control-flow nodes
+- The format must support future extensibility
+
+## Proposed Top-Level Shape
+
+```yaml
+version: "1.0"
+description: Checks service health and notifies on failure
+inputs:
+  uri:
+    type: uri
+    required: true
+    defaultValue: https://kestra.io
+variables: {}
+triggers: []
+nodes: []
+metadata:
+  namespace: ops.monitoring
+```
+
+## Top-Level Sections
+
+### `version`
+
+Optional version of the workflow DSL/schema.
+
+### `description`
+
+Optional workflow description.
+
+### `inputs`
+
+Optional map of workflow input definitions or input defaults.
+
+### `variables`
+
+Optional map of workflow variables. Runtime request variables can override these values.
+
+### `triggers`
+
+Optional array of trigger definitions.
+
+### `nodes`
+
+Required ordered array of workflow node definitions.
+
+### `metadata`
+
+Optional object for authoring, grouping, seed, or UI metadata that does not belong to runtime state.
+
+## Inputs Shape
+
+```yaml
+inputs:
+  uri:
+    type: uri
+    displayName: Target URI
+    required: true
+    defaultValue: https://kestra.io
+    description: Endpoint to check
+```
+
+## Trigger Shape
+
+```yaml
+triggers:
+  - id: dailySchedule
+    type: schedule
+    disabled: false
+    configuration:
+      cron: "0 9 * * *"
+    metadata:
+      inputOverrides:
+        uri: https://kestra.io/docs
+```
+
+## Node Shape
+
+All workflow steps should be represented as nodes.
+
+Suggested base shape:
+
+```yaml
+id: makeRequest
+kind: task
+type: http.request
+name: Make Request
+description: Calls the configured endpoint
+disabled: false
+configuration:
+  uri: "{{ inputs.uri }}"
+metadata: {}
+```
+
+### Node Kinds
+
+Initial node kinds:
+
+- `task`
+- `control`
+- `subflow`
+
+The agreed rule is:
+
+- `kind` is the broad behavior category
+- `type` is the concrete implementation
+
+Agreed v1 node type catalog:
+
+- `log.write`
+- `http.request`
+- `if`
+- `forEach`
+- `parallel`
+
+Deferred node types:
+
+- `switch`
+- `loopUntil`
+- `forEachItem`
+- `subflow`
+
+## Control-Flow Shape
+
+V1 sequencing follows Kestra-style ordered arrays with nested child arrays. We are not using a freeform graph model in v1.
+
+Example `if` node:
+
+```yaml
+id: checkStatus
+kind: control
+type: if
+configuration:
+  condition: "{{ outputs.makeRequest.statusCode == 200 }}"
+then:
+  - id: logSuccess
+    kind: task
+    type: log.write
+    configuration:
+      message: Request successful
+else:
+  - id: notifyFailure
+    kind: task
+    type: notification.slack
+    configuration:
+      message: Request failed
+```
+
+Example `forEach` node:
+
+```yaml
+id: forEachCustomer
+kind: control
+type: forEach
+configuration:
+  values: "{{ inputs.customerIds }}"
+children:
+  - id: processCustomer
+    kind: task
+    type: custom.processCustomer
+    configuration:
+      customerId: "{{ step.value }}"
+```
+
+Example `parallel` node:
+
+```yaml
+id: runChecks
+kind: control
+type: parallel
+configuration: {}
+branches:
+  - id: checkDocs
+    nodes:
+      - id: requestDocs
+        kind: task
+        type: http.request
+        configuration:
+          uri: https://kestra.io/docs
+  - id: checkPlugins
+    nodes:
+      - id: requestPlugins
+        kind: task
+        type: http.request
+        configuration:
+          uri: https://kestra.io/plugins
+```
+
+## V1 Node Type Configuration
+
+Detailed node contracts are captured in `docs/workflow-node-type-contracts.md`.
+
+### `log.write`
+
+```yaml
+id: logStatus
+kind: task
+type: log.write
+configuration:
+  message: "Status Code: {{ outputs.makeRequest.statusCode }}"
+```
+
+### `http.request`
+
+The HTTP node should use `uri`, matching the Kestra convention.
+
+```yaml
+id: makeRequest
+kind: task
+type: http.request
+configuration:
+  method: GET
+  uri: "{{ inputs.uri }}"
+  headers: {}
+  body: null
+```
+
+## Outputs Shape
+
+The current runtime returns the accumulated node outputs as the execution output payload. A future explicit flow-level outputs section can be added when the runtime starts evaluating named output expressions.
+
+## Example Workflow
+
+```yaml
+version: "1.0"
+description: Checks a URI and logs the result
+inputs:
+  uri:
+    type: uri
+    required: true
+    defaultValue: https://kestra.io
+triggers:
+  - id: dailySchedule
+    type: schedule
+    configuration:
+      cron: "0 9 * * *"
+nodes:
+  - id: makeRequest
+    kind: task
+    type: http.request
+    configuration:
+      method: GET
+      uri: "{{ inputs.uri }}"
+  - id: logStatus
+    kind: task
+    type: log.write
+    configuration:
+      message: "Status Code: {{ outputs.makeRequest.statusCode }}"
+metadata:
+  key: daily-health-check
+  displayName: Daily Health Check
+  namespace: ops.monitoring
+  definitionVersion: 0.1.0
+```
+
+## Seeding Metadata
+
+For portable and seeded workflows, we should likely support optional metadata such as:
+
+```yaml
+metadata:
+  seed:
+    sourceModule: workflow-builder
+    seedKey: daily-health-check
+    seedVersion: 1.0.0
+```
+
+This should live under top-level `metadata` unless the runtime grows a first-class seed section.
+
+## Snapshot Guidance
+
+At execution time, the system should preserve the exact YAML definition snapshot used for the run, along with:
+
+- `definitionVersion`
+- `version`
+- checksum
+
+## Open DSL Questions
+
+- Should `nodes` stay as an ordered array with nesting, or should we move to explicit edges later?
+- Should `namespace` be a simple string path in the DSL?
+- Should task configuration schemas be JSON Schema based from day one?
+- How should secret references be encoded inside configuration values?

package/docs/workflow/workflow-module-technical-spec.md

@@ -0,0 +1,293 @@
+# Workflow Builder Technical Specification
+
+## Overview
+
+`workflow-builder` will be implemented as a reusable SolidX module with:
+
+- backend module code in `solid-api`
+- frontend module code in `solid-ui`
+- module metadata for menus, actions, and views
+- YAML-based workflow definitions persisted and executed by the module
+
+## Technical Principles
+
+- Separate definition-time models from runtime execution models
+- Store workflows as canonical YAML containing a structured DSL, not arbitrary opaque text
+- Keep the engine extensible via registries for task and trigger types
+- Preserve immutable execution-time snapshots
+- Design for future revision support without requiring full revision UX in v1
+
+## Proposed Architecture
+
+### 1. Definition Layer
+
+Responsible for:
+
+- workflow CRUD
+- validation of workflow YAML
+- seed/import/export
+- blueprint/template management
+- registration of task and trigger types
+
+### 2. Runtime Layer
+
+Responsible for:
+
+- execution creation
+- context hydration
+- expression resolution
+- step scheduling / traversal
+- step execution
+- state persistence
+- logs, outputs, and artifacts
+
+### 3. Authoring Layer
+
+Responsible for:
+
+- workflow editing UI
+- typed configuration UI for tasks/triggers
+- validation feedback
+- topology / graph visualization over time
+
+## Backend Concerns
+
+### Definition Services
+
+Likely services:
+
+- `WorkflowDefinitionService`
+- `WorkflowValidationService`
+- `WorkflowSeedService`
+- `WorkflowBlueprintService`
+- `WorkflowRegistryService`
+
+### Runtime Services
+
+Likely services:
+
+- `WorkflowExecutionService`
+- `WorkflowContextService`
+- `ExpressionEvaluationService`
+- `WorkflowTraversalService`
+- `TaskExecutionService`
+- `TriggerDispatchService`
+
+### Registries
+
+The system should use registries instead of hard-coded switch blocks for every capability:
+
+- task type registry
+- trigger type registry
+- expression function registry later if needed
+
+For node execution, registry entries should eventually expose:
+
+- type id
+- kind
+- category
+- configuration schema
+- UI schema
+- runtime handler
+- output schema
+- documentation metadata
+
+## Frontend Concerns
+
+### Admin Surfaces
+
+Likely UI areas:
+
+- workflow list
+- workflow editor
+- execution list
+- execution detail
+- blueprint gallery
+- task/trigger type config panels
+
+### Authoring Modes
+
+Recommended phased approach:
+
+- V1: schema-driven form/editor experience
+- V1.x: structured YAML editor + validation
+- Later: canvas/topology builder with drag-drop
+
+## Execution Model
+
+At runtime, an execution should:
+
+1. Resolve the published workflow definition
+2. Persist an immutable definition snapshot or checksum reference
+3. Merge manual or trigger-provided inputs
+4. Build an execution context
+5. Traverse nodes according to orchestration rules
+6. Execute runnable steps
+7. Persist step outputs, logs, and states
+8. Finalize flow outputs and execution status
+
+V1 traversal model:
+
+- top-level workflow nodes are executed as an ordered array
+- control-flow nodes own nested child arrays such as `then`, `else`, `children`, or `branches`
+- we defer full graph/edge traversal until a later phase
+
+V1 node type catalog:
+
+- `log.write`
+- `http.request`
+- `if`
+- `forEach`
+- `parallel`
+
+Deferred node types:
+
+- `switch`
+- `loopUntil`
+- `forEachItem`
+- `subflow`
+- database/query tasks
+- notification tasks beyond logging
+
+## State Model
+
+Suggested initial execution states:
+
+- `created`
+- `running`
+- `success`
+- `failed`
+- `cancelled`
+
+Suggested step states:
+
+- `pending`
+- `running`
+- `success`
+- `failed`
+- `skipped`
+
+## Snapshot vs Revisions
+
+V1 recommendation:
+
+- store the current workflow definition
+- store a version string and/or checksum
+- store the exact definition snapshot used by each execution
+
+Defer until later:
+
+- full revision browsing
+- rollback UX
+- diff tooling
+
+## Trigger Strategy
+
+V1 can support a limited trigger set such as:
+
+- manual
+- schedule
+- webhook
+
+The trigger model should still be extensible enough to support:
+
+- polling
+- event-based triggers
+- flow-completion triggers
+
+## Expression Strategy
+
+We need one engine-wide expression approach. The chosen solution should support:
+
+- context access
+- boolean conditions
+- string interpolation
+- light transformations
+- safe evaluation
+
+Minimum context objects:
+
+- `inputs`
+- `outputs`
+- `workflow`
+- `execution`
+- `trigger`
+- `step` or `taskrun` equivalent where relevant
+
+Recommended DSL syntax:
+
+- `{{ inputs.uri }}`
+- `{{ outputs.makeRequest.statusCode }}`
+- `{{ outputs.makeRequest.statusCode == 200 }}`
+- `{{ step.value }}`
+
+V1 should support string interpolation and expression-only evaluation. Expression evaluation must be safe and constrained; workflow definitions must not be able to execute arbitrary JavaScript or access process, filesystem, network, or module APIs through expressions.
+
+The expression engine should be hidden behind `ExpressionEvaluationService` so we can change implementation without changing the workflow DSL.
+
+## Node Provider Pluggability
+
+Workflow node types should follow the existing SolidX provider pattern used by selection providers, computed field providers, dashboard widget data providers, settings providers, and communication providers.
+
+Recommended pattern:
+
+- define a `WorkflowNodeTypeProvider` interface
+- define a `@WorkflowNodeTypeProvider()` decorator
+- implement node providers as Nest injectable classes
+- discover decorated providers during application bootstrap
+- register discovered providers into a `WorkflowNodeTypeRegistry`
+- resolve providers by `node.type` at validation and execution time
+
+This keeps node availability tied to currently imported SolidX/Nest modules. A consuming app or distributed module can make new workflow nodes available by importing the module that provides decorated node providers.
+
+## Secret Strategy
+
+Definitions should store secret references, not secret values. Resolution should happen only at runtime via a secret provider abstraction.
+
+Recommended abstraction:
+
+- `SecretReference`
+- `SecretResolver`
+
+## Seeding and Portability
+
+Workflow definitions should be portable assets. A seeded workflow should carry metadata such as:
+
+- source module
+- seed key
+- seed version
+- definition version
+
+## Suggested Implementation Phases
+
+### Phase 1
+
+- domain models
+- JSON schema/validator for workflow definition
+- definition CRUD
+- manual execution
+- basic runnable task execution for `log.write` and `http.request`
+- registry-driven node type definitions
+- control-flow execution for `if`, `forEach`, and `parallel`
+- execution persistence
+
+### Phase 2
+
+- triggers
+- control-flow tasks
+- secret references
+- blueprint support
+
+### Phase 3
+
+- richer editor UX
+- broader plugin/task ecosystem
+- replay/retry semantics
+- fuller revision model
+
+## Open Technical Questions
+
+- Should the workflow DSL schema be internal-first or externally published from day one?
+- Should execution snapshots store the full JSON body or a normalized linked artifact?
+- What expression engine best fits NestJS + SolidX constraints?
+- How should file/artifact outputs be stored and referenced?