@rglabs/butterfly 2.0.1

package/docs/WORKFLOW_API.md

@@ -0,0 +1,480 @@
+# Workflow API Reference
+
+This document describes workflow management, node connections, and synchronization features in the CLI.
+
+---
+
+## Table of Contents
+
+- [File Structure](#file-structure)
+- [Downloading Workflows](#downloading-workflows)
+- [Watch Mode (Real-time Sync)](#watch-mode-real-time-sync)
+- [Node Connections](#node-connections)
+- [Version Management](#version-management)
+- [Discovering Node Groups and Types](#discovering-node-groups-and-types)
+- [Creating Workflows and Nodes](#creating-workflows-and-nodes)
+- [API Methods Reference](#api-methods-reference)
+
+---
+
+## File Structure
+
+When downloading workflows, the CLI creates the following structure:
+
+```
+bfy_workflows/
+└── [workflow_name]/                    # Folder named by workflow name
+    ├── workflow.json                   # Main workflow metadata
+    └── v[version_number]/              # Version folder (v1, v2, etc.)
+        ├── version.json                # Version data with status
+        ├── connections.json            # Node connections for this version
+        └── nodes/
+            └── [node_title]_[id]/      # Node folder (e.g., "Process_Data_123")
+                ├── node.json           # Node metadata (position, title, etc.)
+                ├── code.bfy            # Node code (if exists)
+                └── params.yaml         # Node parameters (if exists)
+```
+
+### File Descriptions
+
+| File | Description |
+|------|-------------|
+| `workflow.json` | Main workflow definition (name, description, settings) |
+| `version.json` | Version metadata including status (1=Draft, 2=Published, 3=Archived) |
+| `connections.json` | Array of connections between nodes |
+| `node.json` | Node configuration (position_x, position_y, title, node_group, etc.) |
+| `code.bfy` | Node's executable code (extracted from `params.code` or `node.code`) |
+| `params.yaml` | Node parameters in YAML format (excluding code) |
+
+---
+
+## Downloading Workflows
+
+### Download All Workflows
+
+```bash
+butterfly-cli download -t bfy_workflows
+```
+
+### Download with Cleanup
+
+```bash
+butterfly-cli download -t bfy_workflows --cleanup
+```
+
+### Download All Resources (includes workflows)
+
+```bash
+butterfly-cli download
+```
+
+---
+
+## Watch Mode (Real-time Sync)
+
+Upload changes back to the platform:
+
+```bash
+butterfly-cli upload <path>
+```
+
+### Watched Files
+
+| File | Sync Behavior |
+|------|---------------|
+| `workflow.json` | Updates workflow metadata |
+| `version.json` | Updates version status and metadata |
+| `node.json` | Updates node position, title, and configuration |
+| `code.bfy` | Updates node code (merged into `params.code` or `node.code`) |
+| `params.yaml` | Updates node parameters (merged with code if both exist) |
+| `connections.json` | **Full sync**: creates, updates, and deletes connections |
+
+### Connection Sync Behavior
+
+When you edit `connections.json`, the CLI:
+
+1. **Detects new connections** (entries without `id`) - Creates them on the server and writes back the assigned ID
+2. **Detects modified connections** - Updates changed fields on the server
+3. **Detects deleted connections** - Removes them from the server
+
+The connection cache is initialized at startup to accurately track changes.
+
+---
+
+## Node Connections
+
+### Connection Structure
+
+Each connection in `connections.json` has the following fields:
+
+```json
+{
+  "id": 123,
+  "source_node_id": 1,
+  "target_node_id": 2,
+  "source_output_key": "output",
+  "target_input_key": "input",
+  "connection_type": "data"
+}
+```
+
+| Field | Description |
+|-------|-------------|
+| `id` | Connection ID (assigned by server for new connections) |
+| `source_node_id` | ID of the source node |
+| `target_node_id` | ID of the target node |
+| `source_output_key` | Output connector key on source node |
+| `target_input_key` | Input connector key on target node |
+| `connection_type` | Type of connection (e.g., "data", "control") |
+
+### Creating a New Connection
+
+Add a new entry to `connections.json` **without an `id`**:
+
+```json
+[
+  {
+    "source_node_id": 1,
+    "target_node_id": 2,
+    "source_output_key": "output",
+    "target_input_key": "input"
+  }
+]
+```
+
+Save the file, and watch mode will:
+1. Create the connection on the server
+2. Write back the assigned `id` to the file
+
+### Deleting a Connection
+
+Remove the connection entry from `connections.json` and save. Watch mode will delete it from the server.
+
+### Node Connectors
+
+Nodes have three types of connectors:
+
+| Connector Type | Color | Purpose |
+|----------------|-------|---------|
+| **Input** | Blue | Receive data from other nodes |
+| **Output** | Green | Send data to other nodes |
+| **Preference** | Purple | Configuration values |
+
+---
+
+## Version Management
+
+### Version Statuses
+
+| Status | Value | Description |
+|--------|-------|-------------|
+| Draft | 1 | Editable version |
+| Published | 2 | Read-only, active version |
+| Archived | 3 | Read-only, historical version |
+
+### Changing Version Status
+
+Edit `version.json` and change the `status` field:
+
+```json
+{
+  "id": 1,
+  "bfy_workflow_id": 1,
+  "version_number": 1,
+  "status": 2
+}
+```
+
+Save the file, and watch mode will update the status on the server.
+
+### Version Folder Naming
+
+Versions are stored in folders named `v[version_number]`:
+- `v1/` - Version 1
+- `v2/` - Version 2
+- etc.
+
+---
+
+## Discovering Node Groups and Types
+
+Use the `workflow-info` command to dynamically discover available node groups and types from the API.
+
+### List All Node Groups
+
+```bash
+butterfly-cli workflow-info --groups
+```
+
+### List Nodes in a Group
+
+```bash
+butterfly-cli workflow-info --nodes <group>
+
+# Examples:
+butterfly-cli workflow-info --nodes Database
+butterfly-cli workflow-info --nodes Control
+butterfly-cli workflow-info --nodes Notification
+butterfly-cli workflow-info --nodes Trigger
+butterfly-cli workflow-info --nodes Code
+```
+
+### Get Node Details
+
+```bash
+butterfly-cli workflow-info --details <node> --group <group>
+
+# Examples:
+butterfly-cli workflow-info --details MySQL --group Database
+butterfly-cli workflow-info --details Alarm --group Control
+butterfly-cli workflow-info --details Mailer --group Notification
+butterfly-cli workflow-info --details CustomScript --group Code
+```
+
+---
+
+## Creating Workflows and Nodes
+
+### Create a New Workflow
+
+```bash
+butterfly-cli add -t workflow --title "My Workflow"
+```
+
+### Add Nodes to a Workflow
+
+Use the `--node-group` and `--node-type` values from `workflow-info` output:
+
+```bash
+butterfly-cli add -t workflow-node -w "<workflow-name>" --title "<node-title>" --node-group <group> --node-type <type>
+```
+
+### Node Creation Options
+
+| Option | Description |
+|--------|-------------|
+| `-w, --workflow <name>` | Workflow name or ID (required) |
+| `--title <title>` | Node title |
+| `--node-group <group>` | Node group from `workflow-info --groups` |
+| `--node-type <type>` | Node type from `workflow-info --nodes <group>` |
+| `--system-name <name>` | System name for the node |
+| `-v, --version <number>` | Version number (defaults to latest) |
+| `--connect-from <spec>` | Connect FROM existing node TO new node |
+| `--connect-to <spec>` | Connect FROM new node TO existing node |
+
+---
+
+## Creating Connections Between Existing Nodes
+
+Use the `workflow-connection` type to create connections between existing nodes:
+
+```bash
+butterfly-cli add -t workflow-connection -w "<workflow-name>" --from <source_id> --to <target_id>
+```
+
+### Connection Creation Options
+
+| Option | Description |
+|--------|-------------|
+| `-w, --workflow <name>` | Workflow name or ID (required) |
+| `--from <id>` | Source node ID (required) |
+| `--from-output <key>` | Source node output key (default: `output`) |
+| `--to <id>` | Target node ID (required) |
+| `--to-input <key>` | Target node input key (default: `input`) |
+| `-v, --version <number>` | Version number (defaults to latest) |
+
+### Connection Examples
+
+```bash
+# Create a basic connection between two nodes
+butterfly-cli add -t workflow-connection -w "Order Processing" --from 123 --to 456
+
+# Specify custom output key on source node
+butterfly-cli add -t workflow-connection -w "Order Processing" --from 123 --from-output result --to 456
+
+# Specify custom input key on target node
+butterfly-cli add -t workflow-connection -w "Order Processing" --from 123 --to 456 --to-input data
+
+# Specify both custom keys
+butterfly-cli add -t workflow-connection -w "Order Processing" --from 123 --from-output result --to 456 --to-input data
+
+# Create connection in a specific version
+butterfly-cli add -t workflow-connection -w "Order Processing" -v 2 --from 123 --to 456
+```
+
+### Connection Specification Format
+
+**`--connect-from <node_id>[:<output_key>]`**
+- Connects FROM an existing node's output TO the new node's input
+- `output_key` specifies the source node's output connector (default: `output`)
+- The new node's input connector is always `input`
+
+**`--connect-to <node_id>[:<input_key>]`**
+- Connects FROM the new node's output TO an existing node's input
+- `input_key` specifies the target node's input connector (default: `input`)
+- The new node's output connector is always `output`
+
+**Examples:**
+- `--connect-from 123` - Connect from node 123's "output" to new node's "input"
+- `--connect-from 123:result` - Connect from node 123's "result" output to new node's "input"
+- `--connect-to 456` - Connect from new node's "output" to node 456's "input"
+- `--connect-to 456:data` - Connect from new node's "output" to node 456's "data" input
+
+### Examples
+
+```bash
+# Create a new workflow
+butterfly-cli add -t workflow --title "Order Processing"
+
+# Add a CustomScript node
+butterfly-cli add -t workflow-node -w "Order Processing" --title "Process Data" --node-group Code --node-type CustomScript
+
+# Add a REST API connector
+butterfly-cli add -t workflow-node -w "Order Processing" --title "API Call" --node-group Connector --node-type RESTAPI
+
+# Add a webhook trigger
+butterfly-cli add -t workflow-node -w "Order Processing" --title "Webhook Trigger" --node-group Trigger --node-type WebHook
+
+# Add node to specific version
+butterfly-cli add -t workflow-node -w "Order Processing" -v 1 --title "New Node"
+
+# Add a node and connect FROM an existing node (node 123) TO the new node
+butterfly-cli add -t workflow-node -w "Order Processing" --title "Next Step" --connect-from 123
+
+# Add a node and connect FROM the new node TO an existing node (node 456)
+butterfly-cli add -t workflow-node -w "Order Processing" --title "Pre-Process" --connect-to 456
+
+# Add a node with custom output key from source node
+butterfly-cli add -t workflow-node -w "Order Processing" --title "Transform" --connect-from 123:result
+
+# Add a node with custom input key to target node
+butterfly-cli add -t workflow-node -w "Order Processing" --title "Finalize" --connect-to 456:data
+
+# Add a node with both incoming and outgoing connections
+butterfly-cli add -t workflow-node -w "Order Processing" --title "Middle Node" --connect-from 123 --connect-to 456
+```
+
+---
+
+## API Methods Reference
+
+The following API methods are available in `ButterflyAPI` class for workflow management:
+
+### Connection Methods
+
+| Method | Description |
+|--------|-------------|
+| `createWorkflowConnection(data)` | Create a new connection between nodes |
+| `updateWorkflowConnection(id, data)` | Update an existing connection |
+| `deleteWorkflowConnection(id)` | Delete a connection |
+
+### Node Methods
+
+| Method | Description |
+|--------|-------------|
+| `createWorkflowNode(data)` | Create a new node in a workflow version |
+| `updateWorkflowNode(id, data)` | Update node properties (position, title, code, params) |
+| `deleteWorkflowNode(id)` | Delete a node from a workflow |
+
+### Version Methods
+
+| Method | Description |
+|--------|-------------|
+| `createWorkflowVersion(data)` | Create a new version of a workflow |
+| `updateWorkflowVersion(id, data)` | Update version metadata |
+| `publishWorkflowVersion(id)` | Set version status to Published (2) |
+| `archiveWorkflowVersion(id)` | Set version status to Archived (3) |
+
+### API Endpoints
+
+All workflow operations use the standard CMS object operation endpoint:
+
+```
+POST /admin/ajax/cms_object/operation?do={table}__{action}
+```
+
+| Table | Actions |
+|-------|---------|
+| `bfy_workflows` | `add`, `edit`, `delete` |
+| `bfy_workflow_versions` | `add`, `edit`, `delete` |
+| `bfy_workflow_nodes` | `add`, `edit`, `delete` |
+| `bfy_workflow_connections` | `add`, `edit`, `delete` |
+
+---
+
+## Node Code and Parameters
+
+### Code Extraction
+
+When downloading, node code is extracted to separate files:
+
+1. If `node.code` exists → `code.bfy`
+2. If `params.code` exists → `code.bfy` (and removed from params.yaml)
+3. Other params → `params.yaml`
+
+### Code Sync (Watch Mode)
+
+When syncing back:
+
+1. If `params.yaml` exists:
+   - Code from `code.bfy` is merged into `params.code`
+   - Combined params JSON is sent to server
+2. If only `code.bfy` exists:
+   - Code is sent directly as `node.code`
+
+### Example Node Files
+
+**node.json:**
+```json
+{
+  "id": 123,
+  "bfy_workflow_version_id": 1,
+  "title": "Process Order",
+  "node_group_class_name": "Code",
+  "node_identifier_key": "CustomScript",
+  "position_x": 250,
+  "position_y": 150
+}
+```
+
+**code.bfy:**
+```twig
+{% set order = input.order %}
+{% set total = order.items|reduce((carry, item) => carry + item.price, 0) %}
+
+{{ output('result', { 'total': total }) }}
+```
+
+**params.yaml:**
+```yaml
+timeout: 30
+retry_count: 3
+error_handling: continue
+```
+
+---
+
+## Troubleshooting
+
+### Connection Sync Issues
+
+If connections aren't syncing properly:
+
+1. Check `.butterfly/error_log.txt` for errors
+2. Ensure the workflow version status is Draft (1)
+3. Verify node IDs in connections match existing nodes
+
+### Node Code Not Updating
+
+If node code changes aren't reflected:
+
+1. Check if `params.yaml` exists - code goes into `params.code`
+2. Without `params.yaml`, code goes directly to `node.code`
+3. Both files are merged when syncing
+
+### Version Status Locked
+
+Published (2) and Archived (3) versions are read-only on the server. To edit:
+
+1. Create a new Draft version
+2. Or change status back to Draft (1) via `version.json`

package/docs/bfy-splitting.md

@@ -0,0 +1,126 @@
+# BFY File Splitting
+
+Butterfly CLI supports splitting large `.bfy` files into smaller, manageable parts. This is useful when a single file (like `processing_code.bfy`) becomes too large to manage.
+
+## Marker Syntax
+
+Use these markers to define parts within a `.bfy` file:
+
+```twig
+{# --- part: part_name --- #}
+... part content ...
+{# --- endpart --- #}
+```
+
+**Rules:**
+- Part names must be alphanumeric with underscores or hyphens (`[a-zA-Z0-9_-]+`)
+- Each `part` marker must have a matching `endpart` marker
+- Part names must be unique within the file
+
+## How It Works
+
+### Splitting (Download)
+
+When you download resources, the CLI automatically detects `.bfy` files with part markers and splits them:
+
+**Before splitting:**
+```
+processing_code.bfy
+```
+
+**After splitting:**
+```
+processing_code.bfy          # Container file (with include markers)
+processing_code/             # Parts folder
+  ├── validation.bfy
+  └── calculation.bfy
+```
+
+The container file will have `{# --- include: part_name --- #}` markers where the parts were extracted.
+
+### Merging (Upload)
+
+When you upload or read a split file, the CLI automatically merges the parts back:
+- Detects `{# --- include: name --- #}` markers in container file
+- Looks for corresponding `.bfy` files in the parts folder
+- Replaces include markers with full part content (wrapped in `part`/`endpart` markers)
+
+## Example
+
+### Original File (before download)
+
+```twig
+{% set data = getValue('items') %}
+
+{# --- part: validation --- #}
+{% if data|length == 0 %}
+    {{ errorMessage('No items found') }}
+{% endif %}
+
+{% for item in data %}
+    {% if item.quantity <= 0 %}
+        {{ errorMessage('Invalid quantity for ' ~ item.name) }}
+    {% endif %}
+{% endfor %}
+{# --- endpart --- #}
+
+{# --- part: calculation --- #}
+{% set total = 0 %}
+{% for item in data %}
+    {% set total = total + (item.price * item.quantity) %}
+{% endfor %}
+{{ setValue('total', total) }}
+{# --- endpart --- #}
+```
+
+### After Download (split)
+
+**processing_code.bfy** (container):
+```twig
+{% set data = getValue('items') %}
+
+{# --- include: validation --- #}
+
+{# --- include: calculation --- #}
+```
+
+**processing_code/validation.bfy**:
+```twig
+{% if data|length == 0 %}
+    {{ errorMessage('No items found') }}
+{% endif %}
+
+{% for item in data %}
+    {% if item.quantity <= 0 %}
+        {{ errorMessage('Invalid quantity for ' ~ item.name) }}
+    {% endif %}
+{% endfor %}
+```
+
+**processing_code/calculation.bfy**:
+```twig
+{% set total = 0 %}
+{% for item in data %}
+    {% set total = total + (item.price * item.quantity) %}
+{% endfor %}
+{{ setValue('total', total) }}
+```
+
+## When to Use
+
+Use file splitting when:
+- A `.bfy` file has multiple distinct logical sections
+- The file is too large to navigate easily
+- You want to organize code by functionality (validation, calculation, rendering, etc.)
+
+## Technical Details
+
+| Function | Description |
+|----------|-------------|
+| `splitBfyFile()` | Splits a file with part markers into container + parts folder |
+| `mergeBfyFile()` | Merges parts back into single content for upload |
+| `smartReadBfyContent()` | Automatically merges split files when reading |
+| `hasSplitMarkers()` | Checks if content has `{# --- part: ... --- #}` markers |
+| `hasIncludeMarkers()` | Checks if content has `{# --- include: ... --- #}` markers |
+
+> **Note:** The splitting/merging is handled automatically by the CLI. You only need to add the part markers to your code.