ultravisor 1.0.3 → 1.0.8

package/docs/features/llm.md

@@ -0,0 +1,262 @@
+# LLM Integration
+
+Ultravisor can send requests to large language models (LLMs) through the
+Beacon system. An LLM Beacon wraps an API — OpenAI, Anthropic, a local
+Ollama instance, or any OpenAI-compatible endpoint — and exposes it as a
+capability that operation graphs can use like any other task.
+
+This means you can build workflows that read data, send it to an LLM for
+analysis or generation, and act on the response, all within the standard
+Ultravisor operation graph.
+
+## How It Works
+
+The LLM functionality has two layers:
+
+1. **Beacon Provider** (`LLM`) — runs on the Beacon worker, wraps the
+   actual HTTP calls to the LLM API. This is the "thin wrapper" that
+   handles authentication, request formatting, and response normalization
+   across different backends.
+
+2. **Graph Task Types** (`llm-chat-completion`, `llm-embedding`,
+   `llm-tool-use`) — run on the Ultravisor server, dispatch work to an
+   LLM Beacon, manage conversation history, and route results back into
+   the operation state.
+
+```
+┌──────────────────────┐         ┌────────────────────┐        ┌──────────────┐
+│   Operation Graph    │         │  BeaconCoordinator │        │  LLM Beacon  │
+│                      │         │                    │        │              │
+│  llm-chat-completion │─enqueue─▶  Work Queue       │        │  LLM Provider│
+│  (builds messages,   │         │                    │◀─poll──│              │
+│   manages history)   │         │  assign work       │─work──▶│  HTTP call   │
+│                      │         │                    │        │  to LLM API  │
+│  WaitingForInput...  │         │                    │◀result─│  (OpenAI,    │
+│                      │         │  resumeOperation() │        │   Anthropic, │
+│  updates history,    │◀resume──│                    │        │   Ollama)    │
+│  writes to state     │         │                    │        │              │
+└──────────────────────┘         └────────────────────┘        └──────────────┘
+```
+
+## Supported Backends
+
+| Backend            | Config value         | Description                                |
+|--------------------|----------------------|--------------------------------------------|
+| OpenAI             | `openai`             | GPT-4, GPT-4o, o1, etc.                   |
+| Anthropic          | `anthropic`          | Claude Opus, Sonnet, Haiku                 |
+| Ollama             | `ollama`             | Local models (Llama 3, Mistral, Phi, etc.) |
+| OpenAI-compatible  | `openai-compatible`  | Any API that follows the OpenAI format     |
+
+The provider normalizes differences between backends automatically. For
+example, Anthropic uses `x-api-key` headers and a different message
+format; Ollama puts parameters in an `options` object. You don't need to
+worry about these details — just set the `Backend` config value and
+provide your messages.
+
+## Actions
+
+### ChatCompletion
+
+Send messages to an LLM and receive a text completion.
+
+**Settings:**
+
+| Name            | Required | Description                                      |
+|-----------------|----------|--------------------------------------------------|
+| Messages        | No       | JSON array of `{role, content}` message objects   |
+| SystemPrompt    | No       | System prompt (prepended automatically)           |
+| Model           | No       | Override the default model                        |
+| Temperature     | No       | Sampling temperature (0-2, lower = more focused)  |
+| MaxTokens       | No       | Maximum tokens to generate                        |
+| TopP            | No       | Nucleus sampling parameter                        |
+| StopSequences   | No       | JSON array of stop sequences                      |
+| ResponseFormat  | No       | `"text"` or `"json_object"`                       |
+
+**Outputs:**
+
+| Name             | Description                          |
+|------------------|--------------------------------------|
+| Content          | The generated text                   |
+| Model            | Model that produced the response     |
+| FinishReason     | Why generation stopped (`stop`, `length`, etc.) |
+| PromptTokens     | Token count for the input            |
+| CompletionTokens | Token count for the output           |
+| TotalTokens      | Combined token count                 |
+| Result           | Same as Content (convention)         |
+
+### Embedding
+
+Generate vector embeddings for text.
+
+**Settings:**
+
+| Name  | Required | Description                       |
+|-------|----------|-----------------------------------|
+| Text  | Yes      | Text to embed                     |
+| Model | No       | Override the default model         |
+
+**Outputs:**
+
+| Name       | Description                              |
+|------------|------------------------------------------|
+| Embedding  | JSON array of floating point numbers     |
+| Dimensions | Number of dimensions in the embedding    |
+| Model      | Model used                               |
+
+### ToolUse
+
+Chat completion with tool/function definitions. The LLM can request tool
+calls, which are returned as structured data.
+
+**Settings:**
+
+| Name       | Required | Description                                |
+|------------|----------|--------------------------------------------|
+| Messages   | Yes      | JSON array of message objects              |
+| Tools      | Yes      | JSON array of tool definitions             |
+| Model      | No       | Override model name                        |
+| ToolChoice | No       | `"auto"`, `"none"`, or a specific tool name |
+| Temperature| No       | Sampling temperature                       |
+| MaxTokens  | No       | Maximum tokens to generate                 |
+
+**Outputs:**
+
+| Name             | Description                                |
+|------------------|--------------------------------------------|
+| Content          | Text response (may be empty if tool call)  |
+| ToolCalls        | JSON array of tool call objects            |
+| FinishReason     | `stop`, `tool_calls`, etc.                 |
+| PromptTokens     | Token count for the input                  |
+| CompletionTokens | Token count for the output                 |
+
+Tool call objects are normalized to the OpenAI format regardless of
+backend:
+
+```json
+{
+  "id": "call_abc123",
+  "type": "function",
+  "function": {
+    "name": "get_weather",
+    "arguments": "{\"city\": \"Portland\"}"
+  }
+}
+```
+
+## Graph Task Types
+
+The server-side task types wrap `beacon-dispatch` with LLM-specific
+conveniences: message building, conversation history, and state I/O.
+
+### llm-chat-completion
+
+The primary task for using LLMs in operation graphs.
+
+**Convenience settings** (in addition to all ChatCompletion settings):
+
+| Name               | Description                                              |
+|--------------------|----------------------------------------------------------|
+| UserPrompt         | User message text (builds Messages for you)              |
+| InputAddress       | State address to read context data from — appended to UserPrompt |
+| Destination        | State address to write the completion content to         |
+| AffinityKey        | Route to a specific Beacon                               |
+| TimeoutMs          | Override timeout (default: 120000)                       |
+
+**Example node data:**
+
+```json
+{
+  "SystemPrompt": "You are a code review assistant.",
+  "UserPrompt": "Review this code for bugs:",
+  "InputAddress": "TaskOutput.read-file-node.Content",
+  "Destination": "Operation.ReviewResult",
+  "Temperature": 0.3,
+  "MaxTokens": 2048
+}
+```
+
+### Conversation Management
+
+Multi-turn conversations are supported through shared message history
+stored in operation or global state.
+
+| Setting                    | Description                                            |
+|----------------------------|--------------------------------------------------------|
+| ConversationAddress        | State address for the message history array            |
+| AppendToConversation       | Append this exchange to history (default: true)        |
+| ConversationMaxMessages    | Sliding window — keep only the N most recent messages  |
+| ConversationMaxTokens      | Token budget — trim oldest messages when exceeded      |
+| PersistConversation        | Copy history to GlobalState on completion              |
+| ConversationPersistAddress | GlobalState address for cross-operation persistence    |
+
+**How it works:**
+
+When `ConversationAddress` is set, the task:
+
+1. Reads existing message history from that state address
+2. Prepends `SystemPrompt` if not already present
+3. Appends the new `UserPrompt` as a user message
+4. Applies sliding window limits (max messages, max tokens)
+5. Sends the full history to the LLM
+6. Appends the assistant response to the history
+7. Writes updated history back to the state address
+
+**Within a single operation** — use `Operation.` addresses:
+
+```
+llm-node-1 (ConversationAddress: "Operation.Chat")
+    → llm-node-2 (ConversationAddress: "Operation.Chat")
+    → llm-node-3 (ConversationAddress: "Operation.Chat")
+```
+
+Each node sees the full conversation so far and adds to it.
+
+**Across operations** — use `Global.` addresses:
+
+Set `ConversationAddress: "Global.Sessions.MyAgent"` to store history in
+GlobalState, which persists across operation runs.
+
+Or use `PersistConversation: true` with
+`ConversationPersistAddress: "Global.Sessions.MyAgent"` to work in
+OperationState during the run and copy to GlobalState when done.
+
+### llm-embedding
+
+Dispatches an embedding request. Supports `InputAddress` to read text
+from state and `Destination` to write the embedding.
+
+### llm-tool-use
+
+Like `llm-chat-completion` but includes tool definitions. Supports
+conversation management. Returns both `Content` and `ToolCalls`.
+
+## Routing
+
+All LLM Beacons advertise `Capability: "LLM"`. When an `llm-chat-completion`
+task runs, the coordinator assigns the work to any available LLM Beacon.
+
+To target a specific Beacon (e.g., the one running Claude vs. the one
+running Llama), use `AffinityKey`:
+
+```json
+{
+  "UserPrompt": "Explain quantum computing",
+  "AffinityKey": "claude-worker"
+}
+```
+
+The first request with that affinity key binds to whichever Beacon picks
+it up. Subsequent requests with the same key go to the same Beacon.
+
+## Example Operations
+
+Two example operations are included in `operation-library/`:
+
+- **llm-summarize.json** — Reads a file, sends it to an LLM for
+  summarization, writes the result. Demonstrates single-turn usage with
+  `InputAddress` and `Destination`.
+
+- **llm-multi-turn-analysis.json** — Three chained LLM calls sharing
+  a `ConversationAddress`: initial analysis, follow-up question, and
+  final synthesis. Demonstrates multi-turn conversation with persistence
+  to GlobalState.

package/docs/features/tasks-content-system.md

@@ -0,0 +1,110 @@
+# Content System Tasks
+
+Tasks for reading, writing, and managing files on remote Content System beacon workers. These tasks operate on remote content directories rather than the local file system.
+
+---
+
+## Content Read File
+
+Reads a markdown or content file from a remote Content System beacon.
+
+### Settings
+
+- **FilePath** — Relative path within the content directory on the beacon.
+- **Destination** — State address to store the file content.
+- **AffinityKey** — Worker affinity key to route to a specific beacon.
+- **TimeoutMs** — Timeout in milliseconds (default `300000`).
+
+### Outputs
+
+- **Content** — The file content as a string.
+- **StdOut** — Status message from the beacon.
+
+### Events
+
+- **Complete** — Fires after a successful read.
+- **Error** — Fires if the file is not found or the beacon is unreachable.
+
+### Tips
+
+Content System cards operate on remote Beacon workers, not the local file system. Use AffinityKey to ensure reads and writes go to the same beacon instance. For local file operations, use **Read File** instead.
+
+---
+
+## Content Save File
+
+Saves content to a file on a remote Content System beacon.
+
+### Settings
+
+- **FilePath** — Relative path within the content directory.
+- **Content** — The file content to write. Supports Pict template expressions.
+- **AffinityKey** — Worker affinity key to route to a specific beacon.
+- **TimeoutMs** — Timeout in milliseconds (default `300000`).
+
+### Outputs
+
+- **FilePath** — Path of the saved file.
+- **StdOut** — Status message from the beacon.
+
+### Events
+
+- **Complete** — Fires after a successful save.
+- **Error** — Fires on failure.
+
+### Tips
+
+Use Content Save File to publish generated content (reports, processed data, rendered templates) to a Content System. Pair with **Content Read File** to read-modify-write content files on a beacon.
+
+---
+
+## Content List Files
+
+Lists files in a directory on a remote Content System beacon.
+
+### Settings
+
+- **Path** — Relative directory path within the content directory. Leave empty for the root.
+- **Pattern** — Glob pattern filter (e.g. `*.md`, `*.json`).
+- **Recursive** — When `true`, lists files in subdirectories.
+- **AffinityKey** — Worker affinity key to route to a specific beacon.
+- **TimeoutMs** — Timeout in milliseconds (default `300000`).
+
+### Outputs
+
+- **Files** — JSON array of file entries.
+- **StdOut** — Status message from the beacon.
+
+### Events
+
+- **Complete** — Fires after listing.
+- **Error** — Fires on failure.
+
+### Tips
+
+Use Content List Files to discover available content on a beacon before processing. Combine with **Split Execute** to iterate over the file list and read or transform each file.
+
+---
+
+## Content Create Folder
+
+Creates a folder in the content directory on a remote Content System beacon.
+
+### Settings
+
+- **Path** — Relative folder path to create.
+- **AffinityKey** — Worker affinity key to route to a specific beacon.
+- **TimeoutMs** — Timeout in milliseconds (default `300000`).
+
+### Outputs
+
+- **StdOut** — Status message from the beacon.
+
+### Events
+
+- **Complete** — Fires after the folder is created.
+- **Error** — Fires on failure.
+
+### Tips
+
+Use Content Create Folder before **Content Save File** to ensure the target directory structure exists. The operation is idempotent — creating a folder that already exists succeeds without error.

package/docs/features/tasks-data-transform.md

@@ -0,0 +1,233 @@
+# Data Transform Tasks
+
+Tasks for manipulating values, strings, templates, expressions, and tabular data within operation state.
+
+---
+
+## Set Values
+
+Sets one or more values in operation state at specified addresses. This is the primary card for initializing variables, copying data between state locations, and injecting literal values.
+
+### Settings
+
+- **Mappings** — An array of mapping objects. Each mapping has a `To` address (where to write) and either a `Value` (literal) or `From` address (copy from state).
+
+### Events
+
+- **Complete** — Fires after all mappings are applied.
+- **Error** — Fires if a mapping fails.
+
+### Tips
+
+Use Set Values at the start of a flow to initialize default values before branching logic. Mappings are applied in order, so later entries can reference values set by earlier ones within the same card.
+
+---
+
+## Replace String
+
+Replaces all occurrences of a search string within the input text.
+
+### Settings
+
+- **InputString** — The source text to search within. Supports Pict template expressions.
+- **SearchString** — The text or pattern to find.
+- **ReplaceString** — The replacement text (empty string to delete matches).
+- **UseRegex** — When `true`, treats SearchString as a regular expression.
+- **CaseSensitive** — Case-sensitive matching (default `true`).
+
+### Outputs
+
+- **ReplacedString** — The result after all replacements.
+- **ReplacementCount** — Number of replacements made.
+
+### Events
+
+- **ReplaceComplete** — Fires after replacement.
+- **Error** — Fires on failure (e.g. invalid regex).
+
+### Tips
+
+Enable UseRegex for advanced pattern matching such as removing HTML tags, normalizing whitespace, or extracting structured tokens. Use ReplacementCount to verify that expected substitutions occurred.
+
+---
+
+## String Appender
+
+Appends a string to an existing value at a specified state address. Useful for building up output incrementally across loop iterations or multiple steps.
+
+### Settings
+
+- **InputString** — The text to append. Supports Pict template expressions.
+- **OutputAddress** — State address of the string to append to.
+- **AppendNewline** — When `true`, appends a newline character after the input string.
+- **Separator** — String inserted between the existing content and new content. Overrides AppendNewline when set.
+
+### Outputs
+
+- **AppendedString** — The full accumulated string after appending.
+
+### Events
+
+- **Completed** — Fires after the append.
+
+### Tips
+
+Combine with **Split Execute** to build a report line by line. Use AppendNewline for log-style output or Separator for CSV-style concatenation with a custom delimiter.
+
+---
+
+## Template String
+
+Processes a Pict template string against the current operation state, resolving expressions like `{~D:Record.Name~}` into their runtime values.
+
+### Settings
+
+- **Template** — A Pict template string containing `{~D:...~}` expressions that reference state addresses.
+- **Destination** — State address to store the rendered result. If empty, the result is available at the default output.
+
+### Outputs
+
+- **Result** — The fully rendered template output.
+
+### Events
+
+- **Complete** — Fires after rendering.
+- **Error** — Fires if template parsing fails.
+
+### Tips
+
+Template String is the workhorse for building dynamic URLs, file paths, messages, and prompts. Any Pict template expression is supported, including conditionals and joins. Chain multiple Template String cards to build complex content from intermediate values.
+
+---
+
+## Expression Solver
+
+Evaluates a mathematical or logical expression using the Fable ExpressionParser and stores the result in state.
+
+### Settings
+
+- **Expression** — The expression to evaluate. Can reference state values by address.
+- **Destination** — State address to store the evaluation result.
+
+### Outputs
+
+- **Result** — The computed result as a string.
+
+### Events
+
+- **Complete** — Fires after evaluation.
+- **Error** — Fires on parse or evaluation failure.
+
+### Tips
+
+Use Expression Solver for arithmetic calculations, string length checks, or combining multiple state values into a computed result. The expression language supports standard math operators, comparisons, and common functions.
+
+---
+
+## Parse CSV
+
+Parses CSV text into an array of records (objects with field names as keys).
+
+### Settings
+
+- **SourceAddress** — State address containing the CSV text to parse.
+- **Delimiter** — Column delimiter character (default `,`).
+- **HasHeaders** — When `true`, the first row provides field names. When `false`, fields are indexed numerically.
+- **Destination** — State address to store the parsed records array.
+- **QuoteCharacter** — Character used to quote fields that contain the delimiter (default `"`).
+- **TrimFields** — Trim leading/trailing whitespace from field values.
+- **SkipEmptyLines** — Skip blank lines in the input.
+
+### Outputs
+
+- **Records** — Array of parsed row objects.
+- **ColumnCount** — Number of columns detected.
+- **Headers** — Array of header names from the first row.
+
+### Events
+
+- **Complete** — Fires after parsing.
+
+### Tips
+
+Chain **Read File** → **Parse CSV** → **CSV Transform** for a complete data import pipeline. Use TrimFields and SkipEmptyLines to handle messy real-world CSV exports cleanly.
+
+---
+
+## CSV Transform
+
+Transforms an array of parsed CSV records by applying field mappings, filters, and output field selection.
+
+### Settings
+
+- **SourceAddress** — State address of the records array to transform.
+- **Destination** — State address to store the transformed records.
+- **Delimiter** — Delimiter for re-serialization (default `,`).
+- **FieldMapping** — JSON array of mapping objects with `From`, `To`, and optional `Template` properties for field renaming and transformation.
+- **FilterExpression** — An expression to filter rows. Only rows where the expression evaluates to true are included.
+- **OutputFields** — JSON array of field names to include in the output. Omit to include all fields.
+
+### Outputs
+
+- **Records** — The transformed records array.
+
+### Events
+
+- **Complete** — Fires after transformation.
+
+### Tips
+
+Use FieldMapping to rename columns, compute derived fields with templates, or restructure data. Combine with FilterExpression to extract subsets of records in a single step.
+
+---
+
+## Histogram
+
+Computes a frequency distribution over a specific field in a dataset array.
+
+### Settings
+
+- **SourceAddress** — State address of the data array to analyze.
+- **Field** — Field name to compute frequencies for (default `score`).
+- **Bins** — Number of bins for numeric data (default `5`).
+- **Destination** — State address to store the statistics object.
+- **SortBy** — Sort frequency results by `count` or `key`.
+
+### Outputs
+
+- **Stats** — An object containing the frequency distribution, bin boundaries, and summary statistics.
+
+### Events
+
+- **Complete** — Fires after computation.
+
+### Tips
+
+Use Histogram after parsing or loading a dataset to get a quick statistical overview. Combine with **Template String** to generate a text-based report of the distribution.
+
+---
+
+## Comprehension Intersect
+
+Intersects two arrays by matching records on a common field, similar to an SQL inner join.
+
+### Settings
+
+- **SourceAddressA** — State address of the first array.
+- **SourceAddressB** — State address of the second array.
+- **MatchField** — Field name to match records on. Records from both arrays with the same value in this field are merged.
+- **Destination** — State address to store the intersected results.
+- **JoinType** — Join type (default `inner`). Inner join returns only records that match in both arrays.
+
+### Outputs
+
+- **Result** — Array of merged record objects.
+- **MatchCount** — Number of matched records.
+
+### Events
+
+- **Complete** — Fires after intersection.
+
+### Tips
+
+Use Comprehension Intersect to combine data from two different sources — for example, joining a list of user IDs from one API with user details from another. The matched records are merged into single objects containing fields from both sources.

package/docs/features/tasks-extension.md

@@ -0,0 +1,36 @@
+# Extension Tasks
+
+Tasks for dispatching work to remote Beacon worker nodes, providing low-level access to the distributed execution infrastructure.
+
+---
+
+## Beacon Dispatch
+
+Dispatches a work item to a remote Beacon worker node. The task pauses until the Beacon completes execution and reports results back.
+
+### Settings
+
+- **RemoteCapability** — Required capability on the Beacon (e.g. `Shell`, `FileSystem`, `LLM`).
+- **RemoteAction** — Specific action within the capability (e.g. `Execute`, `Read`).
+- **Command** — Shell command to execute on the Beacon (when using Shell capability).
+- **Parameters** — Command-line parameters for the shell command.
+- **AffinityKey** — Worker affinity routing key. Requests with the same key are routed to the same Beacon.
+- **TimeoutMs** — Work item timeout in milliseconds (default `300000`).
+- **InputData** — JSON data to pass to the Beacon worker.
+- **Destination** — State address to write results to on completion.
+
+### Outputs
+
+- **StdOut** — Standard output from the Beacon execution.
+- **Result** — Result data from the Beacon.
+- **ExitCode** — Exit code of the remote command.
+- **BeaconID** — ID of the Beacon that executed the work.
+
+### Events
+
+- **Complete** — Fires when the Beacon finishes.
+- **Error** — Fires on failure or timeout.
+
+### Tips
+
+Beacon Dispatch is the low-level card for sending arbitrary work to remote workers. Use AffinityKey to ensure related work items run on the same Beacon instance. For common patterns, prefer the specialized cards (LLM Chat Completion, Content Read File, etc.) which use Beacon Dispatch internally with appropriate defaults.