ultravisor 1.0.0 → 1.0.2

package/docs/features/configuration.md

@@ -0,0 +1,245 @@
+# Configuration
+
+Ultravisor uses a layered JSON configuration system. Configuration is
+gathered automatically at startup and merged into a single object.
+
+## Configuration File
+
+The primary configuration file is `.ultravisor.json`, located in the
+working directory. This file stores both runtime settings and persisted
+task/operation definitions.
+
+### Minimal Configuration
+
+```json
+{
+    "UltravisorAPIServerPort": 54321
+}
+```
+
+### Full Configuration
+
+```json
+{
+    "UltravisorAPIServerPort": 54321,
+    "UltravisorFileStorePath": "/var/data/ultravisor_datastore",
+    "UltravisorStagingRoot": "/var/data/ultravisor_staging",
+    "UltravisorTickIntervalMilliseconds": 60000,
+    "Tasks": {
+        "my-task": {
+            "GUIDTask": "my-task",
+            "Name": "My Task",
+            "Type": "Command",
+            "Command": "echo hello"
+        }
+    },
+    "Operations": {
+        "my-op": {
+            "GUIDOperation": "my-op",
+            "Name": "My Operation",
+            "Tasks": ["my-task"]
+        }
+    }
+}
+```
+
+## Configuration Options
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| `UltravisorAPIServerPort` | Number | `54321` | Port for the REST API server |
+| `UltravisorFileStorePath` | String | `${cwd}/dist/ultravisor_datastore` | Path for the output file store |
+| `UltravisorStagingRoot` | String | `${cwd}/dist/ultravisor_staging` | Root folder for per-operation staging directories |
+| `UltravisorTickIntervalMilliseconds` | Number | `60000` | Base tick interval in milliseconds |
+| `UltravisorCommandTimeoutMilliseconds` | Number | `300000` | Timeout for command task execution (5 minutes) |
+| `UltravisorCommandMaxBufferBytes` | Number | `10485760` | Max stdout/stderr buffer size for commands (10 MB) |
+| `Tasks` | Object | `{}` | Map of task GUIDs to task definitions |
+| `Operations` | Object | `{}` | Map of operation GUIDs to operation definitions |
+
+## Configuration Layering
+
+Configuration is resolved in order (later layers override earlier):
+
+1. **Default Program Configuration** -- hardcoded defaults in
+   `source/config/Ultravisor-Default-Command-Configuration.cjs`:
+
+   ```json
+   {
+       "UltravisorAPIServerPort": 54321,
+       "UltravisorFileStorePath": "${cwd}/dist/ultravisor_datastore",
+       "UltravisorStagingRoot": "${cwd}/dist/ultravisor_staging",
+       "UltravisorTickIntervalMilliseconds": 60000,
+       "UltravisorCommandTimeoutMilliseconds": 300000,
+       "UltravisorCommandMaxBufferBytes": 10485760
+   }
+   ```
+
+2. **`.ultravisor.json`** -- project-level file searched from the current
+   working directory upward through parent directories
+
+The merged result is available as `fable.ProgramConfiguration` within
+services.
+
+## Viewing Configuration Resolution
+
+Use the built-in `configuration` command to see how Ultravisor resolved
+its configuration:
+
+```bash
+ultravisor configuration
+```
+
+This shows each gather phase, what file was loaded, and what values came
+from where.
+
+## State Persistence
+
+When tasks or operations are created/updated via the CLI (`updatetask`)
+or the API (`POST /Task`, `POST /Operation`), the state service merges
+the changes back into the `.ultravisor.json` file.
+
+The merge logic:
+- The existing file is read
+- New or updated tasks/operations are merged (existing fields preserved,
+  new fields added)
+- The entire configuration is written back to the file
+
+This means the `.ultravisor.json` file serves as both configuration and
+persistent storage.
+
+## Example Configurations
+
+### Simple API monitoring
+
+```json
+{
+    "UltravisorAPIServerPort": 8080,
+    "Tasks": {
+        "health-check": {
+            "GUIDTask": "health-check",
+            "Name": "API Health Check",
+            "Type": "Request",
+            "URL": "https://api.example.com/health"
+        }
+    },
+    "Operations": {}
+}
+```
+
+### Multi-task data pipeline
+
+```json
+{
+    "UltravisorAPIServerPort": 54321,
+    "UltravisorFileStorePath": "/data/ultravisor",
+    "Tasks": {
+        "extract": {
+            "GUIDTask": "extract",
+            "Name": "Extract Data",
+            "Type": "Command",
+            "Command": "python3 /scripts/extract.py --output /tmp/extract.json"
+        },
+        "transform": {
+            "GUIDTask": "transform",
+            "Name": "Transform Data",
+            "Type": "Command",
+            "Command": "python3 /scripts/transform.py --input /tmp/extract.json --output /tmp/transformed.json"
+        },
+        "load": {
+            "GUIDTask": "load",
+            "Name": "Load Data",
+            "Type": "Command",
+            "Command": "python3 /scripts/load.py --input /tmp/transformed.json"
+        }
+    },
+    "Operations": {
+        "etl": {
+            "GUIDOperation": "etl",
+            "Name": "Full ETL Pipeline",
+            "Tasks": ["extract", "transform", "load"]
+        }
+    }
+}
+```
+
+### IoT sensor monitoring
+
+```json
+{
+    "UltravisorAPIServerPort": 9000,
+    "Tasks": {
+        "read-temp-living": {
+            "GUIDTask": "read-temp-living",
+            "Name": "Read Living Room Temperature",
+            "Type": "Request",
+            "URL": "http://192.168.1.100/api/temperature"
+        },
+        "read-temp-bedroom": {
+            "GUIDTask": "read-temp-bedroom",
+            "Name": "Read Bedroom Temperature",
+            "Type": "Request",
+            "URL": "http://192.168.1.101/api/temperature"
+        },
+        "read-temp-garage": {
+            "GUIDTask": "read-temp-garage",
+            "Name": "Read Garage Temperature",
+            "Type": "Request",
+            "URL": "http://192.168.1.102/api/temperature"
+        },
+        "log-readings": {
+            "GUIDTask": "log-readings",
+            "Name": "Log Temperature Readings",
+            "Type": "Command",
+            "Command": "python3 /scripts/log_temps.py"
+        }
+    },
+    "Operations": {
+        "temp-sweep": {
+            "GUIDOperation": "temp-sweep",
+            "Name": "Temperature Sweep",
+            "Tasks": [
+                "read-temp-living",
+                "read-temp-bedroom",
+                "read-temp-garage",
+                "log-readings"
+            ]
+        }
+    }
+}
+```
+
+### Media processing
+
+```json
+{
+    "UltravisorAPIServerPort": 54321,
+    "UltravisorFileStorePath": "/media/output",
+    "Tasks": {
+        "scan-input": {
+            "GUIDTask": "scan-input",
+            "Name": "Scan Input Directory",
+            "Type": "Command",
+            "Command": "ls -1 /media/input/*.avi > /tmp/pending_files.txt"
+        },
+        "transcode-batch": {
+            "GUIDTask": "transcode-batch",
+            "Name": "Transcode Video Batch",
+            "Type": "Command",
+            "Command": "bash /scripts/transcode_batch.sh /tmp/pending_files.txt /media/output"
+        },
+        "generate-thumbnails": {
+            "GUIDTask": "generate-thumbnails",
+            "Name": "Generate Thumbnails",
+            "Type": "Command",
+            "Command": "bash /scripts/thumbnails.sh /media/output"
+        }
+    },
+    "Operations": {
+        "media-ingest": {
+            "GUIDOperation": "media-ingest",
+            "Name": "Media Ingest Pipeline",
+            "Tasks": ["scan-input", "transcode-batch", "generate-thumbnails"]
+        }
+    }
+}
+```

package/docs/features/manifests.md

@@ -0,0 +1,177 @@
+# Manifests (Metaoutput)
+
+Manifests track the output from executing operations. Every operation
+execution produces a manifest that records timing, success/failure, task
+results and logs.
+
+## Manifest Model
+
+```json
+{
+    "GUIDOperation": "etl-pipeline",
+    "GUIDRun": "etl-pipeline-1707566400000",
+    "Name": "ETL Pipeline",
+    "StagingPath": "/var/data/ultravisor_staging/etl-pipeline",
+    "ManifestFilePath": "/var/data/ultravisor_staging/etl-pipeline/Manifest_etl-pipeline.json",
+    "StartTime": "2026-02-10T02:00:00.000Z",
+    "StopTime": "2026-02-10T02:00:12.500Z",
+    "Status": "Complete",
+    "Success": true,
+    "Summary": "Operation etl-pipeline Complete: 3 task(s) executed.",
+    "TaskResults": [
+        {
+            "GUIDTask": "pull-data",
+            "Name": "Pull Data from API",
+            "Type": "Request",
+            "StartTime": "2026-02-10T02:00:00.100Z",
+            "StopTime": "2026-02-10T02:00:03.200Z",
+            "Status": "Complete",
+            "Success": true,
+            "Output": "{\"records\": 150}",
+            "Log": ["..."]
+        },
+        {
+            "GUIDTask": "process-data",
+            "Name": "Process Downloaded Data",
+            "Type": "Command",
+            "StartTime": "2026-02-10T02:00:03.300Z",
+            "StopTime": "2026-02-10T02:00:10.100Z",
+            "Status": "Complete",
+            "Success": true,
+            "Output": "Processed 150 records\n",
+            "Log": ["..."]
+        },
+        {
+            "GUIDTask": "notify",
+            "Name": "Send Notification",
+            "Type": "Command",
+            "StartTime": "2026-02-10T02:00:10.200Z",
+            "StopTime": "2026-02-10T02:00:12.400Z",
+            "Status": "Complete",
+            "Success": true,
+            "Output": "",
+            "Log": ["..."]
+        }
+    ],
+    "Log": [
+        "Operation etl-pipeline started at 2026-02-10T02:00:00.000Z",
+        "Task pull-data completed with status: Complete",
+        "Task process-data completed with status: Complete",
+        "Task notify completed with status: Complete",
+        "Operation etl-pipeline Complete: 3 task(s) executed."
+    ]
+}
+```
+
+### Fields
+
+| Field | Description |
+|-------|-------------|
+| `GUIDOperation` | The operation that was executed |
+| `GUIDRun` | Unique ID for this specific execution run |
+| `Name` | Operation name |
+| `StartTime` | ISO timestamp when the operation started |
+| `StopTime` | ISO timestamp when the operation finished |
+| `Status` | `Running`, `Complete`, or `Error` |
+| `Success` | `true` if all tasks succeeded, `false` otherwise |
+| `Summary` | Human-readable one-line summary |
+| `StagingPath` | Absolute path to the operation's staging folder |
+| `ManifestFilePath` | Absolute path to the manifest JSON file on disk |
+| `TaskResults` | Array of task manifest entries (see below) |
+| `Log` | Array of log messages from the operation lifecycle |
+
+### Task Result Fields
+
+| Field | Description |
+|-------|-------------|
+| `GUIDTask` | The task that was executed |
+| `Name` | Task name |
+| `Type` | Task type (Command, Request, etc.) |
+| `StartTime` | ISO timestamp when the task started |
+| `StopTime` | ISO timestamp when the task finished |
+| `Status` | `Running`, `Complete`, `Error`, or `Unsupported` |
+| `Success` | `true` if the task succeeded |
+| `Output` | stdout or response body from the execution |
+| `Log` | Array of log messages from the task execution |
+
+## Manifest Lifecycle
+
+1. **Create** -- `createManifest()` initializes a new manifest with status
+   `Running` and assigns a `GUIDRun`
+2. **Add Results** -- as each task completes, `addTaskResult()` appends the
+   task result to `TaskResults`
+3. **Finalize** -- `finalizeManifest()` sets `StopTime`, calculates overall
+   `Success` (all tasks must succeed), sets `Status` and `Summary`, and
+   writes the manifest JSON to the operation's staging folder
+
+## Success Determination
+
+An operation manifest is marked as `Success: true` only if **every** task
+in the `TaskResults` array has `Success: true`. If any task has
+`Success: false`, the entire operation is marked as failed.
+
+## Storage
+
+Manifests are stored in two places:
+
+1. **In memory** -- for the duration of the server session, accessible via
+   the API. Restarting the server clears the in-memory manifest store.
+
+2. **On disk** -- when an operation completes, a
+   `Manifest_{GUIDOperation}.json` file is written to the operation's
+   staging folder. This provides a persistent record that survives
+   restarts. The file path is recorded in the manifest's
+   `ManifestFilePath` field.
+
+The on-disk manifest contains the complete manifest object including all
+task results, timing data and logs. It is written as pretty-printed JSON
+(4-space indentation).
+
+The staging folder location defaults to
+`./dist/ultravisor_staging/{GUIDOperation}/` and can be configured via
+`UltravisorStagingRoot` in `.ultravisor.json` (see
+[Configuration](configuration.md)).
+
+## Accessing Manifests
+
+### Via API
+
+```bash
+# List all manifests
+curl http://localhost:54321/Manifest
+
+# Get a specific manifest
+curl http://localhost:54321/Manifest/etl-pipeline-1707566400000
+```
+
+### Example: Checking if the last run succeeded
+
+```bash
+# Get the manifest list and check the last entry
+curl -s http://localhost:54321/Manifest | jq '.[-1].Success'
+```
+
+## Example: Partial Failure
+
+When one task fails in an operation, the manifest captures it:
+
+```json
+{
+    "GUIDOperation": "data-sync",
+    "Status": "Error",
+    "Success": false,
+    "Summary": "Operation data-sync Error: 2 task(s) executed.",
+    "TaskResults": [
+        {
+            "GUIDTask": "fetch-data",
+            "Status": "Complete",
+            "Success": true
+        },
+        {
+            "GUIDTask": "upload-data",
+            "Status": "Error",
+            "Success": false
+        }
+    ]
+}
+```