ultravisor 1.0.3 → 1.0.8
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/launch.json +11 -0
- package/.claude/ultravisor-dev-config.json +3 -0
- package/card-audit.json +1545 -0
- package/docs/_sidebar.md +17 -0
- package/docs/css/docuserve.css +73 -0
- package/docs/features/beacon-authentication.md +209 -0
- package/docs/features/beacon-providers.md +1299 -0
- package/docs/features/beacons.md +530 -0
- package/docs/features/capabilities.md +193 -0
- package/docs/features/llm-model-setup.md +505 -0
- package/docs/features/llm.md +262 -0
- package/docs/features/tasks-content-system.md +110 -0
- package/docs/features/tasks-data-transform.md +233 -0
- package/docs/features/tasks-extension.md +36 -0
- package/docs/features/tasks-file-system.md +203 -0
- package/docs/features/tasks-flow-control.md +125 -0
- package/docs/features/tasks-http-client.md +119 -0
- package/docs/features/tasks-llm.md +118 -0
- package/docs/features/tasks-meadow-api.md +163 -0
- package/docs/features/tasks-user-interaction.md +49 -0
- package/docs/features/tasks.md +14 -0
- package/docs/index.html +1 -1
- package/docs/retold-catalog.json +1 -1
- package/docs/retold-keyword-index.json +17374 -5
- package/operation-library/api-data-pipeline.json +143 -0
- package/operation-library/beacon-remote-command.json +72 -0
- package/operation-library/conditional-file-backup.json +143 -0
- package/operation-library/config-processor.json +136 -0
- package/operation-library/csv-data-analysis.json +228 -0
- package/operation-library/csv-to-json.json +160 -0
- package/operation-library/directory-inventory.json +128 -0
- package/operation-library/expression-calculator.json +114 -0
- package/operation-library/file-search-replace.json +175 -0
- package/operation-library/git-status-report.json +368 -0
- package/operation-library/health-check-ping.json +329 -0
- package/operation-library/json-config-merge.json +313 -0
- package/operation-library/llm-embedding-tool-use.json +314 -0
- package/operation-library/llm-multi-turn-analysis.json +322 -0
- package/operation-library/llm-summarize.json +264 -0
- package/operation-library/log-line-counter.json +185 -0
- package/operation-library/meadow-crud-lifecycle.json +264 -0
- package/operation-library/npm-project-validator.json +193 -0
- package/operation-library/rest-api-orchestrator.json +218 -0
- package/operation-library/shell-system-info.json +137 -0
- package/operation-library/simple-echo.json +95 -0
- package/operation-library/template-processor.json +299 -0
- package/operation-library/text-sanitizer.json +176 -0
- package/package.json +12 -8
- package/source/Ultravisor.cjs +30 -3
- package/source/beacon/Ultravisor-Beacon-CLI.cjs +143 -0
- package/source/beacon/Ultravisor-Beacon-CapabilityProvider.cjs +129 -0
- package/source/beacon/Ultravisor-Beacon-Client.cjs +568 -0
- package/source/beacon/Ultravisor-Beacon-Executor.cjs +500 -0
- package/source/beacon/Ultravisor-Beacon-ProviderRegistry.cjs +330 -0
- package/source/beacon/providers/Ultravisor-Beacon-Provider-FileSystem.cjs +331 -0
- package/source/beacon/providers/Ultravisor-Beacon-Provider-LLM.cjs +966 -0
- package/source/beacon/providers/Ultravisor-Beacon-Provider-Shell.cjs +95 -0
- package/source/cli/Ultravisor-CLIProgram.cjs +44 -22
- package/source/cli/commands/Ultravisor-Command-SingleOperation.cjs +29 -18
- package/source/cli/commands/Ultravisor-Command-SingleTask.cjs +62 -19
- package/source/cli/commands/Ultravisor-Command-UpdateTask.cjs +27 -15
- package/source/config/Ultravisor-Default-Command-Configuration.cjs +12 -3
- package/source/services/Ultravisor-Beacon-Coordinator.cjs +966 -0
- package/source/services/Ultravisor-ExecutionEngine.cjs +1093 -0
- package/source/services/Ultravisor-ExecutionManifest.cjs +629 -0
- package/source/services/Ultravisor-Hypervisor-State.cjs +270 -97
- package/source/services/Ultravisor-Hypervisor.cjs +185 -75
- package/source/services/Ultravisor-Schedule-Persistence-Base.cjs +45 -0
- package/source/services/Ultravisor-StateManager.cjs +253 -0
- package/source/services/Ultravisor-TaskTypeRegistry.cjs +213 -0
- package/source/services/persistence/Ultravisor-Schedule-Persistence-JSONFile.cjs +140 -0
- package/source/services/tasks/Ultravisor-BuiltIn-TaskConfigs.cjs +19 -0
- package/source/services/tasks/Ultravisor-TaskHelper-BeaconDispatch.cjs +107 -0
- package/source/services/tasks/Ultravisor-TaskType-Base.cjs +125 -0
- package/source/services/tasks/content-system/Ultravisor-TaskConfigs-ContentSystem.cjs +90 -0
- package/source/services/tasks/content-system/definitions/content-create-folder.json +24 -0
- package/source/services/tasks/content-system/definitions/content-list-files.json +27 -0
- package/source/services/tasks/content-system/definitions/content-read-file.json +26 -0
- package/source/services/tasks/content-system/definitions/content-save-file.json +26 -0
- package/source/services/tasks/data-transform/Ultravisor-TaskConfigs-DataTransform.cjs +577 -0
- package/source/services/tasks/data-transform/Ultravisor-TaskType-ReplaceString.cjs +80 -0
- package/source/services/tasks/data-transform/Ultravisor-TaskType-SetValues.cjs +74 -0
- package/source/services/tasks/data-transform/Ultravisor-TaskType-StringAppender.cjs +90 -0
- package/source/services/tasks/data-transform/definitions/comprehension-intersect.json +24 -0
- package/source/services/tasks/data-transform/definitions/csv-transform.json +24 -0
- package/source/services/tasks/data-transform/definitions/expression-solver.json +20 -0
- package/source/services/tasks/data-transform/definitions/histogram.json +23 -0
- package/source/services/tasks/data-transform/definitions/parse-csv.json +27 -0
- package/source/services/tasks/data-transform/definitions/replace-string.json +27 -0
- package/source/services/tasks/data-transform/definitions/set-values.json +17 -0
- package/source/services/tasks/data-transform/definitions/string-appender.json +22 -0
- package/source/services/tasks/data-transform/definitions/template-string.json +20 -0
- package/source/services/tasks/extension/Ultravisor-TaskConfigs-Extension.cjs +87 -0
- package/source/services/tasks/extension/definitions/beacon-dispatch.json +32 -0
- package/source/services/tasks/file-system/Ultravisor-TaskConfigs-FileSystem.cjs +509 -0
- package/source/services/tasks/file-system/Ultravisor-TaskType-ReadFile.cjs +96 -0
- package/source/services/tasks/file-system/Ultravisor-TaskType-ReadFileBuffered.cjs +123 -0
- package/source/services/tasks/file-system/Ultravisor-TaskType-WriteFile.cjs +106 -0
- package/source/services/tasks/file-system/definitions/copy-file.json +27 -0
- package/source/services/tasks/file-system/definitions/list-files.json +27 -0
- package/source/services/tasks/file-system/definitions/read-file-buffered.json +31 -0
- package/source/services/tasks/file-system/definitions/read-file.json +26 -0
- package/source/services/tasks/file-system/definitions/read-json.json +23 -0
- package/source/services/tasks/file-system/definitions/write-file.json +29 -0
- package/source/services/tasks/file-system/definitions/write-json.json +30 -0
- package/source/services/tasks/flow-control/Ultravisor-TaskConfigs-FlowControl.cjs +353 -0
- package/source/services/tasks/flow-control/Ultravisor-TaskType-IfConditional.cjs +125 -0
- package/source/services/tasks/flow-control/Ultravisor-TaskType-LaunchOperation.cjs +186 -0
- package/source/services/tasks/flow-control/Ultravisor-TaskType-SplitExecute.cjs +164 -0
- package/source/services/tasks/flow-control/definitions/if-conditional.json +25 -0
- package/source/services/tasks/flow-control/definitions/launch-operation.json +27 -0
- package/source/services/tasks/flow-control/definitions/split-execute.json +32 -0
- package/source/services/tasks/http-client/Ultravisor-TaskConfigs-HttpClient.cjs +281 -0
- package/source/services/tasks/http-client/definitions/get-json.json +26 -0
- package/source/services/tasks/http-client/definitions/get-text.json +26 -0
- package/source/services/tasks/http-client/definitions/rest-request.json +32 -0
- package/source/services/tasks/http-client/definitions/send-json.json +28 -0
- package/source/services/tasks/llm/Ultravisor-TaskConfigs-LLM.cjs +605 -0
- package/source/services/tasks/llm/definitions/llm-chat-completion.json +46 -0
- package/source/services/tasks/llm/definitions/llm-embedding.json +31 -0
- package/source/services/tasks/llm/definitions/llm-tool-use.json +42 -0
- package/source/services/tasks/meadow-api/Ultravisor-TaskConfigs-MeadowApi.cjs +341 -0
- package/source/services/tasks/meadow-api/definitions/meadow-count.json +26 -0
- package/source/services/tasks/meadow-api/definitions/meadow-create.json +26 -0
- package/source/services/tasks/meadow-api/definitions/meadow-delete.json +23 -0
- package/source/services/tasks/meadow-api/definitions/meadow-read.json +26 -0
- package/source/services/tasks/meadow-api/definitions/meadow-reads.json +29 -0
- package/source/services/tasks/meadow-api/definitions/meadow-update.json +26 -0
- package/source/services/tasks/shell/Ultravisor-TaskConfigs-Shell.cjs +63 -0
- package/source/services/tasks/shell/definitions/command.json +29 -0
- package/source/services/tasks/user-interaction/Ultravisor-TaskConfigs-UserInteraction.cjs +64 -0
- package/source/services/tasks/user-interaction/Ultravisor-TaskType-ErrorMessage.cjs +55 -0
- package/source/services/tasks/user-interaction/Ultravisor-TaskType-ValueInput.cjs +47 -0
- package/source/services/tasks/user-interaction/definitions/error-message.json +18 -0
- package/source/services/tasks/user-interaction/definitions/value-input.json +23 -0
- package/source/web_server/Ultravisor-API-Server.cjs +849 -124
- package/test/Ultravisor_browser_tests.js +2226 -0
- package/test/Ultravisor_operation_library_tests.js +894 -0
- package/test/Ultravisor_tests.js +4946 -5044
- package/test/workflows/test-content-concatenate-all.json +339 -0
- package/test/workflows/test-content-full-pipeline.json +159 -0
- package/test/workflows/test-content-list-files.json +76 -0
- package/test/workflows/test-content-save-and-read.json +106 -0
- package/webinterface/css/ultravisor-themes.css +668 -0
- package/webinterface/css/ultravisor.css +37 -14
- package/webinterface/docs/card-help/beacon-dispatch.md +30 -0
- package/webinterface/docs/card-help/command.md +27 -0
- package/webinterface/docs/card-help/comprehension-intersect.md +24 -0
- package/webinterface/docs/card-help/content-create-folder.md +22 -0
- package/webinterface/docs/card-help/content-list-files.md +25 -0
- package/webinterface/docs/card-help/content-read-file.md +24 -0
- package/webinterface/docs/card-help/content-save-file.md +24 -0
- package/webinterface/docs/card-help/copy-file.md +25 -0
- package/webinterface/docs/card-help/csv-transform.md +24 -0
- package/webinterface/docs/card-help/end.md +11 -0
- package/webinterface/docs/card-help/error-message.md +16 -0
- package/webinterface/docs/card-help/expression-solver.md +21 -0
- package/webinterface/docs/card-help/get-json.md +24 -0
- package/webinterface/docs/card-help/get-text.md +24 -0
- package/webinterface/docs/card-help/histogram.md +23 -0
- package/webinterface/docs/card-help/if-conditional.md +23 -0
- package/webinterface/docs/card-help/launch-operation.md +25 -0
- package/webinterface/docs/card-help/list-files.md +25 -0
- package/webinterface/docs/card-help/llm-chat-completion.md +43 -0
- package/webinterface/docs/card-help/llm-embedding.md +24 -0
- package/webinterface/docs/card-help/llm-tool-use.md +39 -0
- package/webinterface/docs/card-help/meadow-count.md +24 -0
- package/webinterface/docs/card-help/meadow-create.md +24 -0
- package/webinterface/docs/card-help/meadow-delete.md +19 -0
- package/webinterface/docs/card-help/meadow-read.md +24 -0
- package/webinterface/docs/card-help/meadow-reads.md +27 -0
- package/webinterface/docs/card-help/meadow-update.md +24 -0
- package/webinterface/docs/card-help/parse-csv.md +27 -0
- package/webinterface/docs/card-help/read-file-buffered.md +29 -0
- package/webinterface/docs/card-help/read-file.md +24 -0
- package/webinterface/docs/card-help/read-json.md +21 -0
- package/webinterface/docs/card-help/replace-string.md +25 -0
- package/webinterface/docs/card-help/rest-request.md +30 -0
- package/webinterface/docs/card-help/send-json.md +26 -0
- package/webinterface/docs/card-help/set-values.md +16 -0
- package/webinterface/docs/card-help/split-execute.md +35 -0
- package/webinterface/docs/card-help/start.md +11 -0
- package/webinterface/docs/card-help/string-appender.md +22 -0
- package/webinterface/docs/card-help/template-string.md +21 -0
- package/webinterface/docs/card-help/value-input.md +24 -0
- package/webinterface/docs/card-help/write-file.md +27 -0
- package/webinterface/docs/card-help/write-json.md +28 -0
- package/webinterface/html/index.html +5 -0
- package/webinterface/package.json +13 -4
- package/webinterface/source/Pict-Application-Ultravisor.js +564 -70
- package/webinterface/source/Ultravisor-TimingUtils.js +99 -0
- package/webinterface/source/card-help-content.js +46 -0
- package/webinterface/source/cards/Ultravisor-BuiltIn-CardConfigs.js +174 -0
- package/webinterface/source/cards/Ultravisor-CardConfigGenerator.js +431 -0
- package/webinterface/source/data/ExampleFlow-CSVPipeline.js +231 -0
- package/webinterface/source/data/ExampleFlow-FileProcessor.js +315 -0
- package/webinterface/source/data/ExampleFlow-MeadowPipeline.js +328 -0
- package/webinterface/source/panels/Ultravisor-CardSettingsPanel.js +902 -0
- package/webinterface/source/providers/PictRouter-Ultravisor-Configuration.json +20 -8
- package/webinterface/source/views/PictView-Ultravisor-BeaconList.js +817 -0
- package/webinterface/source/views/PictView-Ultravisor-BottomBar.js +5 -5
- package/webinterface/source/views/PictView-Ultravisor-Dashboard.js +25 -25
- package/webinterface/source/views/PictView-Ultravisor-Documentation.js +856 -0
- package/webinterface/source/views/PictView-Ultravisor-FlowEditor.js +777 -0
- package/webinterface/source/views/PictView-Ultravisor-ManifestList.js +321 -58
- package/webinterface/source/views/PictView-Ultravisor-OperationEdit.js +36 -91
- package/webinterface/source/views/PictView-Ultravisor-OperationList.js +388 -16
- package/webinterface/source/views/PictView-Ultravisor-PendingInput.js +314 -0
- package/webinterface/source/views/PictView-Ultravisor-Schedule.js +521 -65
- package/webinterface/source/views/PictView-Ultravisor-TimingView.js +333 -71
- package/webinterface/source/views/PictView-Ultravisor-TopBar.js +257 -21
- package/webinterface/theme-sampler.html +645 -0
- package/debug/Harness.js +0 -6
- package/source/services/Ultravisor-Operation-Manifest.cjs +0 -160
- package/source/services/Ultravisor-Operation.cjs +0 -200
- package/source/services/Ultravisor-Task.cjs +0 -349
- package/source/services/events/Ultravisor-Hypervisor-Event-Solver.cjs +0 -11
- package/source/services/tasks/Ultravisor-Task-Base.cjs +0 -264
- package/source/services/tasks/Ultravisor-Task-CollectValues.cjs +0 -188
- package/source/services/tasks/Ultravisor-Task-Command.cjs +0 -65
- package/source/services/tasks/Ultravisor-Task-CommandEach.cjs +0 -190
- package/source/services/tasks/Ultravisor-Task-Conditional.cjs +0 -104
- package/source/services/tasks/Ultravisor-Task-DateWindow.cjs +0 -72
- package/source/services/tasks/Ultravisor-Task-GeneratePagedOperation.cjs +0 -336
- package/source/services/tasks/Ultravisor-Task-LaunchOperation.cjs +0 -143
- package/source/services/tasks/Ultravisor-Task-LaunchTask.cjs +0 -146
- package/source/services/tasks/Ultravisor-Task-LineMatch.cjs +0 -158
- package/source/services/tasks/Ultravisor-Task-Request.cjs +0 -56
- package/source/services/tasks/Ultravisor-Task-Solver.cjs +0 -89
- package/source/services/tasks/Ultravisor-Task-TemplateString.cjs +0 -93
- package/source/services/tasks/rest/Ultravisor-Task-GetBinary.cjs +0 -127
- package/source/services/tasks/rest/Ultravisor-Task-GetJSON.cjs +0 -119
- package/source/services/tasks/rest/Ultravisor-Task-GetText.cjs +0 -109
- package/source/services/tasks/rest/Ultravisor-Task-GetXML.cjs +0 -112
- package/source/services/tasks/rest/Ultravisor-Task-RestRequest.cjs +0 -499
- package/source/services/tasks/rest/Ultravisor-Task-SendJSON.cjs +0 -150
- package/source/services/tasks/stagingfiles/Ultravisor-Task-CopyFile.cjs +0 -110
- package/source/services/tasks/stagingfiles/Ultravisor-Task-ListFiles.cjs +0 -89
- package/source/services/tasks/stagingfiles/Ultravisor-Task-ReadBinary.cjs +0 -87
- package/source/services/tasks/stagingfiles/Ultravisor-Task-ReadJSON.cjs +0 -67
- package/source/services/tasks/stagingfiles/Ultravisor-Task-ReadText.cjs +0 -66
- package/source/services/tasks/stagingfiles/Ultravisor-Task-ReadXML.cjs +0 -69
- package/source/services/tasks/stagingfiles/Ultravisor-Task-WriteBinary.cjs +0 -95
- package/source/services/tasks/stagingfiles/Ultravisor-Task-WriteJSON.cjs +0 -96
- package/source/services/tasks/stagingfiles/Ultravisor-Task-WriteText.cjs +0 -99
- package/source/services/tasks/stagingfiles/Ultravisor-Task-WriteXML.cjs +0 -102
- package/webinterface/source/views/PictView-Ultravisor-TaskEdit.js +0 -220
- package/webinterface/source/views/PictView-Ultravisor-TaskList.js +0 -248
|
@@ -0,0 +1,1299 @@
|
|
|
1
|
+
# Building Beacon Providers
|
|
2
|
+
|
|
3
|
+
This guide walks through building custom Beacon providers from scratch,
|
|
4
|
+
with two full examples: a headless Chrome browser automation provider and
|
|
5
|
+
an FFmpeg video transcoding provider. By the end, you will know how to
|
|
6
|
+
create a provider, configure it, wire it into a Beacon, and dispatch work
|
|
7
|
+
to it from an operation manifest.
|
|
8
|
+
|
|
9
|
+
For reference-level documentation on the provider interface, built-in
|
|
10
|
+
providers, and progress reporting, see [Beacons](beacons.md).
|
|
11
|
+
|
|
12
|
+
## Quick-Start: Running a Beacon
|
|
13
|
+
|
|
14
|
+
Before writing custom providers, let's make sure you can run a basic Beacon
|
|
15
|
+
against an Ultravisor server.
|
|
16
|
+
|
|
17
|
+
### 1. Start the Ultravisor Server
|
|
18
|
+
|
|
19
|
+
```bash
|
|
20
|
+
node source/Ultravisor-Server.cjs --config ./my-config.json
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
### 2. Start a Shell-Only Beacon
|
|
24
|
+
|
|
25
|
+
The simplest Beacon uses only the built-in Shell provider:
|
|
26
|
+
|
|
27
|
+
```bash
|
|
28
|
+
node source/beacon/Ultravisor-Beacon-CLI.cjs \
|
|
29
|
+
--server http://localhost:54321 \
|
|
30
|
+
--name "shell-worker" \
|
|
31
|
+
--capabilities Shell
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
### 3. Dispatch Work to It
|
|
35
|
+
|
|
36
|
+
Create a manifest with a `beacon-dispatch` node:
|
|
37
|
+
|
|
38
|
+
```json
|
|
39
|
+
{
|
|
40
|
+
"Scope": "Operation",
|
|
41
|
+
"Nodes":
|
|
42
|
+
{
|
|
43
|
+
"start": { "Type": "start" },
|
|
44
|
+
"run-command":
|
|
45
|
+
{
|
|
46
|
+
"Type": "beacon-dispatch",
|
|
47
|
+
"Settings":
|
|
48
|
+
{
|
|
49
|
+
"Capability": "Shell",
|
|
50
|
+
"Action": "Execute",
|
|
51
|
+
"Command": "echo",
|
|
52
|
+
"Parameters": "hello from the Beacon"
|
|
53
|
+
}
|
|
54
|
+
},
|
|
55
|
+
"end": { "Type": "end" }
|
|
56
|
+
},
|
|
57
|
+
"Edges":
|
|
58
|
+
{
|
|
59
|
+
"start": [{ "Target": "run-command", "Event": "Complete" }],
|
|
60
|
+
"run-command": [{ "Target": "end", "Event": "Complete" }]
|
|
61
|
+
}
|
|
62
|
+
}
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
Start an operation from this manifest, and the Beacon picks it up, runs
|
|
66
|
+
`echo hello from the Beacon`, and reports back the output. The graph
|
|
67
|
+
resumes and completes.
|
|
68
|
+
|
|
69
|
+
### 4. Use a Config File
|
|
70
|
+
|
|
71
|
+
For anything beyond quick tests, use a `.ultravisor-beacon.json` file:
|
|
72
|
+
|
|
73
|
+
```json
|
|
74
|
+
{
|
|
75
|
+
"ServerURL": "http://localhost:54321",
|
|
76
|
+
"Name": "dev-worker",
|
|
77
|
+
"Capabilities": ["Shell", "FileSystem"],
|
|
78
|
+
"MaxConcurrent": 2,
|
|
79
|
+
"PollIntervalMs": 3000,
|
|
80
|
+
"StagingPath": "/tmp/beacon-staging"
|
|
81
|
+
}
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
Then start the Beacon with no arguments (it reads from the config file):
|
|
85
|
+
|
|
86
|
+
```bash
|
|
87
|
+
cd /path/to/config && node /path/to/Ultravisor-Beacon-CLI.cjs
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
---
|
|
91
|
+
|
|
92
|
+
## Anatomy of a Provider
|
|
93
|
+
|
|
94
|
+
Every provider is a plain JavaScript class that extends
|
|
95
|
+
`UltravisorBeaconCapabilityProvider`. There is no Fable dependency — keep
|
|
96
|
+
providers lightweight and self-contained.
|
|
97
|
+
|
|
98
|
+
```
|
|
99
|
+
┌──────────────────────────────────────────┐
|
|
100
|
+
│ CapabilityProvider │
|
|
101
|
+
│ │
|
|
102
|
+
│ Name "MyProvider" │
|
|
103
|
+
│ Capability "SomeCapability" │
|
|
104
|
+
│ │
|
|
105
|
+
│ get actions() → { ActionA, ActionB } │
|
|
106
|
+
│ │
|
|
107
|
+
│ initialize(fCallback) ← optional │
|
|
108
|
+
│ shutdown(fCallback) ← optional │
|
|
109
|
+
│ │
|
|
110
|
+
│ execute(pAction, pWorkItem, pContext, │
|
|
111
|
+
│ fCallback, fReportProgress) │
|
|
112
|
+
│ │
|
|
113
|
+
│ _ProviderConfig ← per-provider cfg │
|
|
114
|
+
└──────────────────────────────────────────┘
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
### The `execute` Contract
|
|
118
|
+
|
|
119
|
+
```js
|
|
120
|
+
execute(pAction, pWorkItem, pContext, fCallback, fReportProgress)
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
| Parameter | Description |
|
|
124
|
+
|-----------|-------------|
|
|
125
|
+
| `pAction` | The action string (e.g. `'Transcode'`, `'Screenshot'`) |
|
|
126
|
+
| `pWorkItem` | Work item record. Use `pWorkItem.Settings` for task-specific parameters |
|
|
127
|
+
| `pContext` | Execution context: `{ StagingPath, NodeHash, ... }` |
|
|
128
|
+
| `fCallback(pError, pResult)` | Completion callback. `pResult` must include `{ Outputs: { StdOut, ExitCode, Result }, Log: [] }` |
|
|
129
|
+
| `fReportProgress(pData)` | Optional progress callback: `{ Percent, Message, Step, TotalSteps, Log }` |
|
|
130
|
+
|
|
131
|
+
The `fCallback` result shape:
|
|
132
|
+
|
|
133
|
+
```js
|
|
134
|
+
fCallback(null, {
|
|
135
|
+
Outputs: {
|
|
136
|
+
StdOut: 'Human-readable summary of what happened',
|
|
137
|
+
ExitCode: 0, // 0 = success, non-zero = failure
|
|
138
|
+
Result: '...' // Structured output data (string or JSON string)
|
|
139
|
+
},
|
|
140
|
+
Log: [
|
|
141
|
+
'Step 1: did X',
|
|
142
|
+
'Step 2: did Y'
|
|
143
|
+
]
|
|
144
|
+
});
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
---
|
|
148
|
+
|
|
149
|
+
## Example 1: Headless Chrome Provider
|
|
150
|
+
|
|
151
|
+
This provider launches a headless Chrome browser to take screenshots,
|
|
152
|
+
generate PDFs, or scrape content from web pages. It uses Puppeteer under
|
|
153
|
+
the hood.
|
|
154
|
+
|
|
155
|
+
### Provider Implementation
|
|
156
|
+
|
|
157
|
+
```js
|
|
158
|
+
// headless-chrome-provider.cjs
|
|
159
|
+
|
|
160
|
+
const libBeaconCapabilityProvider = require('ultravisor').BeaconCapabilityProvider;
|
|
161
|
+
|
|
162
|
+
let libPuppeteer;
|
|
163
|
+
try { libPuppeteer = require('puppeteer'); }
|
|
164
|
+
catch (pErr) { libPuppeteer = null; }
|
|
165
|
+
|
|
166
|
+
class HeadlessChromeProvider extends libBeaconCapabilityProvider
|
|
167
|
+
{
|
|
168
|
+
constructor(pProviderConfig)
|
|
169
|
+
{
|
|
170
|
+
super(pProviderConfig);
|
|
171
|
+
|
|
172
|
+
this.Name = 'HeadlessChrome';
|
|
173
|
+
this.Capability = 'Browser';
|
|
174
|
+
|
|
175
|
+
// Provider config with sensible defaults
|
|
176
|
+
this._NavigationTimeoutMs = this._ProviderConfig.NavigationTimeoutMs || 30000;
|
|
177
|
+
this._DefaultViewport = this._ProviderConfig.Viewport || { width: 1280, height: 800 };
|
|
178
|
+
this._AllowedDomains = this._ProviderConfig.AllowedDomains || [];
|
|
179
|
+
|
|
180
|
+
this._Browser = null;
|
|
181
|
+
}
|
|
182
|
+
|
|
183
|
+
get actions()
|
|
184
|
+
{
|
|
185
|
+
return {
|
|
186
|
+
'Screenshot':
|
|
187
|
+
{
|
|
188
|
+
Description: 'Navigate to a URL and capture a screenshot as a PNG file.',
|
|
189
|
+
SettingsSchema:
|
|
190
|
+
[
|
|
191
|
+
{ Name: 'URL', DataType: 'String', Required: true, Description: 'The page URL to screenshot' },
|
|
192
|
+
{ Name: 'OutputFile', DataType: 'String', Required: false, Description: 'Output filename (default: screenshot.png)' },
|
|
193
|
+
{ Name: 'FullPage', DataType: 'Boolean', Required: false, Description: 'Capture the full scrollable page (default: false)' },
|
|
194
|
+
{ Name: 'WaitForSelector', DataType: 'String', Required: false, Description: 'CSS selector to wait for before capture' }
|
|
195
|
+
]
|
|
196
|
+
},
|
|
197
|
+
'PDF':
|
|
198
|
+
{
|
|
199
|
+
Description: 'Navigate to a URL and render the page as a PDF document.',
|
|
200
|
+
SettingsSchema:
|
|
201
|
+
[
|
|
202
|
+
{ Name: 'URL', DataType: 'String', Required: true, Description: 'The page URL to render' },
|
|
203
|
+
{ Name: 'OutputFile', DataType: 'String', Required: false, Description: 'Output filename (default: page.pdf)' },
|
|
204
|
+
{ Name: 'Format', DataType: 'String', Required: false, Description: 'Page format: Letter, A4, etc. (default: Letter)' }
|
|
205
|
+
]
|
|
206
|
+
},
|
|
207
|
+
'Scrape':
|
|
208
|
+
{
|
|
209
|
+
Description: 'Navigate to a URL and extract text content or element data.',
|
|
210
|
+
SettingsSchema:
|
|
211
|
+
[
|
|
212
|
+
{ Name: 'URL', DataType: 'String', Required: true, Description: 'The page URL to scrape' },
|
|
213
|
+
{ Name: 'Selector', DataType: 'String', Required: false, Description: 'CSS selector to extract (default: body)' },
|
|
214
|
+
{ Name: 'ExtractAttribute', DataType: 'String', Required: false, Description: 'Element attribute to extract instead of text (e.g. href, src)' }
|
|
215
|
+
]
|
|
216
|
+
}
|
|
217
|
+
};
|
|
218
|
+
}
|
|
219
|
+
|
|
220
|
+
// -------------------------------------------------------------------
|
|
221
|
+
// Lifecycle
|
|
222
|
+
// -------------------------------------------------------------------
|
|
223
|
+
|
|
224
|
+
initialize(fCallback)
|
|
225
|
+
{
|
|
226
|
+
if (!libPuppeteer)
|
|
227
|
+
{
|
|
228
|
+
return fCallback(new Error('Puppeteer is not installed. Run: npm install puppeteer'));
|
|
229
|
+
}
|
|
230
|
+
|
|
231
|
+
let tmpThis = this;
|
|
232
|
+
|
|
233
|
+
libPuppeteer.launch({
|
|
234
|
+
headless: 'new',
|
|
235
|
+
args: ['--no-sandbox', '--disable-setuid-sandbox']
|
|
236
|
+
})
|
|
237
|
+
.then(function (pBrowser)
|
|
238
|
+
{
|
|
239
|
+
tmpThis._Browser = pBrowser;
|
|
240
|
+
return fCallback(null);
|
|
241
|
+
})
|
|
242
|
+
.catch(function (pError)
|
|
243
|
+
{
|
|
244
|
+
return fCallback(new Error('Failed to launch Chrome: ' + pError.message));
|
|
245
|
+
});
|
|
246
|
+
}
|
|
247
|
+
|
|
248
|
+
shutdown(fCallback)
|
|
249
|
+
{
|
|
250
|
+
if (this._Browser)
|
|
251
|
+
{
|
|
252
|
+
this._Browser.close()
|
|
253
|
+
.then(function () { return fCallback(null); })
|
|
254
|
+
.catch(function () { return fCallback(null); });
|
|
255
|
+
}
|
|
256
|
+
else
|
|
257
|
+
{
|
|
258
|
+
return fCallback(null);
|
|
259
|
+
}
|
|
260
|
+
}
|
|
261
|
+
|
|
262
|
+
// -------------------------------------------------------------------
|
|
263
|
+
// Action dispatcher
|
|
264
|
+
// -------------------------------------------------------------------
|
|
265
|
+
|
|
266
|
+
execute(pAction, pWorkItem, pContext, fCallback, fReportProgress)
|
|
267
|
+
{
|
|
268
|
+
switch (pAction)
|
|
269
|
+
{
|
|
270
|
+
case 'Screenshot':
|
|
271
|
+
return this._executeScreenshot(pWorkItem, pContext, fCallback, fReportProgress);
|
|
272
|
+
case 'PDF':
|
|
273
|
+
return this._executePDF(pWorkItem, pContext, fCallback, fReportProgress);
|
|
274
|
+
case 'Scrape':
|
|
275
|
+
return this._executeScrape(pWorkItem, pContext, fCallback, fReportProgress);
|
|
276
|
+
default:
|
|
277
|
+
return fCallback(null, {
|
|
278
|
+
Outputs: { StdOut: 'Unknown action: ' + pAction, ExitCode: -1, Result: '' },
|
|
279
|
+
Log: ['HeadlessChrome: unsupported action [' + pAction + '].']
|
|
280
|
+
});
|
|
281
|
+
}
|
|
282
|
+
}
|
|
283
|
+
|
|
284
|
+
// -------------------------------------------------------------------
|
|
285
|
+
// Domain guard
|
|
286
|
+
// -------------------------------------------------------------------
|
|
287
|
+
|
|
288
|
+
_isDomainAllowed(pURL)
|
|
289
|
+
{
|
|
290
|
+
if (this._AllowedDomains.length === 0) return true;
|
|
291
|
+
try
|
|
292
|
+
{
|
|
293
|
+
let tmpHostname = new URL(pURL).hostname;
|
|
294
|
+
for (let i = 0; i < this._AllowedDomains.length; i++)
|
|
295
|
+
{
|
|
296
|
+
if (tmpHostname === this._AllowedDomains[i]) return true;
|
|
297
|
+
if (tmpHostname.endsWith('.' + this._AllowedDomains[i])) return true;
|
|
298
|
+
}
|
|
299
|
+
return false;
|
|
300
|
+
}
|
|
301
|
+
catch (pErr)
|
|
302
|
+
{
|
|
303
|
+
return false;
|
|
304
|
+
}
|
|
305
|
+
}
|
|
306
|
+
|
|
307
|
+
// -------------------------------------------------------------------
|
|
308
|
+
// Actions
|
|
309
|
+
// -------------------------------------------------------------------
|
|
310
|
+
|
|
311
|
+
_executeScreenshot(pWorkItem, pContext, fCallback, fReportProgress)
|
|
312
|
+
{
|
|
313
|
+
let tmpSettings = pWorkItem.Settings || {};
|
|
314
|
+
let tmpURL = tmpSettings.URL;
|
|
315
|
+
let tmpOutputFile = tmpSettings.OutputFile || 'screenshot.png';
|
|
316
|
+
let tmpFullPage = tmpSettings.FullPage || false;
|
|
317
|
+
let tmpWaitFor = tmpSettings.WaitForSelector || null;
|
|
318
|
+
let tmpThis = this;
|
|
319
|
+
|
|
320
|
+
if (!tmpURL)
|
|
321
|
+
{
|
|
322
|
+
return fCallback(null, {
|
|
323
|
+
Outputs: { StdOut: 'No URL specified.', ExitCode: -1, Result: '' },
|
|
324
|
+
Log: ['HeadlessChrome Screenshot: no URL specified.']
|
|
325
|
+
});
|
|
326
|
+
}
|
|
327
|
+
|
|
328
|
+
if (!this._isDomainAllowed(tmpURL))
|
|
329
|
+
{
|
|
330
|
+
return fCallback(null, {
|
|
331
|
+
Outputs: { StdOut: 'Domain not in AllowedDomains: ' + tmpURL, ExitCode: -1, Result: '' },
|
|
332
|
+
Log: ['HeadlessChrome Screenshot: domain blocked by AllowedDomains.']
|
|
333
|
+
});
|
|
334
|
+
}
|
|
335
|
+
|
|
336
|
+
// Resolve output path relative to staging
|
|
337
|
+
let libPath = require('path');
|
|
338
|
+
let tmpOutputPath = libPath.isAbsolute(tmpOutputFile)
|
|
339
|
+
? tmpOutputFile
|
|
340
|
+
: libPath.resolve(pContext.StagingPath || process.cwd(), tmpOutputFile);
|
|
341
|
+
|
|
342
|
+
let tmpPage;
|
|
343
|
+
let tmpLog = [];
|
|
344
|
+
|
|
345
|
+
this._Browser.newPage()
|
|
346
|
+
.then(function (pPage)
|
|
347
|
+
{
|
|
348
|
+
tmpPage = pPage;
|
|
349
|
+
tmpLog.push('Opened new browser tab.');
|
|
350
|
+
fReportProgress({ Percent: 10, Message: 'Navigating to ' + tmpURL });
|
|
351
|
+
|
|
352
|
+
return tmpPage.setViewport(tmpThis._DefaultViewport);
|
|
353
|
+
})
|
|
354
|
+
.then(function ()
|
|
355
|
+
{
|
|
356
|
+
return tmpPage.goto(tmpURL, {
|
|
357
|
+
waitUntil: 'networkidle2',
|
|
358
|
+
timeout: tmpThis._NavigationTimeoutMs
|
|
359
|
+
});
|
|
360
|
+
})
|
|
361
|
+
.then(function ()
|
|
362
|
+
{
|
|
363
|
+
tmpLog.push('Navigation complete: ' + tmpURL);
|
|
364
|
+
fReportProgress({ Percent: 50, Message: 'Page loaded' });
|
|
365
|
+
|
|
366
|
+
if (tmpWaitFor)
|
|
367
|
+
{
|
|
368
|
+
tmpLog.push('Waiting for selector: ' + tmpWaitFor);
|
|
369
|
+
return tmpPage.waitForSelector(tmpWaitFor, { timeout: 10000 });
|
|
370
|
+
}
|
|
371
|
+
})
|
|
372
|
+
.then(function ()
|
|
373
|
+
{
|
|
374
|
+
fReportProgress({ Percent: 70, Message: 'Capturing screenshot' });
|
|
375
|
+
|
|
376
|
+
return tmpPage.screenshot({
|
|
377
|
+
path: tmpOutputPath,
|
|
378
|
+
fullPage: tmpFullPage
|
|
379
|
+
});
|
|
380
|
+
})
|
|
381
|
+
.then(function ()
|
|
382
|
+
{
|
|
383
|
+
tmpLog.push('Screenshot saved: ' + tmpOutputPath);
|
|
384
|
+
fReportProgress({ Percent: 100, Message: 'Done' });
|
|
385
|
+
|
|
386
|
+
return tmpPage.close();
|
|
387
|
+
})
|
|
388
|
+
.then(function ()
|
|
389
|
+
{
|
|
390
|
+
return fCallback(null, {
|
|
391
|
+
Outputs: {
|
|
392
|
+
StdOut: 'Screenshot saved to ' + tmpOutputPath,
|
|
393
|
+
ExitCode: 0,
|
|
394
|
+
Result: tmpOutputPath
|
|
395
|
+
},
|
|
396
|
+
Log: tmpLog
|
|
397
|
+
});
|
|
398
|
+
})
|
|
399
|
+
.catch(function (pError)
|
|
400
|
+
{
|
|
401
|
+
tmpLog.push('Error: ' + pError.message);
|
|
402
|
+
if (tmpPage) { tmpPage.close().catch(function () {}); }
|
|
403
|
+
return fCallback(null, {
|
|
404
|
+
Outputs: { StdOut: 'Screenshot failed: ' + pError.message, ExitCode: 1, Result: '' },
|
|
405
|
+
Log: tmpLog
|
|
406
|
+
});
|
|
407
|
+
});
|
|
408
|
+
}
|
|
409
|
+
|
|
410
|
+
_executePDF(pWorkItem, pContext, fCallback, fReportProgress)
|
|
411
|
+
{
|
|
412
|
+
let tmpSettings = pWorkItem.Settings || {};
|
|
413
|
+
let tmpURL = tmpSettings.URL;
|
|
414
|
+
let tmpOutputFile = tmpSettings.OutputFile || 'page.pdf';
|
|
415
|
+
let tmpFormat = tmpSettings.Format || 'Letter';
|
|
416
|
+
let tmpThis = this;
|
|
417
|
+
|
|
418
|
+
if (!tmpURL)
|
|
419
|
+
{
|
|
420
|
+
return fCallback(null, {
|
|
421
|
+
Outputs: { StdOut: 'No URL specified.', ExitCode: -1, Result: '' },
|
|
422
|
+
Log: ['HeadlessChrome PDF: no URL specified.']
|
|
423
|
+
});
|
|
424
|
+
}
|
|
425
|
+
|
|
426
|
+
if (!this._isDomainAllowed(tmpURL))
|
|
427
|
+
{
|
|
428
|
+
return fCallback(null, {
|
|
429
|
+
Outputs: { StdOut: 'Domain not in AllowedDomains: ' + tmpURL, ExitCode: -1, Result: '' },
|
|
430
|
+
Log: ['HeadlessChrome PDF: domain blocked by AllowedDomains.']
|
|
431
|
+
});
|
|
432
|
+
}
|
|
433
|
+
|
|
434
|
+
let libPath = require('path');
|
|
435
|
+
let tmpOutputPath = libPath.isAbsolute(tmpOutputFile)
|
|
436
|
+
? tmpOutputFile
|
|
437
|
+
: libPath.resolve(pContext.StagingPath || process.cwd(), tmpOutputFile);
|
|
438
|
+
|
|
439
|
+
let tmpPage;
|
|
440
|
+
let tmpLog = [];
|
|
441
|
+
|
|
442
|
+
this._Browser.newPage()
|
|
443
|
+
.then(function (pPage)
|
|
444
|
+
{
|
|
445
|
+
tmpPage = pPage;
|
|
446
|
+
fReportProgress({ Percent: 10, Message: 'Navigating to ' + tmpURL });
|
|
447
|
+
return tmpPage.goto(tmpURL, {
|
|
448
|
+
waitUntil: 'networkidle2',
|
|
449
|
+
timeout: tmpThis._NavigationTimeoutMs
|
|
450
|
+
});
|
|
451
|
+
})
|
|
452
|
+
.then(function ()
|
|
453
|
+
{
|
|
454
|
+
tmpLog.push('Navigation complete: ' + tmpURL);
|
|
455
|
+
fReportProgress({ Percent: 60, Message: 'Rendering PDF' });
|
|
456
|
+
return tmpPage.pdf({ path: tmpOutputPath, format: tmpFormat, printBackground: true });
|
|
457
|
+
})
|
|
458
|
+
.then(function ()
|
|
459
|
+
{
|
|
460
|
+
tmpLog.push('PDF saved: ' + tmpOutputPath);
|
|
461
|
+
fReportProgress({ Percent: 100, Message: 'Done' });
|
|
462
|
+
return tmpPage.close();
|
|
463
|
+
})
|
|
464
|
+
.then(function ()
|
|
465
|
+
{
|
|
466
|
+
return fCallback(null, {
|
|
467
|
+
Outputs: { StdOut: 'PDF saved to ' + tmpOutputPath, ExitCode: 0, Result: tmpOutputPath },
|
|
468
|
+
Log: tmpLog
|
|
469
|
+
});
|
|
470
|
+
})
|
|
471
|
+
.catch(function (pError)
|
|
472
|
+
{
|
|
473
|
+
tmpLog.push('Error: ' + pError.message);
|
|
474
|
+
if (tmpPage) { tmpPage.close().catch(function () {}); }
|
|
475
|
+
return fCallback(null, {
|
|
476
|
+
Outputs: { StdOut: 'PDF failed: ' + pError.message, ExitCode: 1, Result: '' },
|
|
477
|
+
Log: tmpLog
|
|
478
|
+
});
|
|
479
|
+
});
|
|
480
|
+
}
|
|
481
|
+
|
|
482
|
+
_executeScrape(pWorkItem, pContext, fCallback, fReportProgress)
|
|
483
|
+
{
|
|
484
|
+
let tmpSettings = pWorkItem.Settings || {};
|
|
485
|
+
let tmpURL = tmpSettings.URL;
|
|
486
|
+
let tmpSelector = tmpSettings.Selector || 'body';
|
|
487
|
+
let tmpAttribute = tmpSettings.ExtractAttribute || null;
|
|
488
|
+
let tmpThis = this;
|
|
489
|
+
|
|
490
|
+
if (!tmpURL)
|
|
491
|
+
{
|
|
492
|
+
return fCallback(null, {
|
|
493
|
+
Outputs: { StdOut: 'No URL specified.', ExitCode: -1, Result: '' },
|
|
494
|
+
Log: ['HeadlessChrome Scrape: no URL specified.']
|
|
495
|
+
});
|
|
496
|
+
}
|
|
497
|
+
|
|
498
|
+
if (!this._isDomainAllowed(tmpURL))
|
|
499
|
+
{
|
|
500
|
+
return fCallback(null, {
|
|
501
|
+
Outputs: { StdOut: 'Domain not in AllowedDomains: ' + tmpURL, ExitCode: -1, Result: '' },
|
|
502
|
+
Log: ['HeadlessChrome Scrape: domain blocked by AllowedDomains.']
|
|
503
|
+
});
|
|
504
|
+
}
|
|
505
|
+
|
|
506
|
+
let tmpPage;
|
|
507
|
+
let tmpLog = [];
|
|
508
|
+
|
|
509
|
+
this._Browser.newPage()
|
|
510
|
+
.then(function (pPage)
|
|
511
|
+
{
|
|
512
|
+
tmpPage = pPage;
|
|
513
|
+
fReportProgress({ Percent: 20, Message: 'Loading page' });
|
|
514
|
+
return tmpPage.goto(tmpURL, {
|
|
515
|
+
waitUntil: 'networkidle2',
|
|
516
|
+
timeout: tmpThis._NavigationTimeoutMs
|
|
517
|
+
});
|
|
518
|
+
})
|
|
519
|
+
.then(function ()
|
|
520
|
+
{
|
|
521
|
+
tmpLog.push('Loaded: ' + tmpURL);
|
|
522
|
+
fReportProgress({ Percent: 60, Message: 'Extracting content' });
|
|
523
|
+
|
|
524
|
+
if (tmpAttribute)
|
|
525
|
+
{
|
|
526
|
+
return tmpPage.$$eval(tmpSelector, function (pElements, pAttr)
|
|
527
|
+
{
|
|
528
|
+
return pElements.map(function (pEl) { return pEl.getAttribute(pAttr) || ''; });
|
|
529
|
+
}, tmpAttribute);
|
|
530
|
+
}
|
|
531
|
+
else
|
|
532
|
+
{
|
|
533
|
+
return tmpPage.$$eval(tmpSelector, function (pElements)
|
|
534
|
+
{
|
|
535
|
+
return pElements.map(function (pEl) { return pEl.textContent.trim(); });
|
|
536
|
+
});
|
|
537
|
+
}
|
|
538
|
+
})
|
|
539
|
+
.then(function (pResults)
|
|
540
|
+
{
|
|
541
|
+
tmpLog.push('Extracted ' + pResults.length + ' elements matching [' + tmpSelector + '].');
|
|
542
|
+
fReportProgress({ Percent: 100, Message: 'Done' });
|
|
543
|
+
return tmpPage.close().then(function () { return pResults; });
|
|
544
|
+
})
|
|
545
|
+
.then(function (pResults)
|
|
546
|
+
{
|
|
547
|
+
let tmpResultStr = JSON.stringify(pResults);
|
|
548
|
+
|
|
549
|
+
return fCallback(null, {
|
|
550
|
+
Outputs: {
|
|
551
|
+
StdOut: 'Scraped ' + pResults.length + ' elements.',
|
|
552
|
+
ExitCode: 0,
|
|
553
|
+
Result: tmpResultStr
|
|
554
|
+
},
|
|
555
|
+
Log: tmpLog
|
|
556
|
+
});
|
|
557
|
+
})
|
|
558
|
+
.catch(function (pError)
|
|
559
|
+
{
|
|
560
|
+
tmpLog.push('Error: ' + pError.message);
|
|
561
|
+
if (tmpPage) { tmpPage.close().catch(function () {}); }
|
|
562
|
+
return fCallback(null, {
|
|
563
|
+
Outputs: { StdOut: 'Scrape failed: ' + pError.message, ExitCode: 1, Result: '' },
|
|
564
|
+
Log: tmpLog
|
|
565
|
+
});
|
|
566
|
+
});
|
|
567
|
+
}
|
|
568
|
+
}
|
|
569
|
+
|
|
570
|
+
module.exports = HeadlessChromeProvider;
|
|
571
|
+
```
|
|
572
|
+
|
|
573
|
+
### Configuration
|
|
574
|
+
|
|
575
|
+
```json
|
|
576
|
+
{
|
|
577
|
+
"ServerURL": "http://orchestrator:54321",
|
|
578
|
+
"Name": "browser-worker-1",
|
|
579
|
+
"MaxConcurrent": 2,
|
|
580
|
+
"StagingPath": "/data/browser-staging",
|
|
581
|
+
"Providers": [
|
|
582
|
+
{ "Source": "Shell" },
|
|
583
|
+
{ "Source": "FileSystem" },
|
|
584
|
+
{
|
|
585
|
+
"Source": "./headless-chrome-provider.cjs",
|
|
586
|
+
"Config": {
|
|
587
|
+
"NavigationTimeoutMs": 60000,
|
|
588
|
+
"Viewport": { "width": 1920, "height": 1080 },
|
|
589
|
+
"AllowedDomains": ["example.com", "internal.corp.net"]
|
|
590
|
+
}
|
|
591
|
+
}
|
|
592
|
+
]
|
|
593
|
+
}
|
|
594
|
+
```
|
|
595
|
+
|
|
596
|
+
### Dispatching Browser Work
|
|
597
|
+
|
|
598
|
+
Screenshot a page:
|
|
599
|
+
|
|
600
|
+
```json
|
|
601
|
+
{
|
|
602
|
+
"Type": "beacon-dispatch",
|
|
603
|
+
"Settings":
|
|
604
|
+
{
|
|
605
|
+
"Capability": "Browser",
|
|
606
|
+
"Action": "Screenshot",
|
|
607
|
+
"URL": "https://example.com/dashboard",
|
|
608
|
+
"OutputFile": "dashboard.png",
|
|
609
|
+
"FullPage": true,
|
|
610
|
+
"WaitForSelector": ".dashboard-loaded"
|
|
611
|
+
}
|
|
612
|
+
}
|
|
613
|
+
```
|
|
614
|
+
|
|
615
|
+
Generate a PDF report:
|
|
616
|
+
|
|
617
|
+
```json
|
|
618
|
+
{
|
|
619
|
+
"Type": "beacon-dispatch",
|
|
620
|
+
"Settings":
|
|
621
|
+
{
|
|
622
|
+
"Capability": "Browser",
|
|
623
|
+
"Action": "PDF",
|
|
624
|
+
"URL": "https://internal.corp.net/reports/monthly",
|
|
625
|
+
"OutputFile": "monthly-report.pdf",
|
|
626
|
+
"Format": "A4"
|
|
627
|
+
}
|
|
628
|
+
}
|
|
629
|
+
```
|
|
630
|
+
|
|
631
|
+
Scrape product prices:
|
|
632
|
+
|
|
633
|
+
```json
|
|
634
|
+
{
|
|
635
|
+
"Type": "beacon-dispatch",
|
|
636
|
+
"Settings":
|
|
637
|
+
{
|
|
638
|
+
"Capability": "Browser",
|
|
639
|
+
"Action": "Scrape",
|
|
640
|
+
"URL": "https://example.com/products",
|
|
641
|
+
"Selector": ".product-card .price",
|
|
642
|
+
"ExtractAttribute": null
|
|
643
|
+
}
|
|
644
|
+
}
|
|
645
|
+
```
|
|
646
|
+
|
|
647
|
+
### Key Design Decisions
|
|
648
|
+
|
|
649
|
+
1. **Browser pool via `initialize`/`shutdown`.** Chrome launches once when the
|
|
650
|
+
Beacon starts and stays alive across work items. Each action opens a new
|
|
651
|
+
tab, does its work, and closes the tab. This avoids the cost of launching
|
|
652
|
+
a fresh browser for every request.
|
|
653
|
+
|
|
654
|
+
2. **Domain allow-list.** The `AllowedDomains` config prevents the Beacon
|
|
655
|
+
from being used to scrape arbitrary websites. When the list is empty, all
|
|
656
|
+
domains are allowed (useful for development).
|
|
657
|
+
|
|
658
|
+
3. **Progress reporting.** Each action reports progress at key milestones
|
|
659
|
+
(navigation, rendering, completion) so the orchestrator can show real-time
|
|
660
|
+
status for long-loading pages.
|
|
661
|
+
|
|
662
|
+
4. **Output paths resolve against StagingPath.** Relative `OutputFile` values
|
|
663
|
+
resolve against the Beacon's staging directory, keeping outputs organized
|
|
664
|
+
and accessible to subsequent tasks via affinity.
|
|
665
|
+
|
|
666
|
+
---
|
|
667
|
+
|
|
668
|
+
## Example 2: FFmpeg Transcode Provider
|
|
669
|
+
|
|
670
|
+
This provider transcodes video files to browser-streamable formats using
|
|
671
|
+
FFmpeg. It supports configurable presets and reports real-time encoding
|
|
672
|
+
progress by parsing FFmpeg's output.
|
|
673
|
+
|
|
674
|
+
### Provider Implementation
|
|
675
|
+
|
|
676
|
+
```js
|
|
677
|
+
// ffmpeg-transcode-provider.cjs
|
|
678
|
+
|
|
679
|
+
const libBeaconCapabilityProvider = require('ultravisor').BeaconCapabilityProvider;
|
|
680
|
+
const libChildProcess = require('child_process');
|
|
681
|
+
const libPath = require('path');
|
|
682
|
+
const libFS = require('fs');
|
|
683
|
+
|
|
684
|
+
class FFmpegTranscodeProvider extends libBeaconCapabilityProvider
|
|
685
|
+
{
|
|
686
|
+
constructor(pProviderConfig)
|
|
687
|
+
{
|
|
688
|
+
super(pProviderConfig);
|
|
689
|
+
|
|
690
|
+
this.Name = 'FFmpegTranscode';
|
|
691
|
+
this.Capability = 'MediaProcessing';
|
|
692
|
+
|
|
693
|
+
this._FFmpegPath = this._ProviderConfig.FFmpegPath || 'ffmpeg';
|
|
694
|
+
this._FFprobePath = this._ProviderConfig.FFprobePath || 'ffprobe';
|
|
695
|
+
|
|
696
|
+
// Preset library — each preset is an array of FFmpeg arguments
|
|
697
|
+
this._Presets =
|
|
698
|
+
{
|
|
699
|
+
// H.264 + AAC in MP4 — plays in all modern browsers
|
|
700
|
+
'browser-mp4':
|
|
701
|
+
[
|
|
702
|
+
'-c:v', 'libx264',
|
|
703
|
+
'-preset', 'medium',
|
|
704
|
+
'-crf', '23',
|
|
705
|
+
'-c:a', 'aac',
|
|
706
|
+
'-b:a', '128k',
|
|
707
|
+
'-movflags', '+faststart', // moves moov atom for streaming
|
|
708
|
+
'-pix_fmt', 'yuv420p' // broad compatibility
|
|
709
|
+
],
|
|
710
|
+
|
|
711
|
+
// VP9 + Opus in WebM — smaller files, good for web
|
|
712
|
+
'browser-webm':
|
|
713
|
+
[
|
|
714
|
+
'-c:v', 'libvpx-vp9',
|
|
715
|
+
'-crf', '30',
|
|
716
|
+
'-b:v', '0',
|
|
717
|
+
'-c:a', 'libopus',
|
|
718
|
+
'-b:a', '96k'
|
|
719
|
+
],
|
|
720
|
+
|
|
721
|
+
// H.264 Low — mobile-friendly, small file
|
|
722
|
+
'browser-mobile':
|
|
723
|
+
[
|
|
724
|
+
'-c:v', 'libx264',
|
|
725
|
+
'-preset', 'fast',
|
|
726
|
+
'-crf', '28',
|
|
727
|
+
'-vf', 'scale=-2:720', // cap at 720p
|
|
728
|
+
'-c:a', 'aac',
|
|
729
|
+
'-b:a', '96k',
|
|
730
|
+
'-movflags', '+faststart',
|
|
731
|
+
'-pix_fmt', 'yuv420p'
|
|
732
|
+
],
|
|
733
|
+
|
|
734
|
+
// Thumbnail extraction — single frame to JPEG
|
|
735
|
+
'thumbnail':
|
|
736
|
+
[
|
|
737
|
+
'-vframes', '1',
|
|
738
|
+
'-q:v', '2'
|
|
739
|
+
],
|
|
740
|
+
|
|
741
|
+
// Audio extraction — AAC in M4A container
|
|
742
|
+
'audio-only':
|
|
743
|
+
[
|
|
744
|
+
'-vn',
|
|
745
|
+
'-c:a', 'aac',
|
|
746
|
+
'-b:a', '192k'
|
|
747
|
+
]
|
|
748
|
+
};
|
|
749
|
+
|
|
750
|
+
// Merge any custom presets from config
|
|
751
|
+
if (this._ProviderConfig.Presets)
|
|
752
|
+
{
|
|
753
|
+
let tmpCustom = this._ProviderConfig.Presets;
|
|
754
|
+
for (let tmpName in tmpCustom)
|
|
755
|
+
{
|
|
756
|
+
if (tmpCustom.hasOwnProperty(tmpName))
|
|
757
|
+
{
|
|
758
|
+
this._Presets[tmpName] = tmpCustom[tmpName];
|
|
759
|
+
}
|
|
760
|
+
}
|
|
761
|
+
}
|
|
762
|
+
}
|
|
763
|
+
|
|
764
|
+
get actions()
|
|
765
|
+
{
|
|
766
|
+
return {
|
|
767
|
+
'Transcode':
|
|
768
|
+
{
|
|
769
|
+
Description: 'Transcode a video file using a named preset or custom FFmpeg arguments.',
|
|
770
|
+
SettingsSchema:
|
|
771
|
+
[
|
|
772
|
+
{ Name: 'InputFile', DataType: 'String', Required: true, Description: 'Path to the source video file' },
|
|
773
|
+
{ Name: 'OutputFile', DataType: 'String', Required: true, Description: 'Path for the transcoded output' },
|
|
774
|
+
{ Name: 'Preset', DataType: 'String', Required: false, Description: 'Named preset (browser-mp4, browser-webm, browser-mobile, thumbnail, audio-only)' },
|
|
775
|
+
{ Name: 'CustomArgs', DataType: 'String', Required: false, Description: 'Custom FFmpeg arguments (used instead of preset)' }
|
|
776
|
+
]
|
|
777
|
+
},
|
|
778
|
+
'Probe':
|
|
779
|
+
{
|
|
780
|
+
Description: 'Inspect a media file and return format/codec/duration metadata.',
|
|
781
|
+
SettingsSchema:
|
|
782
|
+
[
|
|
783
|
+
{ Name: 'InputFile', DataType: 'String', Required: true, Description: 'Path to the media file' }
|
|
784
|
+
]
|
|
785
|
+
}
|
|
786
|
+
};
|
|
787
|
+
}
|
|
788
|
+
|
|
789
|
+
// -------------------------------------------------------------------
|
|
790
|
+
// Lifecycle — validate FFmpeg is installed
|
|
791
|
+
// -------------------------------------------------------------------
|
|
792
|
+
|
|
793
|
+
initialize(fCallback)
|
|
794
|
+
{
|
|
795
|
+
let tmpThis = this;
|
|
796
|
+
|
|
797
|
+
libChildProcess.exec(this._FFmpegPath + ' -version', function (pError, pStdOut)
|
|
798
|
+
{
|
|
799
|
+
if (pError)
|
|
800
|
+
{
|
|
801
|
+
return fCallback(new Error(
|
|
802
|
+
'FFmpeg not found at [' + tmpThis._FFmpegPath + ']. ' +
|
|
803
|
+
'Install it or set FFmpegPath in provider config.'
|
|
804
|
+
));
|
|
805
|
+
}
|
|
806
|
+
|
|
807
|
+
// Extract version for logging
|
|
808
|
+
let tmpMatch = pStdOut.match(/ffmpeg version (\S+)/);
|
|
809
|
+
let tmpVersion = tmpMatch ? tmpMatch[1] : 'unknown';
|
|
810
|
+
console.log('[FFmpegTranscode] Initialized with FFmpeg ' + tmpVersion);
|
|
811
|
+
console.log('[FFmpegTranscode] Available presets: ' + Object.keys(tmpThis._Presets).join(', '));
|
|
812
|
+
return fCallback(null);
|
|
813
|
+
});
|
|
814
|
+
}
|
|
815
|
+
|
|
816
|
+
// -------------------------------------------------------------------
|
|
817
|
+
// Action dispatcher
|
|
818
|
+
// -------------------------------------------------------------------
|
|
819
|
+
|
|
820
|
+
execute(pAction, pWorkItem, pContext, fCallback, fReportProgress)
|
|
821
|
+
{
|
|
822
|
+
switch (pAction)
|
|
823
|
+
{
|
|
824
|
+
case 'Transcode':
|
|
825
|
+
return this._executeTranscode(pWorkItem, pContext, fCallback, fReportProgress);
|
|
826
|
+
case 'Probe':
|
|
827
|
+
return this._executeProbe(pWorkItem, pContext, fCallback, fReportProgress);
|
|
828
|
+
default:
|
|
829
|
+
return fCallback(null, {
|
|
830
|
+
Outputs: { StdOut: 'Unknown action: ' + pAction, ExitCode: -1, Result: '' },
|
|
831
|
+
Log: ['FFmpegTranscode: unsupported action [' + pAction + '].']
|
|
832
|
+
});
|
|
833
|
+
}
|
|
834
|
+
}
|
|
835
|
+
|
|
836
|
+
// -------------------------------------------------------------------
|
|
837
|
+
// Transcode
|
|
838
|
+
// -------------------------------------------------------------------
|
|
839
|
+
|
|
840
|
+
_executeTranscode(pWorkItem, pContext, fCallback, fReportProgress)
|
|
841
|
+
{
|
|
842
|
+
let tmpSettings = pWorkItem.Settings || {};
|
|
843
|
+
let tmpInputFile = tmpSettings.InputFile || '';
|
|
844
|
+
let tmpOutputFile = tmpSettings.OutputFile || '';
|
|
845
|
+
let tmpPresetName = tmpSettings.Preset || 'browser-mp4';
|
|
846
|
+
let tmpCustomArgs = tmpSettings.CustomArgs || '';
|
|
847
|
+
let tmpThis = this;
|
|
848
|
+
|
|
849
|
+
if (!tmpInputFile || !tmpOutputFile)
|
|
850
|
+
{
|
|
851
|
+
return fCallback(null, {
|
|
852
|
+
Outputs: { StdOut: 'InputFile and OutputFile are required.', ExitCode: -1, Result: '' },
|
|
853
|
+
Log: ['FFmpegTranscode: missing InputFile or OutputFile.']
|
|
854
|
+
});
|
|
855
|
+
}
|
|
856
|
+
|
|
857
|
+
// Resolve relative paths
|
|
858
|
+
let tmpStagingPath = pContext.StagingPath || process.cwd();
|
|
859
|
+
if (!libPath.isAbsolute(tmpInputFile)) tmpInputFile = libPath.resolve(tmpStagingPath, tmpInputFile);
|
|
860
|
+
if (!libPath.isAbsolute(tmpOutputFile)) tmpOutputFile = libPath.resolve(tmpStagingPath, tmpOutputFile);
|
|
861
|
+
|
|
862
|
+
if (!libFS.existsSync(tmpInputFile))
|
|
863
|
+
{
|
|
864
|
+
return fCallback(null, {
|
|
865
|
+
Outputs: { StdOut: 'Input file not found: ' + tmpInputFile, ExitCode: 1, Result: '' },
|
|
866
|
+
Log: ['FFmpegTranscode: input not found: ' + tmpInputFile]
|
|
867
|
+
});
|
|
868
|
+
}
|
|
869
|
+
|
|
870
|
+
// Ensure output directory exists
|
|
871
|
+
let tmpOutputDir = libPath.dirname(tmpOutputFile);
|
|
872
|
+
if (!libFS.existsSync(tmpOutputDir))
|
|
873
|
+
{
|
|
874
|
+
libFS.mkdirSync(tmpOutputDir, { recursive: true });
|
|
875
|
+
}
|
|
876
|
+
|
|
877
|
+
let tmpLog = [];
|
|
878
|
+
|
|
879
|
+
// Step 1: Probe input to get duration (for progress %)
|
|
880
|
+
tmpThis._getDuration(tmpInputFile, function (pDurationSec)
|
|
881
|
+
{
|
|
882
|
+
tmpLog.push('Input file: ' + tmpInputFile);
|
|
883
|
+
tmpLog.push('Duration: ' + (pDurationSec > 0 ? pDurationSec + 's' : 'unknown'));
|
|
884
|
+
fReportProgress({ Percent: 5, Message: 'Starting transcode' });
|
|
885
|
+
|
|
886
|
+
// Build FFmpeg command
|
|
887
|
+
let tmpArgs = ['-i', tmpInputFile, '-y']; // -y = overwrite output
|
|
888
|
+
|
|
889
|
+
if (tmpCustomArgs)
|
|
890
|
+
{
|
|
891
|
+
// Use custom args (split on spaces, respecting quotes)
|
|
892
|
+
tmpArgs = tmpArgs.concat(tmpCustomArgs.split(/\s+/));
|
|
893
|
+
}
|
|
894
|
+
else
|
|
895
|
+
{
|
|
896
|
+
let tmpPreset = tmpThis._Presets[tmpPresetName];
|
|
897
|
+
if (!tmpPreset)
|
|
898
|
+
{
|
|
899
|
+
return fCallback(null, {
|
|
900
|
+
Outputs: {
|
|
901
|
+
StdOut: 'Unknown preset: ' + tmpPresetName + '. Available: ' +
|
|
902
|
+
Object.keys(tmpThis._Presets).join(', '),
|
|
903
|
+
ExitCode: -1,
|
|
904
|
+
Result: ''
|
|
905
|
+
},
|
|
906
|
+
Log: tmpLog.concat(['Unknown preset: ' + tmpPresetName])
|
|
907
|
+
});
|
|
908
|
+
}
|
|
909
|
+
tmpArgs = tmpArgs.concat(tmpPreset);
|
|
910
|
+
tmpLog.push('Using preset: ' + tmpPresetName);
|
|
911
|
+
}
|
|
912
|
+
|
|
913
|
+
tmpArgs.push(tmpOutputFile);
|
|
914
|
+
|
|
915
|
+
// Step 2: Run FFmpeg with progress parsing
|
|
916
|
+
let tmpProcess = libChildProcess.spawn(tmpThis._FFmpegPath, tmpArgs, {
|
|
917
|
+
stdio: ['pipe', 'pipe', 'pipe']
|
|
918
|
+
});
|
|
919
|
+
|
|
920
|
+
let tmpStdErr = '';
|
|
921
|
+
|
|
922
|
+
tmpProcess.stderr.on('data', function (pData)
|
|
923
|
+
{
|
|
924
|
+
tmpStdErr += pData.toString();
|
|
925
|
+
|
|
926
|
+
// Parse FFmpeg progress output: "time=00:01:23.45"
|
|
927
|
+
if (pDurationSec > 0)
|
|
928
|
+
{
|
|
929
|
+
let tmpTimeMatch = pData.toString().match(/time=(\d+):(\d+):(\d+\.\d+)/);
|
|
930
|
+
if (tmpTimeMatch)
|
|
931
|
+
{
|
|
932
|
+
let tmpCurrentSec = (parseInt(tmpTimeMatch[1]) * 3600) +
|
|
933
|
+
(parseInt(tmpTimeMatch[2]) * 60) +
|
|
934
|
+
parseFloat(tmpTimeMatch[3]);
|
|
935
|
+
let tmpPercent = Math.min(95, Math.round((tmpCurrentSec / pDurationSec) * 100));
|
|
936
|
+
fReportProgress({
|
|
937
|
+
Percent: tmpPercent,
|
|
938
|
+
Message: 'Encoding: ' + Math.round(tmpCurrentSec) + 's / ' + Math.round(pDurationSec) + 's'
|
|
939
|
+
});
|
|
940
|
+
}
|
|
941
|
+
}
|
|
942
|
+
});
|
|
943
|
+
|
|
944
|
+
tmpProcess.on('close', function (pCode)
|
|
945
|
+
{
|
|
946
|
+
if (pCode !== 0)
|
|
947
|
+
{
|
|
948
|
+
// Extract last few lines of stderr for the error message
|
|
949
|
+
let tmpErrLines = tmpStdErr.trim().split('\n');
|
|
950
|
+
let tmpErrMsg = tmpErrLines.slice(-3).join(' | ');
|
|
951
|
+
tmpLog.push('FFmpeg exited with code ' + pCode + ': ' + tmpErrMsg);
|
|
952
|
+
return fCallback(null, {
|
|
953
|
+
Outputs: { StdOut: 'Transcode failed (exit code ' + pCode + ')', ExitCode: pCode, Result: '' },
|
|
954
|
+
Log: tmpLog
|
|
955
|
+
});
|
|
956
|
+
}
|
|
957
|
+
|
|
958
|
+
// Get output file size
|
|
959
|
+
let tmpStats = libFS.statSync(tmpOutputFile);
|
|
960
|
+
let tmpSizeMB = (tmpStats.size / (1024 * 1024)).toFixed(2);
|
|
961
|
+
tmpLog.push('Output: ' + tmpOutputFile + ' (' + tmpSizeMB + ' MB)');
|
|
962
|
+
fReportProgress({ Percent: 100, Message: 'Transcode complete' });
|
|
963
|
+
|
|
964
|
+
return fCallback(null, {
|
|
965
|
+
Outputs: {
|
|
966
|
+
StdOut: 'Transcoded to ' + tmpOutputFile + ' (' + tmpSizeMB + ' MB)',
|
|
967
|
+
ExitCode: 0,
|
|
968
|
+
Result: JSON.stringify({
|
|
969
|
+
OutputFile: tmpOutputFile,
|
|
970
|
+
SizeBytes: tmpStats.size,
|
|
971
|
+
Preset: tmpPresetName
|
|
972
|
+
})
|
|
973
|
+
},
|
|
974
|
+
Log: tmpLog
|
|
975
|
+
});
|
|
976
|
+
});
|
|
977
|
+
|
|
978
|
+
tmpProcess.on('error', function (pError)
|
|
979
|
+
{
|
|
980
|
+
tmpLog.push('Spawn error: ' + pError.message);
|
|
981
|
+
return fCallback(null, {
|
|
982
|
+
Outputs: { StdOut: 'Failed to start FFmpeg: ' + pError.message, ExitCode: 1, Result: '' },
|
|
983
|
+
Log: tmpLog
|
|
984
|
+
});
|
|
985
|
+
});
|
|
986
|
+
});
|
|
987
|
+
}
|
|
988
|
+
|
|
989
|
+
// -------------------------------------------------------------------
|
|
990
|
+
// Probe — return media metadata as JSON
|
|
991
|
+
// -------------------------------------------------------------------
|
|
992
|
+
|
|
993
|
+
_executeProbe(pWorkItem, pContext, fCallback, fReportProgress)
|
|
994
|
+
{
|
|
995
|
+
let tmpSettings = pWorkItem.Settings || {};
|
|
996
|
+
let tmpInputFile = tmpSettings.InputFile || '';
|
|
997
|
+
|
|
998
|
+
if (!tmpInputFile)
|
|
999
|
+
{
|
|
1000
|
+
return fCallback(null, {
|
|
1001
|
+
Outputs: { StdOut: 'No InputFile specified.', ExitCode: -1, Result: '' },
|
|
1002
|
+
Log: ['FFmpegTranscode Probe: no InputFile specified.']
|
|
1003
|
+
});
|
|
1004
|
+
}
|
|
1005
|
+
|
|
1006
|
+
let tmpStagingPath = pContext.StagingPath || process.cwd();
|
|
1007
|
+
if (!libPath.isAbsolute(tmpInputFile)) tmpInputFile = libPath.resolve(tmpStagingPath, tmpInputFile);
|
|
1008
|
+
|
|
1009
|
+
fReportProgress({ Percent: 50, Message: 'Probing ' + libPath.basename(tmpInputFile) });
|
|
1010
|
+
|
|
1011
|
+
let tmpCmd = this._FFprobePath +
|
|
1012
|
+
' -v quiet -print_format json -show_format -show_streams "' +
|
|
1013
|
+
tmpInputFile + '"';
|
|
1014
|
+
|
|
1015
|
+
libChildProcess.exec(tmpCmd, { maxBuffer: 5 * 1024 * 1024 }, function (pError, pStdOut)
|
|
1016
|
+
{
|
|
1017
|
+
if (pError)
|
|
1018
|
+
{
|
|
1019
|
+
return fCallback(null, {
|
|
1020
|
+
Outputs: { StdOut: 'Probe failed: ' + pError.message, ExitCode: 1, Result: '' },
|
|
1021
|
+
Log: ['FFprobe error: ' + pError.message]
|
|
1022
|
+
});
|
|
1023
|
+
}
|
|
1024
|
+
|
|
1025
|
+
fReportProgress({ Percent: 100, Message: 'Probe complete' });
|
|
1026
|
+
|
|
1027
|
+
return fCallback(null, {
|
|
1028
|
+
Outputs: {
|
|
1029
|
+
StdOut: 'Probe complete for ' + tmpInputFile,
|
|
1030
|
+
ExitCode: 0,
|
|
1031
|
+
Result: pStdOut.trim()
|
|
1032
|
+
},
|
|
1033
|
+
Log: ['FFprobe succeeded for ' + tmpInputFile]
|
|
1034
|
+
});
|
|
1035
|
+
});
|
|
1036
|
+
}
|
|
1037
|
+
|
|
1038
|
+
// -------------------------------------------------------------------
|
|
1039
|
+
// Helper: get duration in seconds via FFprobe
|
|
1040
|
+
// -------------------------------------------------------------------
|
|
1041
|
+
|
|
1042
|
+
_getDuration(pFilePath, fCallback)
|
|
1043
|
+
{
|
|
1044
|
+
let tmpCmd = this._FFprobePath +
|
|
1045
|
+
' -v quiet -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 "' +
|
|
1046
|
+
pFilePath + '"';
|
|
1047
|
+
|
|
1048
|
+
libChildProcess.exec(tmpCmd, function (pError, pStdOut)
|
|
1049
|
+
{
|
|
1050
|
+
if (pError || !pStdOut.trim())
|
|
1051
|
+
{
|
|
1052
|
+
return fCallback(0);
|
|
1053
|
+
}
|
|
1054
|
+
return fCallback(parseFloat(pStdOut.trim()) || 0);
|
|
1055
|
+
});
|
|
1056
|
+
}
|
|
1057
|
+
}
|
|
1058
|
+
|
|
1059
|
+
module.exports = FFmpegTranscodeProvider;
|
|
1060
|
+
```
|
|
1061
|
+
|
|
1062
|
+
### Configuration
|
|
1063
|
+
|
|
1064
|
+
```json
|
|
1065
|
+
{
|
|
1066
|
+
"ServerURL": "http://orchestrator:54321",
|
|
1067
|
+
"Name": "media-worker-1",
|
|
1068
|
+
"MaxConcurrent": 2,
|
|
1069
|
+
"StagingPath": "/data/media-staging",
|
|
1070
|
+
"Providers": [
|
|
1071
|
+
{ "Source": "Shell" },
|
|
1072
|
+
{
|
|
1073
|
+
"Source": "FileSystem",
|
|
1074
|
+
"Config": {
|
|
1075
|
+
"AllowedPaths": ["/data/media-staging", "/data/media-output"]
|
|
1076
|
+
}
|
|
1077
|
+
},
|
|
1078
|
+
{
|
|
1079
|
+
"Source": "./ffmpeg-transcode-provider.cjs",
|
|
1080
|
+
"Config": {
|
|
1081
|
+
"FFmpegPath": "/usr/bin/ffmpeg",
|
|
1082
|
+
"FFprobePath": "/usr/bin/ffprobe",
|
|
1083
|
+
"Presets": {
|
|
1084
|
+
"hls-720p": [
|
|
1085
|
+
"-c:v", "libx264", "-preset", "fast",
|
|
1086
|
+
"-crf", "23", "-vf", "scale=-2:720",
|
|
1087
|
+
"-c:a", "aac", "-b:a", "128k",
|
|
1088
|
+
"-f", "hls", "-hls_time", "6",
|
|
1089
|
+
"-hls_playlist_type", "vod"
|
|
1090
|
+
]
|
|
1091
|
+
}
|
|
1092
|
+
}
|
|
1093
|
+
}
|
|
1094
|
+
]
|
|
1095
|
+
}
|
|
1096
|
+
```
|
|
1097
|
+
|
|
1098
|
+
### Dispatching Transcode Work
|
|
1099
|
+
|
|
1100
|
+
Convert a raw upload to browser-streamable MP4:
|
|
1101
|
+
|
|
1102
|
+
```json
|
|
1103
|
+
{
|
|
1104
|
+
"Type": "beacon-dispatch",
|
|
1105
|
+
"Settings":
|
|
1106
|
+
{
|
|
1107
|
+
"Capability": "MediaProcessing",
|
|
1108
|
+
"Action": "Transcode",
|
|
1109
|
+
"InputFile": "uploads/raw-footage.mov",
|
|
1110
|
+
"OutputFile": "output/footage-web.mp4",
|
|
1111
|
+
"Preset": "browser-mp4"
|
|
1112
|
+
}
|
|
1113
|
+
}
|
|
1114
|
+
```
|
|
1115
|
+
|
|
1116
|
+
Generate a mobile-optimized version:
|
|
1117
|
+
|
|
1118
|
+
```json
|
|
1119
|
+
{
|
|
1120
|
+
"Type": "beacon-dispatch",
|
|
1121
|
+
"Settings":
|
|
1122
|
+
{
|
|
1123
|
+
"Capability": "MediaProcessing",
|
|
1124
|
+
"Action": "Transcode",
|
|
1125
|
+
"InputFile": "uploads/raw-footage.mov",
|
|
1126
|
+
"OutputFile": "output/footage-mobile.mp4",
|
|
1127
|
+
"Preset": "browser-mobile"
|
|
1128
|
+
}
|
|
1129
|
+
}
|
|
1130
|
+
```
|
|
1131
|
+
|
|
1132
|
+
Extract a thumbnail at the 10-second mark:
|
|
1133
|
+
|
|
1134
|
+
```json
|
|
1135
|
+
{
|
|
1136
|
+
"Type": "beacon-dispatch",
|
|
1137
|
+
"Settings":
|
|
1138
|
+
{
|
|
1139
|
+
"Capability": "MediaProcessing",
|
|
1140
|
+
"Action": "Transcode",
|
|
1141
|
+
"InputFile": "uploads/raw-footage.mov",
|
|
1142
|
+
"OutputFile": "output/thumb.jpg",
|
|
1143
|
+
"Preset": "thumbnail",
|
|
1144
|
+
"CustomArgs": "-ss 10"
|
|
1145
|
+
}
|
|
1146
|
+
}
|
|
1147
|
+
```
|
|
1148
|
+
|
|
1149
|
+
Probe file metadata before deciding how to transcode:
|
|
1150
|
+
|
|
1151
|
+
```json
|
|
1152
|
+
{
|
|
1153
|
+
"Type": "beacon-dispatch",
|
|
1154
|
+
"Settings":
|
|
1155
|
+
{
|
|
1156
|
+
"Capability": "MediaProcessing",
|
|
1157
|
+
"Action": "Probe",
|
|
1158
|
+
"InputFile": "uploads/raw-footage.mov"
|
|
1159
|
+
}
|
|
1160
|
+
}
|
|
1161
|
+
```
|
|
1162
|
+
|
|
1163
|
+
### Full Pipeline Manifest
|
|
1164
|
+
|
|
1165
|
+
A complete manifest that probes a file, then transcodes it into browser-MP4,
|
|
1166
|
+
mobile-MP4, and thumbnail — distributed across media workers:
|
|
1167
|
+
|
|
1168
|
+
```json
|
|
1169
|
+
{
|
|
1170
|
+
"Scope": "Operation",
|
|
1171
|
+
"Nodes":
|
|
1172
|
+
{
|
|
1173
|
+
"start": { "Type": "start" },
|
|
1174
|
+
|
|
1175
|
+
"probe-input":
|
|
1176
|
+
{
|
|
1177
|
+
"Type": "beacon-dispatch",
|
|
1178
|
+
"Settings":
|
|
1179
|
+
{
|
|
1180
|
+
"Capability": "MediaProcessing",
|
|
1181
|
+
"Action": "Probe",
|
|
1182
|
+
"InputFile": "{~D:Record.State.InputFile~}",
|
|
1183
|
+
"AffinityKey": "{~D:Record.State.InputFile~}"
|
|
1184
|
+
}
|
|
1185
|
+
},
|
|
1186
|
+
|
|
1187
|
+
"transcode-web":
|
|
1188
|
+
{
|
|
1189
|
+
"Type": "beacon-dispatch",
|
|
1190
|
+
"Settings":
|
|
1191
|
+
{
|
|
1192
|
+
"Capability": "MediaProcessing",
|
|
1193
|
+
"Action": "Transcode",
|
|
1194
|
+
"InputFile": "{~D:Record.State.InputFile~}",
|
|
1195
|
+
"OutputFile": "{~D:Record.State.OutputDir~}/web.mp4",
|
|
1196
|
+
"Preset": "browser-mp4",
|
|
1197
|
+
"AffinityKey": "{~D:Record.State.InputFile~}"
|
|
1198
|
+
}
|
|
1199
|
+
},
|
|
1200
|
+
|
|
1201
|
+
"transcode-mobile":
|
|
1202
|
+
{
|
|
1203
|
+
"Type": "beacon-dispatch",
|
|
1204
|
+
"Settings":
|
|
1205
|
+
{
|
|
1206
|
+
"Capability": "MediaProcessing",
|
|
1207
|
+
"Action": "Transcode",
|
|
1208
|
+
"InputFile": "{~D:Record.State.InputFile~}",
|
|
1209
|
+
"OutputFile": "{~D:Record.State.OutputDir~}/mobile.mp4",
|
|
1210
|
+
"Preset": "browser-mobile",
|
|
1211
|
+
"AffinityKey": "{~D:Record.State.InputFile~}"
|
|
1212
|
+
}
|
|
1213
|
+
},
|
|
1214
|
+
|
|
1215
|
+
"extract-thumbnail":
|
|
1216
|
+
{
|
|
1217
|
+
"Type": "beacon-dispatch",
|
|
1218
|
+
"Settings":
|
|
1219
|
+
{
|
|
1220
|
+
"Capability": "MediaProcessing",
|
|
1221
|
+
"Action": "Transcode",
|
|
1222
|
+
"InputFile": "{~D:Record.State.InputFile~}",
|
|
1223
|
+
"OutputFile": "{~D:Record.State.OutputDir~}/thumb.jpg",
|
|
1224
|
+
"Preset": "thumbnail",
|
|
1225
|
+
"AffinityKey": "{~D:Record.State.InputFile~}"
|
|
1226
|
+
}
|
|
1227
|
+
},
|
|
1228
|
+
|
|
1229
|
+
"end": { "Type": "end" }
|
|
1230
|
+
},
|
|
1231
|
+
"Edges":
|
|
1232
|
+
{
|
|
1233
|
+
"start": [{ "Target": "probe-input", "Event": "Complete" }],
|
|
1234
|
+
"probe-input": [{ "Target": "transcode-web", "Event": "Complete" }],
|
|
1235
|
+
"transcode-web": [{ "Target": "transcode-mobile", "Event": "Complete" }],
|
|
1236
|
+
"transcode-mobile": [{ "Target": "extract-thumbnail","Event": "Complete" }],
|
|
1237
|
+
"extract-thumbnail": [{ "Target": "end", "Event": "Complete" }]
|
|
1238
|
+
}
|
|
1239
|
+
}
|
|
1240
|
+
```
|
|
1241
|
+
|
|
1242
|
+
All four `beacon-dispatch` nodes share the same `AffinityKey` (the input
|
|
1243
|
+
filename), so the coordinator pins them all to whichever media worker picks
|
|
1244
|
+
up the first task. This avoids copying the source file across workers.
|
|
1245
|
+
|
|
1246
|
+
### Key Design Decisions
|
|
1247
|
+
|
|
1248
|
+
1. **Preset system.** Named presets encapsulate best-practice FFmpeg args for
|
|
1249
|
+
common targets. The `browser-mp4` preset uses `-movflags +faststart` so
|
|
1250
|
+
the browser can begin playback before the full file downloads.
|
|
1251
|
+
|
|
1252
|
+
2. **Real-time progress from FFmpeg output.** The provider parses
|
|
1253
|
+
`time=HH:MM:SS.ss` from FFmpeg's stderr to compute a percentage against
|
|
1254
|
+
the probed duration. This feeds into `fReportProgress` so the
|
|
1255
|
+
orchestrator shows encoding progress.
|
|
1256
|
+
|
|
1257
|
+
3. **`initialize` validates FFmpeg.** The Beacon will not start if FFmpeg is
|
|
1258
|
+
missing. This prevents the Beacon from registering and then failing every
|
|
1259
|
+
work item.
|
|
1260
|
+
|
|
1261
|
+
4. **Affinity for multi-output pipelines.** The example manifest uses the
|
|
1262
|
+
same AffinityKey across all transcode steps so they land on the same
|
|
1263
|
+
worker, where the source file is already on disk.
|
|
1264
|
+
|
|
1265
|
+
5. **`spawn` vs `exec`.** Transcode uses `spawn` (not `exec`) so we can
|
|
1266
|
+
stream stderr for progress parsing without buffering the entire output.
|
|
1267
|
+
Probe uses `exec` because it produces small, bounded JSON output.
|
|
1268
|
+
|
|
1269
|
+
---
|
|
1270
|
+
|
|
1271
|
+
## Provider Checklist
|
|
1272
|
+
|
|
1273
|
+
Use this as a reference when building your own provider:
|
|
1274
|
+
|
|
1275
|
+
- [ ] **Extend `CapabilityProvider`** — set `Name` and `Capability` in the
|
|
1276
|
+
constructor.
|
|
1277
|
+
- [ ] **Declare actions** — implement `get actions()` returning the action
|
|
1278
|
+
map. Each action should have a `Description` and optionally a
|
|
1279
|
+
`SettingsSchema` array.
|
|
1280
|
+
- [ ] **Implement `execute`** — dispatch on `pAction`, read from
|
|
1281
|
+
`pWorkItem.Settings`, resolve paths against `pContext.StagingPath`, call
|
|
1282
|
+
`fCallback` when done.
|
|
1283
|
+
- [ ] **Return structured results** — always include `Outputs.ExitCode`
|
|
1284
|
+
(0 = success), `Outputs.StdOut` (summary), and `Outputs.Result` (data).
|
|
1285
|
+
- [ ] **Use `fReportProgress`** — for any operation that takes more than a
|
|
1286
|
+
few seconds, report progress so the orchestrator can track it.
|
|
1287
|
+
- [ ] **Implement `initialize`** — validate prerequisites (is the binary
|
|
1288
|
+
installed? can we connect to the service?). Return an error to prevent
|
|
1289
|
+
the Beacon from starting if the provider cannot function.
|
|
1290
|
+
- [ ] **Implement `shutdown`** — release resources (close browser, disconnect
|
|
1291
|
+
from service, clean up temp files).
|
|
1292
|
+
- [ ] **Handle errors gracefully** — catch exceptions and return them as
|
|
1293
|
+
`ExitCode: 1` results rather than throwing. The Beacon must stay alive
|
|
1294
|
+
even if individual work items fail.
|
|
1295
|
+
- [ ] **Resolve paths** — use `pContext.StagingPath` to resolve relative file
|
|
1296
|
+
paths. This keeps file operations predictable across machines.
|
|
1297
|
+
- [ ] **Guard external access** — if the provider makes network requests or
|
|
1298
|
+
accesses the filesystem, add configuration to restrict what it can reach
|
|
1299
|
+
(e.g. `AllowedDomains`, `AllowedPaths`).
|