@genroc/eval-node 0.0.0-edge.e4a77190 → 0.0.0-edge.e50224ac
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +114 -306
- package/dist/import.js +17 -15
- package/dist/realm.js +4 -1
- package/dist/worker.js +22 -9
- package/import.ts +23 -20
- package/package.json +1 -1
- package/realm.ts +4 -1
- package/worker.ts +23 -9
package/README.md
CHANGED
|
@@ -1,337 +1,145 @@
|
|
|
1
1
|
# @genroc/eval-node
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
TypeScript script tasks for [genroc](https://genroc.org). The package provides two binaries:
|
|
4
4
|
|
|
5
|
-
*
|
|
6
|
-
|
|
7
|
-
*
|
|
8
|
-
|
|
5
|
+
* **`genroc-import`**: the bundler. `genctl` runs it on every `apply` and `generate` to turn
|
|
6
|
+
`$import: ./script.ts` into one typechecked, self-contained module string.
|
|
7
|
+
* **`genroc-eval-node`**: the worker. It claims script tasks from genroc's queue and runs each
|
|
8
|
+
one in a fresh worker thread.
|
|
9
9
|
|
|
10
|
-
|
|
10
|
+
Requires **Node 24+**. Full guide:
|
|
11
|
+
[TypeScript evaluation](https://genroc.org/guides/process-definition/typescript-evaluation/).
|
|
11
12
|
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
Then register the resolver in a `.genroc` beside your definitions — a dotfile, so it does not
|
|
15
|
-
read as another process definition. Discovery walks up from the source file, so nothing depends
|
|
16
|
-
on the cwd:
|
|
17
|
-
|
|
18
|
-
resolvers:
|
|
19
|
-
import:
|
|
20
|
-
phase: code
|
|
21
|
-
ext: .ts
|
|
22
|
-
command: [npx, genroc-import]
|
|
23
|
-
|
|
24
|
-
`genctl apply` and `genctl types` now resolve `$import` directives. Typechecking is the
|
|
25
|
-
resolver's exit code, so a stored definition cannot hold code that failed to typecheck.
|
|
26
|
-
|
|
27
|
-
## Building
|
|
28
|
-
|
|
29
|
-
pnpm run build # tsc -p tsconfig.build.json -> dist/
|
|
30
|
-
|
|
31
|
-
The published package is JavaScript, not TypeScript. **Node refuses to strip types for files
|
|
32
|
-
under `node_modules`**, so shipping `.ts` works from a checkout and fails for every consumer —
|
|
33
|
-
which is why `prepublishOnly` builds and CI installs the packed tarball rather than only running
|
|
34
|
-
from source. The source keeps its `.ts` import extensions; `rewriteRelativeImportExtensions`
|
|
35
|
-
turns them into `.js` on the way out.
|
|
36
|
-
|
|
37
|
-
## Running the worker
|
|
38
|
-
|
|
39
|
-
docker run -e GENROC_SERVER=http://host:8448 ghcr.io/genroc/eval-node:preview
|
|
40
|
-
|
|
41
|
-
or locally, if you already have Node:
|
|
42
|
-
|
|
43
|
-
GENROC_SERVER=http://localhost:8448 npx genroc-eval-node
|
|
44
|
-
|
|
45
|
-
Add `GENROC_TOKEN` once the server has auth on; `worker` is the only permission it needs.
|
|
46
|
-
|
|
47
|
-
**The bundler cannot be the image.** It runs per edit and must resolve your project's own
|
|
48
|
-
`node_modules`, so a container start per invocation would break the editor loop.
|
|
49
|
-
|
|
50
|
-
Design record: [specs/external-task-queue.md](../specs/external-task-queue.md) for the queue,
|
|
51
|
-
[specs/script-tasks.md](../specs/script-tasks.md) for the runtime.
|
|
13
|
+
## Quick start
|
|
52
14
|
|
|
53
|
-
|
|
15
|
+
genctl init --eval-node
|
|
54
16
|
|
|
55
|
-
|
|
17
|
+
This scaffolds a project with everything set up: the package, a `.genroc` that registers the
|
|
18
|
+
resolver, a `tsconfig.json`, a reusable `script-node` child process, and a `compose.yaml` that
|
|
19
|
+
runs the worker.
|
|
56
20
|
|
|
57
|
-
|
|
58
|
-
GENROC_SERVER=http://localhost:8448 GENROC_TOKEN=$TOKEN node eval-node/worker.ts
|
|
21
|
+
## Adding it to an existing project
|
|
59
22
|
|
|
60
|
-
|
|
61
|
-
externalized input and answer, and nothing else. This is the credential most likely to end up on
|
|
62
|
-
a machine you trust least, so it is worth scoping rather than reusing an admin token: a leaked
|
|
63
|
-
one cannot read a definition, list an instance, or mint another token.
|
|
64
|
-
|
|
65
|
-
A credential problem **exits the worker** rather than being retried. Polling through a 401 looks
|
|
66
|
-
like a healthy worker that never picks anything up, which is the worst shape to debug.
|
|
67
|
-
|
|
68
|
-
It needs **Node 24 or newer**: the sources are TypeScript and are run as-is, by Node's own
|
|
69
|
-
type stripping.
|
|
70
|
-
|
|
71
|
-
| env | default | |
|
|
72
|
-
|---|---|---|
|
|
73
|
-
| `GENROC_SERVER` | `http://localhost:8448` | where to claim from |
|
|
74
|
-
| `GENROC_TOKEN` | *(none)* | credential, when the server runs with `--auth token`. Needs the **`worker`** permission and nothing more |
|
|
75
|
-
| `WORKER_ID` | `evaluator-<pid>` | the claim holder; renewals are scoped to it |
|
|
76
|
-
| `CONCURRENCY` | `4` | how many scripts run at once — **this worker's** decision |
|
|
77
|
-
| `LEASE_MS` | `30000` | visibility timeout; renewed at a third of it while working |
|
|
78
|
-
| `POLL_MS` | `250` | idle poll interval; a non-empty claim polls again immediately |
|
|
79
|
-
| `PROCESS` / `TASK` | unset | claim only this process / task id |
|
|
80
|
-
|
|
81
|
-
**The worker calls genroc, not the other way round.** That is the whole reason this is a
|
|
82
|
-
queue worker rather than the HTTP sidecar it used to be:
|
|
83
|
-
|
|
84
|
-
- **Concurrency is the worker's to set.** Under the old `fetch` shape genroc decided how many
|
|
85
|
-
scripts ran at once (`--max-concurrent`, default 200) and the evaluator accepted every one.
|
|
86
|
-
A backlog is now a queue, not 200 threads fighting over a core.
|
|
87
|
-
- **No engine worker is held.** A `fetch` occupies one of genroc's advance slots for its whole
|
|
88
|
-
duration, so a slow evaluator starved unrelated tasks. An `external` task parks.
|
|
89
|
-
- **The connection direction inverts.** `/eval` was unauthenticated arbitrary code execution
|
|
90
|
-
that genroc had to be able to reach. A worker only needs outbound access, so it can live
|
|
91
|
-
anywhere — behind NAT, in another trust zone.
|
|
92
|
-
|
|
93
|
-
The cost of that inversion is that genroc cannot reach a running worker, so anything it needs
|
|
94
|
-
to say has to ride the renewal the worker already makes. **Renewing is therefore mandatory,
|
|
95
|
-
not an optimisation** — a worker that stops renewing is indistinguishable from one that died.
|
|
96
|
-
The response answers per token:
|
|
97
|
-
|
|
98
|
-
| list | what it means | what this worker does |
|
|
99
|
-
|---|---|---|
|
|
100
|
-
| `renewed` | still yours | carry on |
|
|
101
|
-
| `lost` | already someone else's | stop; do **not** release — that would bump the new holder's claim |
|
|
102
|
-
| `cancelled` | still yours, nobody wants it | abort the script and release the claim |
|
|
103
|
-
|
|
104
|
-
`cancelled` is how an operator's `genctl cancel` reaches work already running: the evaluation
|
|
105
|
-
is aborted through an `AbortSignal`, its realm is terminated, and no outcome is submitted —
|
|
106
|
-
the process is stopping, so there is nothing left to answer. `renew_before_ms` on the claim
|
|
107
|
-
says how long the worker may wait before renewing again, so the interval is a value it reads
|
|
108
|
-
rather than one it guesses.
|
|
109
|
-
|
|
110
|
-
## The task input
|
|
23
|
+
npm i -D @genroc/eval-node
|
|
111
24
|
|
|
112
|
-
|
|
25
|
+
Register the resolver in the `.genroc` beside your definitions:
|
|
113
26
|
|
|
114
|
-
```
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
|
|
27
|
+
```yaml
|
|
28
|
+
resolvers:
|
|
29
|
+
- name: import
|
|
30
|
+
phase: typed
|
|
31
|
+
ext: [.ts]
|
|
32
|
+
command: [npx, genroc-import]
|
|
33
|
+
types:
|
|
34
|
+
Input: task.action.input.input
|
|
35
|
+
Output: task.action.result
|
|
120
36
|
```
|
|
121
37
|
|
|
122
|
-
|
|
123
|
-
`
|
|
124
|
-
top level. A module exporting anything else — or nothing — is a `compile_error`: there is
|
|
125
|
-
nothing to call, and so is one that will not parse; a throw from its top level is a `threw`
|
|
126
|
-
like any other. `import` of a node builtin resolves as it does anywhere else, and that is what
|
|
127
|
-
a bundled builtin lands on.
|
|
128
|
-
|
|
129
|
-
## The answer — the failure kind IS the error code
|
|
130
|
-
|
|
131
|
-
genroc's `on_error` matches error codes, and an external task's `raises` declares the closed
|
|
132
|
-
set a worker may send. So the classification lives in the **code**, where a rule can match it,
|
|
133
|
-
rather than in a body a `switch` has to read.
|
|
134
|
-
|
|
135
|
-
| outcome | the worker submits | |
|
|
136
|
-
|---|---|---|
|
|
137
|
-
| the script returned | `result: <the return value>` | `return;` sends nothing, which genroc reads as `null` |
|
|
138
|
-
| the script faulted | `error: {code, message, data}` | `code` is the kind, below |
|
|
139
|
-
| the **evaluator** faulted | *nothing* — it releases the claim | the task returns to the queue for another worker |
|
|
140
|
-
|
|
141
|
-
The five kinds, every one of them **permanent** — a retry re-runs the same code on the same
|
|
142
|
-
input and fails identically:
|
|
143
|
-
|
|
144
|
-
`compile_error` · `threw` · `timeout` · `nonserializable` · `exited`
|
|
145
|
-
|
|
146
|
-
`data` carries `{name, stack?}`: `name` is what a script sets to tell one refusal from another
|
|
147
|
-
(`e.name = 'LimitExceeded'`), and `stack` is renumbered to the lines the author wrote and
|
|
148
|
-
trimmed to 2 KiB.
|
|
149
|
-
|
|
150
|
-
**The retryable class has no code on purpose.** A runner that faults releases its claim, which
|
|
151
|
-
is how a queue spells "try this somewhere else" — it puts the task in front of a different
|
|
152
|
-
worker instead of spending the definition's `on_error` budget on this one's bad day. There is
|
|
153
|
-
no retry policy to write.
|
|
154
|
-
|
|
155
|
-
## The genroc side
|
|
38
|
+
Then reference a script from a task. The simplest way is to spread the `script-node` child that
|
|
39
|
+
`genctl init --eval-node` generates:
|
|
156
40
|
|
|
157
41
|
```yaml
|
|
158
|
-
- id:
|
|
42
|
+
- id: greet
|
|
159
43
|
action:
|
|
160
|
-
type:
|
|
44
|
+
type: child
|
|
45
|
+
<<: "$process: ./script-node.genroc.yaml"
|
|
161
46
|
input:
|
|
162
|
-
code:
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
}
|
|
169
|
-
return { fee: input.amount * 0.1 };
|
|
170
|
-
}
|
|
171
|
-
input: "$: input"
|
|
172
|
-
result_schema: { type: object, properties: { fee: { type: number } }, required: [fee] }
|
|
173
|
-
raises:
|
|
174
|
-
threw: { $ref: "#/$defs/script_error" }
|
|
175
|
-
timeout: { $ref: "#/$defs/script_error" }
|
|
176
|
-
compile_error: { $ref: "#/$defs/script_error" }
|
|
177
|
-
nonserializable: { $ref: "#/$defs/script_error" }
|
|
178
|
-
exited: { $ref: "#/$defs/script_error" }
|
|
179
|
-
timeout: 30s
|
|
180
|
-
on_error:
|
|
181
|
-
- code: [threw]
|
|
182
|
-
goto: $script_failed
|
|
183
|
-
- code: [compile_error, nonserializable, exited]
|
|
184
|
-
panic: { code: script_broken, message: "the script did not run: ${error.code}" }
|
|
185
|
-
switch: [{ goto: end }]
|
|
186
|
-
|
|
187
|
-
- id: script_failed
|
|
188
|
-
switch:
|
|
189
|
-
- case: 'last_error.data.name == "LimitExceeded"'
|
|
190
|
-
raise: { code: limit_exceeded, message: "the script rejected the amount" }
|
|
191
|
-
- raise: { code: script_failed, message: "the script failed" }
|
|
192
|
-
```
|
|
193
|
-
|
|
194
|
-
**A script cannot name a genroc error code.** `raise`/`panic` codes are literals, never
|
|
195
|
-
expressions, so the mapping from a thrown error to an authored code is this `switch` — one
|
|
196
|
-
task the definition owns, reading `error.data`. That is the whole error protocol: the worker
|
|
197
|
-
classifies into a code, the definition names the outcome.
|
|
198
|
-
|
|
199
|
-
Set the task `timeout` **above** `timeout_ms`. If the task's deadline fires first the code is
|
|
200
|
-
`external.timeout`, which is in `errcode.Unknowable()` — permanently unretryable on an
|
|
201
|
-
`only_once` task, and indistinguishable from no evaluator running at all.
|
|
202
|
-
|
|
203
|
-
`only_once` is worth considering if your scripts have side effects: without it a worker that
|
|
204
|
-
dies mid-script has its task re-claimed and the script runs again. With it the task is never
|
|
205
|
-
handed out twice, and the instance gets a catchable `external.lost` instead.
|
|
206
|
-
|
|
207
|
-
## `${` must be escaped as `$${` — when the code is inline
|
|
208
|
-
|
|
209
|
-
An external task's `input` is a Shape, so `${…}` is genroc's interpolation marker and a JS
|
|
210
|
-
template literal inside `code` is read by genroc rather than passed through. Write `` `<$${x}>` ``.
|
|
211
|
-
A leading `$:` on the code string needs `$$:` for the same reason. See
|
|
212
|
-
[specs/typed-values.md](../specs/typed-values.md). Moving the code into a `.ts` file
|
|
213
|
-
removes this entirely — see the next section.
|
|
214
|
-
|
|
215
|
-
## `import.ts` — the author-time half
|
|
216
|
-
|
|
217
|
-
`import.ts` is the **code-phase resolver** genctl runs before a definition is applied. It
|
|
218
|
-
never touches the queue and the worker never runs it; the two halves share this package only
|
|
219
|
-
because they share a calling convention, which is exactly the coupling that breaks silently
|
|
220
|
-
if they version apart.
|
|
221
|
-
|
|
222
|
-
Register it in the project's `.genroc`:
|
|
223
|
-
|
|
224
|
-
```yaml
|
|
225
|
-
resolvers:
|
|
226
|
-
import: { phase: code, ext: .ts, command: [node, ../eval-node/import.ts] }
|
|
47
|
+
code: "$import: ./greet.ts"
|
|
48
|
+
input: { who: "$: input.who" }
|
|
49
|
+
result_schema:
|
|
50
|
+
type: object
|
|
51
|
+
properties: { greeting: { type: string } }
|
|
52
|
+
required: [greeting]
|
|
227
53
|
```
|
|
228
54
|
|
|
229
|
-
|
|
55
|
+
## Writing a script
|
|
230
56
|
|
|
231
|
-
|
|
232
|
-
|
|
233
|
-
code: "$import: ./fee.ts"
|
|
234
|
-
input: "$: input"
|
|
235
|
-
```
|
|
57
|
+
A script is an ES module. Its default export receives `input`, and its return value becomes the
|
|
58
|
+
task result. The function may be async.
|
|
236
59
|
|
|
237
60
|
```ts
|
|
238
|
-
import type { Input, Output } from "./
|
|
61
|
+
import type { Input, Output } from "./greet.genroc";
|
|
239
62
|
|
|
240
|
-
export default
|
|
241
|
-
return {
|
|
63
|
+
export default function (input: Input): Output {
|
|
64
|
+
return { greeting: `hello, ${input.who}` };
|
|
242
65
|
}
|
|
243
66
|
```
|
|
244
67
|
|
|
245
|
-
`genctl
|
|
246
|
-
|
|
247
|
-
|
|
248
|
-
|
|
249
|
-
|
|
250
|
-
|
|
68
|
+
* `genctl generate` writes `greet.genroc.d.ts` beside the script. `Input` and `Output` come from
|
|
69
|
+
the definition, so a mismatch fails the typecheck.
|
|
70
|
+
* `genctl apply` typechecks and bundles. **A type error or a missing default export fails the
|
|
71
|
+
apply.**
|
|
72
|
+
* Imported files and npm packages are inlined into the bundle, so a stored definition never
|
|
73
|
+
depends on what is installed later. Node builtins (`node:fs`, …) stay as imports and resolve
|
|
74
|
+
on the worker.
|
|
75
|
+
* The generated config extends your nearest `tsconfig.json`. To use Node globals and
|
|
76
|
+
builtins, add `"types": ["node"]` to it.
|
|
251
77
|
|
|
252
|
-
|
|
253
|
-
wraps or rewrites it, so what the editor checks and what the realm runs are the same module, and
|
|
254
|
-
**a missing default export is a failed apply** rather than a fault at run time. Imports resolve
|
|
255
|
-
through TypeScript under the same config the check ran with, so a `paths` alias that typechecks
|
|
256
|
-
also bundles. They are inlined at build time, so the string a definition version stores is
|
|
257
|
-
self-contained forever — with one exception: **node builtins stay as imports**, which the realm
|
|
258
|
-
resolves. A package is frozen into the definition; `node:fs` is resolved by whatever runner
|
|
259
|
-
executes it.
|
|
260
|
-
|
|
261
|
-
### Your tsconfig, your types
|
|
78
|
+
## Running the worker
|
|
262
79
|
|
|
263
|
-
|
|
264
|
-
|
|
265
|
-
|
|
80
|
+
```sh
|
|
81
|
+
docker run -e GENROC_SERVER=http://host:8448 ghcr.io/genroc/eval-node:latest
|
|
82
|
+
# or
|
|
83
|
+
GENROC_SERVER=http://localhost:8448 npx genroc-eval-node
|
|
84
|
+
```
|
|
266
85
|
|
|
267
|
-
|
|
86
|
+
The worker connects to genroc, not the other way round, so it only needs outbound access.
|
|
268
87
|
|
|
269
|
-
|
|
|
88
|
+
| env | default | |
|
|
270
89
|
|---|---|---|
|
|
271
|
-
| `
|
|
272
|
-
| `
|
|
273
|
-
| `
|
|
274
|
-
|
|
275
|
-
|
|
276
|
-
|
|
277
|
-
|
|
278
|
-
|
|
279
|
-
|
|
280
|
-
|
|
281
|
-
|
|
282
|
-
|
|
283
|
-
|
|
284
|
-
|
|
285
|
-
|
|
286
|
-
|
|
287
|
-
|
|
288
|
-
|
|
289
|
-
|
|
290
|
-
|
|
291
|
-
|
|
292
|
-
|
|
293
|
-
|
|
294
|
-
|
|
295
|
-
|
|
296
|
-
|
|
297
|
-
|
|
298
|
-
|
|
299
|
-
|
|
300
|
-
|
|
301
|
-
|
|
302
|
-
|
|
303
|
-
|
|
304
|
-
|
|
305
|
-
|
|
306
|
-
|
|
307
|
-
|
|
308
|
-
|
|
309
|
-
|
|
310
|
-
|
|
311
|
-
|
|
312
|
-
|
|
313
|
-
|
|
314
|
-
|
|
315
|
-
|
|
316
|
-
the
|
|
317
|
-
|
|
318
|
-
|
|
319
|
-
|
|
320
|
-
|
|
321
|
-
|
|
322
|
-
|
|
323
|
-
|
|
324
|
-
|
|
325
|
-
|
|
326
|
-
|
|
327
|
-
the subprocess strategy is for — `eval.ts` keeps HTTP out precisely so it can be swapped
|
|
328
|
-
underneath.
|
|
329
|
-
- **Concurrency is capped by this worker, not by genroc.** Each evaluation is a thread, and
|
|
330
|
-
`CONCURRENCY` is how many it will claim at once. Raising it past what the host can run turns
|
|
331
|
-
a queue back into threads fighting over a core.
|
|
332
|
-
- **Not where imports and type checking happen.** The evaluator takes one self-contained
|
|
333
|
-
module and knows nothing about TypeScript; `import.ts` is what bundles an author's script and
|
|
334
|
-
its dependencies into that module, at author time. See above.
|
|
335
|
-
- **`eval.ts` and `realm.ts` know nothing about genroc.** `worker.ts` is the entire
|
|
336
|
-
queue-facing half, which is what keeps the containment strategy swappable — and what lets
|
|
337
|
-
the realm's own properties be tested by calling `evaluate()` directly.
|
|
90
|
+
| `GENROC_SERVER` | `http://localhost:8448` | the genroc server |
|
|
91
|
+
| `GENROC_TOKEN` / `GENROC_TOKEN_FILE` | *(none)* | token, or a file holding it, when the server uses `--auth token` |
|
|
92
|
+
| `PROCESS` / `TASK` | `script-node` / `eval_node` | claim only this process / task id; the defaults are script-node's. Set `PROCESS` for a renamed copy, or `""` to drop a filter; **with both `""`, the worker claims every external task.** |
|
|
93
|
+
| `CONCURRENCY` | `4` | scripts run at once |
|
|
94
|
+
| `LEASE_MS` | `30000` | claim lease, renewed every third of it |
|
|
95
|
+
| `POLL_MS` | `250` | idle poll interval |
|
|
96
|
+
| `WORKER_ID` | `evaluator-<pid>` | the claim holder's name |
|
|
97
|
+
|
|
98
|
+
With auth enabled, create a token that has only the `worker` permission:
|
|
99
|
+
|
|
100
|
+
genctl token create --perms worker --label evaluator -q
|
|
101
|
+
|
|
102
|
+
If the token is rejected, the worker exits instead of retrying.
|
|
103
|
+
|
|
104
|
+
## Errors
|
|
105
|
+
|
|
106
|
+
When a script fails, the worker reports one of five error codes for your `on_error` rules to
|
|
107
|
+
match. Retrying does not help with any of them, because the same code fails the same way:
|
|
108
|
+
|
|
109
|
+
| code | when |
|
|
110
|
+
|---|---|
|
|
111
|
+
| `threw` | the script threw or rejected |
|
|
112
|
+
| `timeout` | it ran past the task's `timeout` (5000 ms when the task has none) |
|
|
113
|
+
| `compile_error` | the module does not parse, or has no default-export function |
|
|
114
|
+
| `nonserializable` | the return value cannot be serialized to JSON (a cycle, a `BigInt`) |
|
|
115
|
+
| `exited` | the script called `process.exit()` |
|
|
116
|
+
|
|
117
|
+
`error.data` is `{name, stack?}`. `name` is the error's class name, so throwing your own
|
|
118
|
+
`class LimitExceeded extends Error {}` tells your failures apart; setting `e.name` overrides it,
|
|
119
|
+
and a thrown non-`Error` is `Thrown`. The stack is mapped back to your source lines.
|
|
120
|
+
|
|
121
|
+
If the worker itself faults, it releases the task for another worker and reports no error.
|
|
122
|
+
|
|
123
|
+
The task's `timeout` is the script's budget, waiting in the queue included. The worker stops
|
|
124
|
+
the script just before it and answers `timeout`, so `external.timeout` means no worker answered
|
|
125
|
+
at all. With `script-node`, set it through the `timeout_ms` input.
|
|
126
|
+
|
|
127
|
+
## Limits
|
|
128
|
+
|
|
129
|
+
* **Not a sandbox.** Scripts have the worker host's filesystem, network and environment.
|
|
130
|
+
Run only code you trust.
|
|
131
|
+
* **Not deterministic.** On a retry the script runs again with the real clock and random
|
|
132
|
+
numbers. Pass any value that must stay stable through `input`.
|
|
133
|
+
* **Memory is not isolated.** A script that runs out of memory takes down the whole worker.
|
|
134
|
+
* **Run-once is opt-in.** If a worker dies mid-script, the task is claimed again and the script
|
|
135
|
+
runs again. Mark the task `only_once` if your script has side effects.
|
|
136
|
+
* **Inline code needs escaping.** When you write `code` directly in YAML rather than
|
|
137
|
+
`$import`, write `${` as `$${`, because genroc reads `${` as interpolation. Scripts in `.ts`
|
|
138
|
+
files need no escaping.
|
|
139
|
+
|
|
140
|
+
## Links
|
|
141
|
+
|
|
142
|
+
* [TypeScript evaluation guide](https://genroc.org/guides/process-definition/typescript-evaluation/)
|
|
143
|
+
* [External tasks and the queue](https://genroc.org/guides/process-definition/external-tasks/)
|
|
144
|
+
* [Error handling](https://genroc.org/guides/process-definition/error-handling/)
|
|
145
|
+
* [Source on GitHub](https://github.com/genroc/genroc/tree/main/eval-node)
|
package/dist/import.js
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
|
-
// The
|
|
3
|
-
// with a non-zero exit (specs/source-resolution.md). Two modes share one `tsc` pass: "
|
|
4
|
-
// writes declarations only, "
|
|
2
|
+
// The typed-phase resolver: manifest on stdin, `{"values": [...]}` on stdout, diagnostic on stderr
|
|
3
|
+
// with a non-zero exit (specs/source-resolution.md). Two modes share one `tsc` pass: "generate"
|
|
4
|
+
// writes declarations only, "resolve" typechecks and bundles.
|
|
5
5
|
import { existsSync } from "node:fs";
|
|
6
6
|
import { access, writeFile } from "node:fs/promises";
|
|
7
7
|
import { builtinModules } from "node:module";
|
|
@@ -116,7 +116,7 @@ function declarations(at) {
|
|
|
116
116
|
emitted.push(`export type ${identifier(next)} = ${body};`);
|
|
117
117
|
}
|
|
118
118
|
return [
|
|
119
|
-
"// Generated by genroc. Do not edit - regenerate with `genctl
|
|
119
|
+
"// Generated by genroc. Do not edit - regenerate with `genctl generate`.",
|
|
120
120
|
`// ${where.name} (${address(site.pointer)})`,
|
|
121
121
|
"",
|
|
122
122
|
...emitted,
|
|
@@ -307,11 +307,14 @@ function address(pointer) {
|
|
|
307
307
|
.join("")
|
|
308
308
|
.replace(/^\./, "");
|
|
309
309
|
}
|
|
310
|
-
const located = manifest.processes.flatMap((where) => where.sites.map((site) =>
|
|
311
|
-
site
|
|
312
|
-
|
|
313
|
-
|
|
314
|
-
})
|
|
310
|
+
const located = manifest.processes.flatMap((where) => where.sites.map((site) => {
|
|
311
|
+
const [script, ...rest] = site.args;
|
|
312
|
+
if (script === undefined || rest.length > 0) {
|
|
313
|
+
die(`${join(where.dir, where.file)}: ${address(site.pointer)}: $import takes one script, ` +
|
|
314
|
+
`and is given ${site.args.length} arguments`);
|
|
315
|
+
}
|
|
316
|
+
return { site, where, file: resolve(where.dir, script) };
|
|
317
|
+
}));
|
|
315
318
|
// The `code`-field contract is this resolver's, not genctl's, so it checks the directive landed
|
|
316
319
|
// there: in a child call (the scaffold's shape) or an external task's input.
|
|
317
320
|
for (const at of located) {
|
|
@@ -328,7 +331,7 @@ for (const at of located) {
|
|
|
328
331
|
for (const at of located) {
|
|
329
332
|
if (!existsSync(at.file)) {
|
|
330
333
|
die(`${join(at.where.dir, at.where.file)}: ${address(at.site.pointer)}: ` +
|
|
331
|
-
`"$import: ${at.site.
|
|
334
|
+
`"$import: ${at.site.args[0]}" names no file (looked at ${at.file})`);
|
|
332
335
|
}
|
|
333
336
|
}
|
|
334
337
|
// One script at two sites with different input types is a refusal, not a union: the union
|
|
@@ -347,13 +350,12 @@ for (const at of located) {
|
|
|
347
350
|
for (const at of byPath.values()) {
|
|
348
351
|
await writeFile(typesPathFor(at.file), declarations(at));
|
|
349
352
|
}
|
|
350
|
-
if (manifest.mode === "
|
|
351
|
-
console.log(JSON.stringify(manifest, null, 2));
|
|
353
|
+
if (manifest.mode === "generate") {
|
|
352
354
|
process.exit(0);
|
|
353
355
|
}
|
|
354
356
|
await typecheck([...byPath.values()]);
|
|
355
|
-
const
|
|
357
|
+
const values = [];
|
|
356
358
|
for (const at of located) {
|
|
357
|
-
|
|
359
|
+
values.push(await bundle(at));
|
|
358
360
|
}
|
|
359
|
-
process.stdout.write(JSON.stringify({
|
|
361
|
+
process.stdout.write(JSON.stringify({ values }));
|
package/dist/realm.js
CHANGED
|
@@ -36,7 +36,10 @@ function scriptStack(err) {
|
|
|
36
36
|
}
|
|
37
37
|
function describe(err, kind) {
|
|
38
38
|
if (err instanceof Error) {
|
|
39
|
-
|
|
39
|
+
// A subclass inherits `name` from Error.prototype, so `class LimitExceeded extends Error {}`
|
|
40
|
+
// would report "Error"; a name the script set still wins.
|
|
41
|
+
const name = err.name === "Error" && err.constructor.name ? err.constructor.name : err.name;
|
|
42
|
+
return { kind, name, message: err.message, stack: scriptStack(err) };
|
|
40
43
|
}
|
|
41
44
|
// A script may throw a non-Error (`throw {code: "x"}`), so name/message must not assume one.
|
|
42
45
|
return { kind, name: "Thrown", message: safeText(err) };
|
package/dist/worker.js
CHANGED
|
@@ -19,8 +19,13 @@ const POLL_MS = Number(process.env.POLL_MS ?? 250);
|
|
|
19
19
|
// should return its task quickly rather than holding it for the whole budget.
|
|
20
20
|
const LEASE_MS = Number(process.env.LEASE_MS ?? 30_000);
|
|
21
21
|
const RENEW_MS = Math.max(1_000, Math.floor(LEASE_MS / 3));
|
|
22
|
-
|
|
23
|
-
|
|
22
|
+
// The answer has to land before the task's deadline fires, or a script that overran surfaces as
|
|
23
|
+
// external.timeout ("nobody answered") rather than as its own `timeout`.
|
|
24
|
+
const ANSWER_MARGIN_MS = 250;
|
|
25
|
+
// script-node's names. A renamed copy needs PROCESS set; an explicit "" drops that filter, and
|
|
26
|
+
// both "" claims every external task, other runtimes' included.
|
|
27
|
+
const PROCESS_FILTER = process.env.PROCESS ?? "script-node";
|
|
28
|
+
const TASK_FILTER = process.env.TASK ?? "eval_node";
|
|
24
29
|
// Large values arrive as refs; a bundle is one object per definition version, fetched once.
|
|
25
30
|
// Refs are content hashes, so the cache never invalidates.
|
|
26
31
|
const objectCache = new Map();
|
|
@@ -94,17 +99,13 @@ async function call(path, body) {
|
|
|
94
99
|
}
|
|
95
100
|
/** A task input that is not an EvalRequest is the definition's fault, reported as compile_error:
|
|
96
101
|
* the nearest permanent kind, since no retry fixes the definition. */
|
|
97
|
-
function asEvalRequest(input) {
|
|
102
|
+
function asEvalRequest(input, timeoutMs) {
|
|
98
103
|
if (typeof input !== "object" || input === null)
|
|
99
104
|
return "the task input is not an object";
|
|
100
105
|
const r = input;
|
|
101
106
|
if (typeof r.code !== "string")
|
|
102
107
|
return "the task input has no `code` string";
|
|
103
|
-
return {
|
|
104
|
-
code: r.code,
|
|
105
|
-
input: r.input,
|
|
106
|
-
timeout_ms: typeof r.timeout_ms === "number" ? r.timeout_ms : undefined,
|
|
107
|
-
};
|
|
108
|
+
return { code: r.code, input: r.input, timeout_ms: timeoutMs };
|
|
108
109
|
}
|
|
109
110
|
const inFlight = new Map();
|
|
110
111
|
let running = true;
|
|
@@ -161,6 +162,9 @@ async function answer(token, outcome) {
|
|
|
161
162
|
await release(token);
|
|
162
163
|
}
|
|
163
164
|
async function run(job, signal) {
|
|
165
|
+
// The task's own timeout is the script's budget. Taken on arrival, because fetching the
|
|
166
|
+
// objects below spends the same deadline.
|
|
167
|
+
const due = job.deadline_in_ms === undefined ? undefined : performance.now() + job.deadline_in_ms;
|
|
164
168
|
let resolved;
|
|
165
169
|
try {
|
|
166
170
|
resolved = await resolveObjects(job);
|
|
@@ -172,7 +176,16 @@ async function run(job, signal) {
|
|
|
172
176
|
await release(job.token);
|
|
173
177
|
return;
|
|
174
178
|
}
|
|
175
|
-
|
|
179
|
+
let timeoutMs;
|
|
180
|
+
if (due !== undefined) {
|
|
181
|
+
timeoutMs = Math.floor(due - performance.now() - ANSWER_MARGIN_MS);
|
|
182
|
+
if (timeoutMs <= 0) {
|
|
183
|
+
// Too late to run anything: the deadline fires as external.timeout, which is the truth.
|
|
184
|
+
await release(job.token);
|
|
185
|
+
return;
|
|
186
|
+
}
|
|
187
|
+
}
|
|
188
|
+
const req = asEvalRequest(resolved, timeoutMs);
|
|
176
189
|
if (typeof req === "string") {
|
|
177
190
|
await answer(job.token, {
|
|
178
191
|
error: { code: "compile_error", message: req, data: { name: "BadTaskInput" } },
|
package/import.ts
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
|
-
// The
|
|
3
|
-
// with a non-zero exit (specs/source-resolution.md). Two modes share one `tsc` pass: "
|
|
4
|
-
// writes declarations only, "
|
|
2
|
+
// The typed-phase resolver: manifest on stdin, `{"values": [...]}` on stdout, diagnostic on stderr
|
|
3
|
+
// with a non-zero exit (specs/source-resolution.md). Two modes share one `tsc` pass: "generate"
|
|
4
|
+
// writes declarations only, "resolve" typechecks and bundles.
|
|
5
5
|
|
|
6
6
|
import { existsSync } from "node:fs";
|
|
7
7
|
import { access, writeFile } from "node:fs/promises";
|
|
@@ -25,9 +25,9 @@ type Site = {
|
|
|
25
25
|
child?: string;
|
|
26
26
|
// The slot as a path of keys and indices, so nothing has to unescape a JSON Pointer.
|
|
27
27
|
pointer: (string | number)[];
|
|
28
|
-
//
|
|
29
|
-
// this resolver's reading of
|
|
30
|
-
|
|
28
|
+
// The argument's words, split as sh splits single quotes. genctl does not read them as paths —
|
|
29
|
+
// that is this resolver's reading of them.
|
|
30
|
+
args: string[];
|
|
31
31
|
// What .genroc asked genctl to type, keyed by the name it chose. This resolver reads Input
|
|
32
32
|
// and Output; which ADDRESS each came from is the config's business, not ours.
|
|
33
33
|
types?: Record<string, Schema>;
|
|
@@ -45,8 +45,7 @@ type ManifestProcess = {
|
|
|
45
45
|
};
|
|
46
46
|
|
|
47
47
|
type Manifest = {
|
|
48
|
-
mode: "
|
|
49
|
-
root: string;
|
|
48
|
+
mode: "generate" | "resolve";
|
|
50
49
|
processes: ManifestProcess[];
|
|
51
50
|
};
|
|
52
51
|
|
|
@@ -170,7 +169,7 @@ function declarations(at: Located): string {
|
|
|
170
169
|
}
|
|
171
170
|
|
|
172
171
|
return [
|
|
173
|
-
"// Generated by genroc. Do not edit - regenerate with `genctl
|
|
172
|
+
"// Generated by genroc. Do not edit - regenerate with `genctl generate`.",
|
|
174
173
|
`// ${where.name} (${address(site.pointer)})`,
|
|
175
174
|
"",
|
|
176
175
|
...emitted,
|
|
@@ -401,11 +400,16 @@ function address(pointer: (string | number)[]): string {
|
|
|
401
400
|
.replace(/^\./, "");
|
|
402
401
|
}
|
|
403
402
|
const located: Located[] = manifest.processes.flatMap((where) =>
|
|
404
|
-
where.sites.map((site) =>
|
|
405
|
-
site
|
|
406
|
-
|
|
407
|
-
|
|
408
|
-
|
|
403
|
+
where.sites.map((site) => {
|
|
404
|
+
const [script, ...rest] = site.args;
|
|
405
|
+
if (script === undefined || rest.length > 0) {
|
|
406
|
+
die(
|
|
407
|
+
`${join(where.dir, where.file)}: ${address(site.pointer)}: $import takes one script, ` +
|
|
408
|
+
`and is given ${site.args.length} arguments`,
|
|
409
|
+
);
|
|
410
|
+
}
|
|
411
|
+
return { site, where, file: resolve(where.dir, script) };
|
|
412
|
+
}),
|
|
409
413
|
);
|
|
410
414
|
|
|
411
415
|
// The `code`-field contract is this resolver's, not genctl's, so it checks the directive landed
|
|
@@ -428,7 +432,7 @@ for (const at of located) {
|
|
|
428
432
|
if (!existsSync(at.file)) {
|
|
429
433
|
die(
|
|
430
434
|
`${join(at.where.dir, at.where.file)}: ${address(at.site.pointer)}: ` +
|
|
431
|
-
`"$import: ${at.site.
|
|
435
|
+
`"$import: ${at.site.args[0]}" names no file (looked at ${at.file})`,
|
|
432
436
|
);
|
|
433
437
|
}
|
|
434
438
|
}
|
|
@@ -455,15 +459,14 @@ for (const at of byPath.values()) {
|
|
|
455
459
|
await writeFile(typesPathFor(at.file), declarations(at));
|
|
456
460
|
}
|
|
457
461
|
|
|
458
|
-
if (manifest.mode === "
|
|
459
|
-
console.log(JSON.stringify(manifest, null, 2));
|
|
462
|
+
if (manifest.mode === "generate") {
|
|
460
463
|
process.exit(0);
|
|
461
464
|
}
|
|
462
465
|
|
|
463
466
|
await typecheck([...byPath.values()]);
|
|
464
467
|
|
|
465
|
-
const
|
|
468
|
+
const values: string[] = [];
|
|
466
469
|
for (const at of located) {
|
|
467
|
-
|
|
470
|
+
values.push(await bundle(at));
|
|
468
471
|
}
|
|
469
|
-
process.stdout.write(JSON.stringify({
|
|
472
|
+
process.stdout.write(JSON.stringify({ values }));
|
package/package.json
CHANGED
package/realm.ts
CHANGED
|
@@ -35,7 +35,10 @@ function scriptStack(err: unknown): string | undefined {
|
|
|
35
35
|
|
|
36
36
|
function describe(err: unknown, kind: FailureKind): EvalFailure {
|
|
37
37
|
if (err instanceof Error) {
|
|
38
|
-
|
|
38
|
+
// A subclass inherits `name` from Error.prototype, so `class LimitExceeded extends Error {}`
|
|
39
|
+
// would report "Error"; a name the script set still wins.
|
|
40
|
+
const name = err.name === "Error" && err.constructor.name ? err.constructor.name : err.name;
|
|
41
|
+
return { kind, name, message: err.message, stack: scriptStack(err) };
|
|
39
42
|
}
|
|
40
43
|
// A script may throw a non-Error (`throw {code: "x"}`), so name/message must not assume one.
|
|
41
44
|
return { kind, name: "Thrown", message: safeText(err) };
|
package/worker.ts
CHANGED
|
@@ -22,8 +22,13 @@ const POLL_MS = Number(process.env.POLL_MS ?? 250);
|
|
|
22
22
|
// should return its task quickly rather than holding it for the whole budget.
|
|
23
23
|
const LEASE_MS = Number(process.env.LEASE_MS ?? 30_000);
|
|
24
24
|
const RENEW_MS = Math.max(1_000, Math.floor(LEASE_MS / 3));
|
|
25
|
-
|
|
26
|
-
|
|
25
|
+
// The answer has to land before the task's deadline fires, or a script that overran surfaces as
|
|
26
|
+
// external.timeout ("nobody answered") rather than as its own `timeout`.
|
|
27
|
+
const ANSWER_MARGIN_MS = 250;
|
|
28
|
+
// script-node's names. A renamed copy needs PROCESS set; an explicit "" drops that filter, and
|
|
29
|
+
// both "" claims every external task, other runtimes' included.
|
|
30
|
+
const PROCESS_FILTER = process.env.PROCESS ?? "script-node";
|
|
31
|
+
const TASK_FILTER = process.env.TASK ?? "eval_node";
|
|
27
32
|
|
|
28
33
|
type ObjectEntry = { path: (string | number)[]; ref: string; size: number };
|
|
29
34
|
|
|
@@ -34,6 +39,7 @@ type QueueTask = {
|
|
|
34
39
|
external_input: unknown;
|
|
35
40
|
objects?: ObjectEntry[];
|
|
36
41
|
raises?: Record<string, unknown>;
|
|
42
|
+
deadline_in_ms?: number;
|
|
37
43
|
};
|
|
38
44
|
|
|
39
45
|
// Large values arrive as refs; a bundle is one object per definition version, fetched once.
|
|
@@ -106,15 +112,11 @@ async function call(path: string, body: unknown): Promise<{ ok: boolean; status:
|
|
|
106
112
|
|
|
107
113
|
/** A task input that is not an EvalRequest is the definition's fault, reported as compile_error:
|
|
108
114
|
* the nearest permanent kind, since no retry fixes the definition. */
|
|
109
|
-
function asEvalRequest(input: unknown): EvalRequest | string {
|
|
115
|
+
function asEvalRequest(input: unknown, timeoutMs: number | undefined): EvalRequest | string {
|
|
110
116
|
if (typeof input !== "object" || input === null) return "the task input is not an object";
|
|
111
117
|
const r = input as Record<string, unknown>;
|
|
112
118
|
if (typeof r.code !== "string") return "the task input has no `code` string";
|
|
113
|
-
return {
|
|
114
|
-
code: r.code,
|
|
115
|
-
input: r.input,
|
|
116
|
-
timeout_ms: typeof r.timeout_ms === "number" ? r.timeout_ms : undefined,
|
|
117
|
-
};
|
|
119
|
+
return { code: r.code, input: r.input, timeout_ms: timeoutMs };
|
|
118
120
|
}
|
|
119
121
|
|
|
120
122
|
/** genroc cannot call us, so a cancellation arrives on our own heartbeat and reaches the
|
|
@@ -183,6 +185,9 @@ async function answer(token: string, outcome: Record<string, unknown>): Promise<
|
|
|
183
185
|
}
|
|
184
186
|
|
|
185
187
|
async function run(job: QueueTask, signal: AbortSignal): Promise<void> {
|
|
188
|
+
// The task's own timeout is the script's budget. Taken on arrival, because fetching the
|
|
189
|
+
// objects below spends the same deadline.
|
|
190
|
+
const due = job.deadline_in_ms === undefined ? undefined : performance.now() + job.deadline_in_ms;
|
|
186
191
|
let resolved: unknown;
|
|
187
192
|
try {
|
|
188
193
|
resolved = await resolveObjects(job);
|
|
@@ -193,7 +198,16 @@ async function run(job: QueueTask, signal: AbortSignal): Promise<void> {
|
|
|
193
198
|
await release(job.token);
|
|
194
199
|
return;
|
|
195
200
|
}
|
|
196
|
-
|
|
201
|
+
let timeoutMs: number | undefined;
|
|
202
|
+
if (due !== undefined) {
|
|
203
|
+
timeoutMs = Math.floor(due - performance.now() - ANSWER_MARGIN_MS);
|
|
204
|
+
if (timeoutMs <= 0) {
|
|
205
|
+
// Too late to run anything: the deadline fires as external.timeout, which is the truth.
|
|
206
|
+
await release(job.token);
|
|
207
|
+
return;
|
|
208
|
+
}
|
|
209
|
+
}
|
|
210
|
+
const req = asEvalRequest(resolved, timeoutMs);
|
|
197
211
|
if (typeof req === "string") {
|
|
198
212
|
await answer(job.token, {
|
|
199
213
|
error: { code: "compile_error", message: req, data: { name: "BadTaskInput" } },
|