@genroc/eval-node 0.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +302 -0
- package/bin/import.mjs +4 -0
- package/bin/worker.mjs +2 -0
- package/eval.ts +98 -0
- package/import.ts +395 -0
- package/package.json +40 -0
- package/realm.ts +132 -0
- package/tsconfig.json +23 -0
- package/worker.ts +300 -0
package/README.md
ADDED
|
@@ -0,0 +1,302 @@
|
|
|
1
|
+
# @genroc/eval-node
|
|
2
|
+
|
|
3
|
+
Script tasks for Node. Two halves with different jobs:
|
|
4
|
+
|
|
5
|
+
* **the bundler** (`genroc-import`) — an author-time resolver genctl runs on every apply and
|
|
6
|
+
every edit, turning `$import: ./x.ts` into a bundled, typechecked string
|
|
7
|
+
* **the worker** (`genroc-eval-node`) — claims parked `external` script tasks off genroc's queue
|
|
8
|
+
and evaluates each in its own realm
|
|
9
|
+
|
|
10
|
+
## Setting up a project
|
|
11
|
+
|
|
12
|
+
npm i -D @genroc/eval-node
|
|
13
|
+
|
|
14
|
+
Then register the resolver in a `genroc.yaml` beside your definitions — discovery walks up from
|
|
15
|
+
the file, so nothing depends on the cwd:
|
|
16
|
+
|
|
17
|
+
resolvers:
|
|
18
|
+
import:
|
|
19
|
+
phase: code
|
|
20
|
+
ext: .ts
|
|
21
|
+
command: [npx, genroc-import]
|
|
22
|
+
|
|
23
|
+
`genctl apply` and `genctl types` now resolve `$import` directives. Typechecking is the
|
|
24
|
+
resolver's exit code, so a stored definition cannot hold code that failed to typecheck.
|
|
25
|
+
|
|
26
|
+
## Running the worker
|
|
27
|
+
|
|
28
|
+
docker run -e GENROC_SERVER=http://host:8448 ghcr.io/genroc/eval-node:preview
|
|
29
|
+
|
|
30
|
+
or locally, if you already have Node:
|
|
31
|
+
|
|
32
|
+
GENROC_SERVER=http://localhost:8448 npx genroc-eval-node
|
|
33
|
+
|
|
34
|
+
Add `GENROC_TOKEN` once the server has auth on; `worker` is the only permission it needs.
|
|
35
|
+
|
|
36
|
+
**The bundler cannot be the image.** It runs per edit and must resolve your project's own
|
|
37
|
+
`node_modules`, so a container start per invocation would break the editor loop.
|
|
38
|
+
|
|
39
|
+
Design record: [specs/external-task-queue.md](../specs/external-task-queue.md) for the queue,
|
|
40
|
+
[specs/script-tasks.md](../specs/script-tasks.md) for the runtime.
|
|
41
|
+
|
|
42
|
+
GENROC_SERVER=http://localhost:8448 node eval-node/worker.ts
|
|
43
|
+
|
|
44
|
+
Against a server started with `--auth token`, mint a scoped credential first:
|
|
45
|
+
|
|
46
|
+
TOKEN=$(genctl token create --perms worker --label evaluator -q)
|
|
47
|
+
GENROC_SERVER=http://localhost:8448 GENROC_TOKEN=$TOKEN node eval-node/worker.ts
|
|
48
|
+
|
|
49
|
+
`worker` reaches the four queue verbs and `GET /api/objects/{ref}` — enough to claim, fetch an
|
|
50
|
+
externalized input and answer, and nothing else. This is the credential most likely to end up on
|
|
51
|
+
a machine you trust least, so it is worth scoping rather than reusing an admin token: a leaked
|
|
52
|
+
one cannot read a definition, list an instance, or mint another token.
|
|
53
|
+
|
|
54
|
+
A credential problem **exits the worker** rather than being retried. Polling through a 401 looks
|
|
55
|
+
like a healthy worker that never picks anything up, which is the worst shape to debug.
|
|
56
|
+
|
|
57
|
+
It needs **Node 24 or newer**: the sources are TypeScript and are run as-is, by Node's own
|
|
58
|
+
type stripping.
|
|
59
|
+
|
|
60
|
+
| env | default | |
|
|
61
|
+
|---|---|---|
|
|
62
|
+
| `GENROC_SERVER` | `http://localhost:8448` | where to claim from |
|
|
63
|
+
| `GENROC_TOKEN` | *(none)* | credential, when the server runs with `--auth token`. Needs the **`worker`** permission and nothing more |
|
|
64
|
+
| `WORKER_ID` | `evaluator-<pid>` | the claim holder; renewals are scoped to it |
|
|
65
|
+
| `CONCURRENCY` | `4` | how many scripts run at once — **this worker's** decision |
|
|
66
|
+
| `LEASE_MS` | `30000` | visibility timeout; renewed at a third of it while working |
|
|
67
|
+
| `POLL_MS` | `250` | idle poll interval; a non-empty claim polls again immediately |
|
|
68
|
+
| `PROCESS` / `TASK` | unset | claim only this process / task id |
|
|
69
|
+
|
|
70
|
+
**The worker calls genroc, not the other way round.** That is the whole reason this is a
|
|
71
|
+
queue worker rather than the HTTP sidecar it used to be:
|
|
72
|
+
|
|
73
|
+
- **Concurrency is the worker's to set.** Under the old `fetch` shape genroc decided how many
|
|
74
|
+
scripts ran at once (`--max-concurrent`, default 200) and the evaluator accepted every one.
|
|
75
|
+
A backlog is now a queue, not 200 threads fighting over a core.
|
|
76
|
+
- **No engine worker is held.** A `fetch` occupies one of genroc's advance slots for its whole
|
|
77
|
+
duration, so a slow evaluator starved unrelated tasks. An `external` task parks.
|
|
78
|
+
- **The connection direction inverts.** `/eval` was unauthenticated arbitrary code execution
|
|
79
|
+
that genroc had to be able to reach. A worker only needs outbound access, so it can live
|
|
80
|
+
anywhere — behind NAT, in another trust zone.
|
|
81
|
+
|
|
82
|
+
## The task input
|
|
83
|
+
|
|
84
|
+
The `input` of the external task IS the evaluation request:
|
|
85
|
+
|
|
86
|
+
```jsonc
|
|
87
|
+
{
|
|
88
|
+
"code": "return { fee: input.amount * 0.1 };", // required — an async function body
|
|
89
|
+
"input": { "amount": 250 }, // optional — bound as `input`
|
|
90
|
+
"timeout_ms": 5000 // optional — default 5000
|
|
91
|
+
}
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
`code` is the **body of an async function**, so `await` works and the value reaches genroc
|
|
95
|
+
through `return`. It is compiled with `input` and `require` as parameters, under
|
|
96
|
+
`"use strict"` — `require` is what a bundled `import` of a node builtin lands on.
|
|
97
|
+
|
|
98
|
+
## The answer — the failure kind IS the error code
|
|
99
|
+
|
|
100
|
+
genroc's `on_error` matches error codes, and an external task's `raises` declares the closed
|
|
101
|
+
set a worker may send. So the classification lives in the **code**, where a rule can match it,
|
|
102
|
+
rather than in a body a `switch` has to read.
|
|
103
|
+
|
|
104
|
+
| outcome | the worker submits | |
|
|
105
|
+
|---|---|---|
|
|
106
|
+
| the script returned | `result: <the return value>` | `return;` sends nothing, which genroc reads as `null` |
|
|
107
|
+
| the script faulted | `error: {code, message, data}` | `code` is the kind, below |
|
|
108
|
+
| the **evaluator** faulted | *nothing* — it releases the claim | the task returns to the queue for another worker |
|
|
109
|
+
|
|
110
|
+
The five kinds, every one of them **permanent** — a retry re-runs the same code on the same
|
|
111
|
+
input and fails identically:
|
|
112
|
+
|
|
113
|
+
`compile_error` · `threw` · `timeout` · `nonserializable` · `exited`
|
|
114
|
+
|
|
115
|
+
`data` carries `{name, stack?}`: `name` is what a script sets to tell one refusal from another
|
|
116
|
+
(`e.name = 'LimitExceeded'`), and `stack` is renumbered to the lines the author wrote and
|
|
117
|
+
trimmed to 2 KiB.
|
|
118
|
+
|
|
119
|
+
**The retryable class has no code on purpose.** A runner that faults releases its claim, which
|
|
120
|
+
is how a queue spells "try this somewhere else" — it puts the task in front of a different
|
|
121
|
+
worker instead of spending the definition's `on_error` budget on this one's bad day. There is
|
|
122
|
+
no retry policy to write.
|
|
123
|
+
|
|
124
|
+
## The genroc side
|
|
125
|
+
|
|
126
|
+
```yaml
|
|
127
|
+
- id: price
|
|
128
|
+
action:
|
|
129
|
+
type: external
|
|
130
|
+
input:
|
|
131
|
+
code: |
|
|
132
|
+
if (input.amount > 100) {
|
|
133
|
+
const e = new Error('amount over the limit');
|
|
134
|
+
e.name = 'LimitExceeded';
|
|
135
|
+
throw e;
|
|
136
|
+
}
|
|
137
|
+
return { fee: input.amount * 0.1 };
|
|
138
|
+
input: "$: input"
|
|
139
|
+
result_schema: { type: object, properties: { fee: { type: number } }, required: [fee] }
|
|
140
|
+
raises:
|
|
141
|
+
threw: { $ref: "#/$defs/script_error" }
|
|
142
|
+
timeout: { $ref: "#/$defs/script_error" }
|
|
143
|
+
compile_error: { $ref: "#/$defs/script_error" }
|
|
144
|
+
nonserializable: { $ref: "#/$defs/script_error" }
|
|
145
|
+
exited: { $ref: "#/$defs/script_error" }
|
|
146
|
+
timeout: 30s
|
|
147
|
+
on_error:
|
|
148
|
+
- code: [threw]
|
|
149
|
+
goto: $script_failed
|
|
150
|
+
- code: [compile_error, nonserializable, exited]
|
|
151
|
+
panic: { code: script_broken, message: "the script did not run: ${error.code}" }
|
|
152
|
+
switch: [{ goto: end }]
|
|
153
|
+
|
|
154
|
+
- id: script_failed
|
|
155
|
+
switch:
|
|
156
|
+
- case: 'error.data.name == "LimitExceeded"'
|
|
157
|
+
raise: { code: limit_exceeded, message: "the script rejected the amount" }
|
|
158
|
+
- raise: { code: script_failed, message: "the script failed" }
|
|
159
|
+
```
|
|
160
|
+
|
|
161
|
+
**A script cannot name a genroc error code.** `raise`/`panic` codes are literals, never
|
|
162
|
+
expressions, so the mapping from a thrown error to an authored code is this `switch` — one
|
|
163
|
+
task the definition owns, reading `error.data`. That is the whole error protocol: the worker
|
|
164
|
+
classifies into a code, the definition names the outcome.
|
|
165
|
+
|
|
166
|
+
Set the task `timeout` **above** `timeout_ms`. If the task's deadline fires first the code is
|
|
167
|
+
`external.timeout`, which is in `errcode.Unknowable()` — permanently unretryable on an
|
|
168
|
+
`only_once` task, and indistinguishable from no evaluator running at all.
|
|
169
|
+
|
|
170
|
+
`only_once` is worth considering if your scripts have side effects: without it a worker that
|
|
171
|
+
dies mid-script has its task re-claimed and the script runs again. With it the task is never
|
|
172
|
+
handed out twice, and the instance gets a catchable `external.lost` instead.
|
|
173
|
+
|
|
174
|
+
## `${` must be escaped as `$${` — when the code is inline
|
|
175
|
+
|
|
176
|
+
An external task's `input` is a Shape, so `${…}` is genroc's interpolation marker and a JS
|
|
177
|
+
template literal inside `code` is read by genroc rather than passed through. Write `` `<$${x}>` ``.
|
|
178
|
+
A leading `$:` on the code string needs `$$:` for the same reason. See
|
|
179
|
+
[specs/typed-values.md](../specs/typed-values.md). Moving the code into a `.ts` file
|
|
180
|
+
removes this entirely — see the next section.
|
|
181
|
+
|
|
182
|
+
## `import.ts` — the author-time half
|
|
183
|
+
|
|
184
|
+
`import.ts` is the **code-phase resolver** genctl runs before a definition is applied. It
|
|
185
|
+
never touches the queue and the worker never runs it; the two halves share this package only
|
|
186
|
+
because they share a calling convention, which is exactly the coupling that breaks silently
|
|
187
|
+
if they version apart.
|
|
188
|
+
|
|
189
|
+
Register it in the project's `genroc.yaml`:
|
|
190
|
+
|
|
191
|
+
```yaml
|
|
192
|
+
resolvers:
|
|
193
|
+
import: { phase: code, ext: .ts, command: [node, ../eval-node/import.ts] }
|
|
194
|
+
```
|
|
195
|
+
|
|
196
|
+
then write the script as a module and name it from the definition:
|
|
197
|
+
|
|
198
|
+
```yaml
|
|
199
|
+
body:
|
|
200
|
+
code: "$import: ./fee.ts"
|
|
201
|
+
input: "$: input"
|
|
202
|
+
```
|
|
203
|
+
|
|
204
|
+
```ts
|
|
205
|
+
import type { Input, Output } from "./fee.genroc";
|
|
206
|
+
|
|
207
|
+
export default async function (input: Input): Promise<Output> {
|
|
208
|
+
return { fee: input.amount * 0.1 };
|
|
209
|
+
}
|
|
210
|
+
```
|
|
211
|
+
|
|
212
|
+
`genctl types -f process.yaml` writes `fee.genroc.d.ts` beside the script — named for the
|
|
213
|
+
**script's path**, not the task, so renaming a task cannot break the import line. `Input` is
|
|
214
|
+
the inferred type of what the definition passes; `Output` is what it declares
|
|
215
|
+
(`responses.200`, or `result_schema` on a child). `genctl apply` regenerates them, runs
|
|
216
|
+
`tsc --noEmit`, and bundles — so **a type error is a failed apply**, and a stored definition
|
|
217
|
+
cannot hold code that failed to typecheck.
|
|
218
|
+
|
|
219
|
+
The bundle is emitted as CJS and wrapped as a function body, so the evaluator needs to know
|
|
220
|
+
nothing about modules. Imports resolve through TypeScript under the same config the check
|
|
221
|
+
ran with, so a `paths` alias that typechecks also bundles. They are inlined at build time, so the string a
|
|
222
|
+
definition version stores is self-contained forever — with one exception: **node builtins
|
|
223
|
+
stay as `require` calls**, which the realm satisfies. A package is frozen into the
|
|
224
|
+
definition; `node:fs` is resolved by whatever runner executes it.
|
|
225
|
+
|
|
226
|
+
### Your tsconfig, your types
|
|
227
|
+
|
|
228
|
+
The generated project config `extends` **the nearest `tsconfig.json` above the script** —
|
|
229
|
+
the one your editor already reads, so the two cannot disagree. Two scripts under two
|
|
230
|
+
different configs are two `tsc` runs.
|
|
231
|
+
|
|
232
|
+
Of that config, three keys are the toolchain's and the rest are yours:
|
|
233
|
+
|
|
234
|
+
| key | owner | why |
|
|
235
|
+
|---|---|---|
|
|
236
|
+
| `lib` | the toolchain | Describes the realm. A worker thread has no `document`, whatever a config claims. |
|
|
237
|
+
| `include` | the toolchain | Forced to `[]`. A base `include` survives beside our `files` and would drag your whole tree in to be checked as scripts. |
|
|
238
|
+
| `types` | **you** | How a script opts into the node globals. The realm has them, so refusing the declarations would only lie. With no tsconfig at all the default is `[]`. |
|
|
239
|
+
|
|
240
|
+
```jsonc
|
|
241
|
+
// tsconfig.json beside your scripts
|
|
242
|
+
{ "compilerOptions": { "types": ["node"] } } // now `import { appendFile } from "node:fs/promises"` typechecks
|
|
243
|
+
```
|
|
244
|
+
|
|
245
|
+
**This is what removes the `$${` escaping above** — a template literal in a `.ts` file is
|
|
246
|
+
never read by genroc, because genctl doubles every `$` on splice.
|
|
247
|
+
|
|
248
|
+
## The realm — one Worker per execution
|
|
249
|
+
|
|
250
|
+
`evaluate()` starts a Worker (`realm.ts`), posts the code into it, and races the reply against the budget;
|
|
251
|
+
`terminate()` runs on every path. That thread is what the contract rests on, and it buys
|
|
252
|
+
exactly three things the previous in-process evaluator could not:
|
|
253
|
+
|
|
254
|
+
- **The budget is enforced, not merely reported.** A synchronous `while(true){}` never
|
|
255
|
+
yields, so no in-process timer can interrupt it — the old evaluator hung forever on that
|
|
256
|
+
input, and said so in this file. Killing the thread is the only bound. Measured on Node
|
|
257
|
+
24: `terminate()` stops a spinning worker, and the CPU it was burning goes with it.
|
|
258
|
+
- **A fresh global object per execution.** One script cannot configure the next. It is also
|
|
259
|
+
why there is no compile cache any more: a cache inside a discarded realm can never be hit.
|
|
260
|
+
- **The script's mistakes stay the script's.** An uncaught throw, and `process.exit()`, end
|
|
261
|
+
the realm and come back as a `422` — neither reaches the runner.
|
|
262
|
+
|
|
263
|
+
It costs about **50ms per execution** end to end for a trivial script (Node 24, M-series
|
|
264
|
+
laptop), a 200 KiB body about 63ms. Roughly 19ms of that is Node re-stripping `worker.ts`'s
|
|
265
|
+
types on every realm — precompiling it to JavaScript would buy that back, and is deliberately
|
|
266
|
+
not done: a build artefact that goes stale against its source fails silently, and this file
|
|
267
|
+
is the one where a wrong line number is invisible.
|
|
268
|
+
|
|
269
|
+
That also changes an old trade-off. A subprocess per execution measures ~48ms here — within
|
|
270
|
+
noise of the thread — where on the previous runtime it was ten times the thread's cost. The
|
|
271
|
+
thread no longer wins on price, and a subprocess contains the two things a thread cannot
|
|
272
|
+
(below), so it is the live upgrade path rather than a theoretical one.
|
|
273
|
+
|
|
274
|
+
## What this is not
|
|
275
|
+
|
|
276
|
+
- **Not deterministic.** A script reads the real clock and the real RNG, and a retry
|
|
277
|
+
re-executes — so attempt two can differ from attempt one. An earlier version injected a
|
|
278
|
+
pinned `Date` and a seeded `Math`; nothing could supply a stable `now` (the expression
|
|
279
|
+
environment has no clock), so the pin was the wall clock under another name, and the `ctx`
|
|
280
|
+
it needed was surface with nothing behind it. A value that must survive a retry belongs in
|
|
281
|
+
the definition, passed through `input`.
|
|
282
|
+
- **Not a sandbox.** The realm isolates *execution*, not *authority*: a script gets the
|
|
283
|
+
worker's filesystem, network and environment, and `require` of any node builtin. That is
|
|
284
|
+
deliberate — a script task is meant to do real work — but the trust boundary stays the
|
|
285
|
+
same-trust-domain one (your genroc, your worker host). It is not the multi-tenant story, and
|
|
286
|
+
nothing here should be mistaken for one. Pulling does move the boundary in one useful way:
|
|
287
|
+
the worker needs no inbound reachability, so nothing but the worker can ask it to run code.
|
|
288
|
+
- **A thread does not contain memory or a native crash.** A worker shares the process
|
|
289
|
+
address space, so a script that exhausts memory takes the runner with it — and so does a
|
|
290
|
+
fault in the runtime itself, which is not hypothetical: the previous one segfaulted on
|
|
291
|
+
resume from laptop sleep, taking the in-flight evaluation with it. Containing that is what
|
|
292
|
+
the subprocess strategy is for — `eval.ts` keeps HTTP out precisely so it can be swapped
|
|
293
|
+
underneath.
|
|
294
|
+
- **Concurrency is capped by this worker, not by genroc.** Each evaluation is a thread, and
|
|
295
|
+
`CONCURRENCY` is how many it will claim at once. Raising it past what the host can run turns
|
|
296
|
+
a queue back into threads fighting over a core.
|
|
297
|
+
- **Not where imports and type checking happen.** The evaluator still takes one
|
|
298
|
+
self-contained function body and knows nothing about TypeScript; `import.ts` is what turns
|
|
299
|
+
a module into that body, at author time. See above.
|
|
300
|
+
- **`eval.ts` and `realm.ts` know nothing about genroc.** `worker.ts` is the entire
|
|
301
|
+
queue-facing half, which is what keeps the containment strategy swappable — and what lets
|
|
302
|
+
the realm's own properties be tested by calling `evaluate()` directly.
|
package/bin/import.mjs
ADDED
package/bin/worker.mjs
ADDED
package/eval.ts
ADDED
|
@@ -0,0 +1,98 @@
|
|
|
1
|
+
// Evaluation core: run a code string in its OWN realm and classify every outcome into one of
|
|
2
|
+
// the failure kinds in README.md. Nothing here knows about genroc — worker.ts is the only
|
|
3
|
+
// thing that talks to the queue — so this stays testable and the containment stays swappable.
|
|
4
|
+
//
|
|
5
|
+
// The containment is a Worker per execution (realm.ts). It is what makes the budget real:
|
|
6
|
+
// a synchronous busy loop never yields, so no in-process timer can interrupt it, and only a
|
|
7
|
+
// thread the host can kill bounds it.
|
|
8
|
+
|
|
9
|
+
import { Worker } from "node:worker_threads";
|
|
10
|
+
|
|
11
|
+
export type EvalRequest = {
|
|
12
|
+
code: string;
|
|
13
|
+
input?: unknown;
|
|
14
|
+
timeout_ms?: number;
|
|
15
|
+
};
|
|
16
|
+
|
|
17
|
+
/** Every kind here is PERMANENT: a retry re-runs the same code on the same input and fails
|
|
18
|
+
* identically. The retryable class has no kind because it is not an outcome — a runner that
|
|
19
|
+
* faults releases its claim and lets another worker take the task. */
|
|
20
|
+
export type FailureKind = "compile_error" | "threw" | "timeout" | "nonserializable" | "exited";
|
|
21
|
+
|
|
22
|
+
export type EvalFailure = {
|
|
23
|
+
kind: FailureKind;
|
|
24
|
+
name: string;
|
|
25
|
+
message: string;
|
|
26
|
+
stack?: string;
|
|
27
|
+
};
|
|
28
|
+
|
|
29
|
+
/** `body` is JSON TEXT, not a value: serialising in the realm is what makes a nonserializable
|
|
30
|
+
* return a script fault rather than a 500 thrown out of the response path. It is also what
|
|
31
|
+
* crosses the worker boundary — structured clone would refuse a different set of values. */
|
|
32
|
+
export type EvalResult =
|
|
33
|
+
| { ok: true; body: string }
|
|
34
|
+
| { ok: false; failure: EvalFailure };
|
|
35
|
+
|
|
36
|
+
/** The message the host posts into the realm, and the only reply it accepts back. */
|
|
37
|
+
export type WorkerRequest = { code: string; input?: unknown };
|
|
38
|
+
export type WorkerReply = EvalResult;
|
|
39
|
+
|
|
40
|
+
const DEFAULT_TIMEOUT_MS = 5_000;
|
|
41
|
+
const REALM_URL = new URL("./realm.ts", import.meta.url);
|
|
42
|
+
|
|
43
|
+
/** Thrown, not returned: a realm that fails to start is the RUNNER faulting, which worker.ts
|
|
44
|
+
* answers by releasing the claim rather than by reporting an outcome. A script fault is a
|
|
45
|
+
* return value. */
|
|
46
|
+
class RealmFault extends Error {
|
|
47
|
+
constructor(message: string) {
|
|
48
|
+
super(message);
|
|
49
|
+
this.name = "RealmFault";
|
|
50
|
+
}
|
|
51
|
+
}
|
|
52
|
+
|
|
53
|
+
export async function evaluate(req: EvalRequest): Promise<EvalResult> {
|
|
54
|
+
const budget = typeof req.timeout_ms === "number" ? req.timeout_ms : DEFAULT_TIMEOUT_MS;
|
|
55
|
+
|
|
56
|
+
const worker = new Worker(REALM_URL);
|
|
57
|
+
let timer: ReturnType<typeof setTimeout> | undefined;
|
|
58
|
+
try {
|
|
59
|
+
return await new Promise<EvalResult>((resolve, reject) => {
|
|
60
|
+
timer = setTimeout(() => resolve(timedOut(budget)), budget);
|
|
61
|
+
worker.once("message", (reply: WorkerReply) => resolve(reply));
|
|
62
|
+
// A script may end its own realm (`process.exit()`), which is not a throw and would
|
|
63
|
+
// otherwise present as a hang until the budget expired. Our own terminate() raises
|
|
64
|
+
// this too, by which time the promise has settled and the first result stands.
|
|
65
|
+
worker.once("exit", (code: number) => resolve(exited(code)));
|
|
66
|
+
worker.once("error", (err: Error) => reject(new RealmFault(errorText(err))));
|
|
67
|
+
worker.postMessage({ code: req.code, input: req.input } satisfies WorkerRequest);
|
|
68
|
+
});
|
|
69
|
+
} finally {
|
|
70
|
+
clearTimeout(timer);
|
|
71
|
+
// Awaited, and the whole point: on the timeout path a thread is still burning a core, and
|
|
72
|
+
// resolving before it is gone would report an evaluation the machine is still running.
|
|
73
|
+
await worker.terminate();
|
|
74
|
+
}
|
|
75
|
+
}
|
|
76
|
+
|
|
77
|
+
function timedOut(ms: number): EvalResult {
|
|
78
|
+
return {
|
|
79
|
+
ok: false,
|
|
80
|
+
failure: { kind: "timeout", name: "TimeoutError", message: `script exceeded its ${ms}ms budget` },
|
|
81
|
+
};
|
|
82
|
+
}
|
|
83
|
+
|
|
84
|
+
function exited(code: number): EvalResult {
|
|
85
|
+
return {
|
|
86
|
+
ok: false,
|
|
87
|
+
failure: {
|
|
88
|
+
kind: "exited",
|
|
89
|
+
name: "RealmExited",
|
|
90
|
+
message: `the script ended its own realm with code ${code} instead of returning`,
|
|
91
|
+
},
|
|
92
|
+
};
|
|
93
|
+
}
|
|
94
|
+
|
|
95
|
+
function errorText(e: unknown): string {
|
|
96
|
+
const message = (e as { message?: unknown } | null)?.message;
|
|
97
|
+
return typeof message === "string" && message !== "" ? message : "the evaluation realm failed to start";
|
|
98
|
+
}
|
package/import.ts
ADDED
|
@@ -0,0 +1,395 @@
|
|
|
1
|
+
// The code-phase resolver: manifest on stdin, `{"code": [...]}` on stdout, non-zero exit
|
|
2
|
+
// with the diagnostic on stderr. genctl never parses TypeScript and this never parses YAML
|
|
3
|
+
// — the manifest is the whole contract. See specs/source-resolution.md.
|
|
4
|
+
//
|
|
5
|
+
// Two modes, one binary: "types" writes the declarations an editor needs and returns no
|
|
6
|
+
// code; "build" typechecks and bundles. A separate types hook would mean a second `tsc`
|
|
7
|
+
// over the same project.
|
|
8
|
+
|
|
9
|
+
import { spawn } from "node:child_process";
|
|
10
|
+
import { access, mkdir, writeFile } from "node:fs/promises";
|
|
11
|
+
import { builtinModules } from "node:module";
|
|
12
|
+
import { dirname, join, relative } from "node:path";
|
|
13
|
+
import { fileURLToPath } from "node:url";
|
|
14
|
+
|
|
15
|
+
import commonjs from "@rollup/plugin-commonjs";
|
|
16
|
+
import json from "@rollup/plugin-json";
|
|
17
|
+
import { nodeResolve } from "@rollup/plugin-node-resolve";
|
|
18
|
+
import { rollup, type Plugin } from "rollup";
|
|
19
|
+
import ts from "typescript";
|
|
20
|
+
|
|
21
|
+
type Schema = Record<string, any>;
|
|
22
|
+
|
|
23
|
+
type Site = {
|
|
24
|
+
resolver: string;
|
|
25
|
+
process: string;
|
|
26
|
+
task?: string;
|
|
27
|
+
pointer: string;
|
|
28
|
+
path: string;
|
|
29
|
+
input?: Schema;
|
|
30
|
+
output?: Schema;
|
|
31
|
+
};
|
|
32
|
+
|
|
33
|
+
type Manifest = {
|
|
34
|
+
mode: "types" | "build";
|
|
35
|
+
root: string;
|
|
36
|
+
schemas: Record<string, Schema>;
|
|
37
|
+
sites: Site[];
|
|
38
|
+
};
|
|
39
|
+
|
|
40
|
+
function die(message: string): never {
|
|
41
|
+
console.error(message);
|
|
42
|
+
process.exit(1);
|
|
43
|
+
}
|
|
44
|
+
|
|
45
|
+
async function exists(path: string): Promise<boolean> {
|
|
46
|
+
try {
|
|
47
|
+
await access(path);
|
|
48
|
+
return true;
|
|
49
|
+
} catch {
|
|
50
|
+
return false;
|
|
51
|
+
}
|
|
52
|
+
}
|
|
53
|
+
|
|
54
|
+
/** Creates the parent directory, which `.genroc/` relies on: nothing else makes it. */
|
|
55
|
+
async function write(path: string, content: string): Promise<void> {
|
|
56
|
+
await mkdir(dirname(path), { recursive: true });
|
|
57
|
+
await writeFile(path, content);
|
|
58
|
+
}
|
|
59
|
+
|
|
60
|
+
// ── JSON Schema → TypeScript ───────────────────────────────────────────────────
|
|
61
|
+
|
|
62
|
+
/** Only genroc's keyword set is handled; anything else its strict decoder would have
|
|
63
|
+
* refused before this ran (internal/schema, allowedKeywords). */
|
|
64
|
+
function tsType(s: Schema | undefined, used: Set<string>): string {
|
|
65
|
+
if (s === undefined || s === null) return "unknown";
|
|
66
|
+
if (typeof s.$ref === "string") {
|
|
67
|
+
const name = s.$ref.replace(/^#\/\$defs\//, "");
|
|
68
|
+
used.add(name);
|
|
69
|
+
return identifier(name);
|
|
70
|
+
}
|
|
71
|
+
if (Array.isArray(s.enum)) {
|
|
72
|
+
return s.enum.map((v: unknown) => JSON.stringify(v)).join(" | ") || "never";
|
|
73
|
+
}
|
|
74
|
+
if (Array.isArray(s.anyOf)) return union(s.anyOf.map((a: Schema) => tsType(a, used)));
|
|
75
|
+
if (Array.isArray(s.oneOf)) return union(s.oneOf.map((a: Schema) => tsType(a, used)));
|
|
76
|
+
if (Array.isArray(s.allOf)) {
|
|
77
|
+
return s.allOf.map((a: Schema) => tsType(a, used)).join(" & ") || "unknown";
|
|
78
|
+
}
|
|
79
|
+
|
|
80
|
+
const types: string[] = s.type === undefined ? [] : Array.isArray(s.type) ? s.type : [s.type];
|
|
81
|
+
if (types.length === 0) {
|
|
82
|
+
// The top type: `{}` means unknown, not "an empty object". specs/unknown-type.md.
|
|
83
|
+
return s.properties ? objectType(s, used) : "unknown";
|
|
84
|
+
}
|
|
85
|
+
return union(types.map((t) => scalarType(t, s, used)));
|
|
86
|
+
}
|
|
87
|
+
|
|
88
|
+
function scalarType(t: string, s: Schema, used: Set<string>): string {
|
|
89
|
+
switch (t) {
|
|
90
|
+
case "object":
|
|
91
|
+
return objectType(s, used);
|
|
92
|
+
case "array":
|
|
93
|
+
return s.items ? `Array<${tsType(s.items, used)}>` : "unknown[]";
|
|
94
|
+
case "string":
|
|
95
|
+
return "string";
|
|
96
|
+
case "number":
|
|
97
|
+
case "integer":
|
|
98
|
+
return "number";
|
|
99
|
+
case "boolean":
|
|
100
|
+
return "boolean";
|
|
101
|
+
case "null":
|
|
102
|
+
return "null";
|
|
103
|
+
default:
|
|
104
|
+
return "unknown";
|
|
105
|
+
}
|
|
106
|
+
}
|
|
107
|
+
|
|
108
|
+
function objectType(s: Schema, used: Set<string>): string {
|
|
109
|
+
const props: Record<string, Schema> = s.properties ?? {};
|
|
110
|
+
const required = new Set<string>(s.required ?? []);
|
|
111
|
+
const lines: string[] = [];
|
|
112
|
+
for (const [key, sub] of Object.entries(props)) {
|
|
113
|
+
const doc = typeof sub.description === "string" ? ` /** ${sub.description} */\n` : "";
|
|
114
|
+
lines.push(`${doc} ${propKey(key)}${required.has(key) ? "" : "?"}: ${tsType(sub, used)};`);
|
|
115
|
+
}
|
|
116
|
+
if (s.additionalProperties && typeof s.additionalProperties === "object") {
|
|
117
|
+
lines.push(` [key: string]: ${tsType(s.additionalProperties, used)};`);
|
|
118
|
+
}
|
|
119
|
+
if (lines.length === 0) return "Record<string, unknown>";
|
|
120
|
+
return `{\n${lines.join("\n")}\n}`;
|
|
121
|
+
}
|
|
122
|
+
|
|
123
|
+
function union(parts: string[]): string {
|
|
124
|
+
const seen = [...new Set(parts)];
|
|
125
|
+
return seen.length === 0 ? "unknown" : seen.join(" | ");
|
|
126
|
+
}
|
|
127
|
+
|
|
128
|
+
const IDENT = /^[A-Za-z_$][A-Za-z0-9_$]*$/;
|
|
129
|
+
const propKey = (k: string) => (IDENT.test(k) ? k : JSON.stringify(k));
|
|
130
|
+
const identifier = (n: string) => (IDENT.test(n) ? n : `Def_${n.replace(/[^A-Za-z0-9_$]/g, "_")}`);
|
|
131
|
+
|
|
132
|
+
function deref(s: Schema | undefined, defs: Record<string, Schema>): Schema | undefined {
|
|
133
|
+
let cur = s;
|
|
134
|
+
for (let i = 0; cur && typeof cur.$ref === "string" && i < 16; i++) {
|
|
135
|
+
cur = defs[cur.$ref.replace(/^#\/\$defs\//, "")];
|
|
136
|
+
}
|
|
137
|
+
return cur;
|
|
138
|
+
}
|
|
139
|
+
|
|
140
|
+
/** The manifest's `input` is the type of the whole ACTION input — for /eval that is
|
|
141
|
+
* `{code, input, timeout_ms, …}`, and only its `input` field is bound as the script's
|
|
142
|
+
* parameter. genroc cannot know that; this file owns the evaluator's wire contract, so
|
|
143
|
+
* the navigation belongs here rather than in genctl. */
|
|
144
|
+
function scriptInput(site: Site, defs: Record<string, Schema>): Schema | undefined {
|
|
145
|
+
const action = deref(site.input, defs);
|
|
146
|
+
const bound = action?.properties?.input;
|
|
147
|
+
return bound ?? site.input;
|
|
148
|
+
}
|
|
149
|
+
|
|
150
|
+
/** Emits one named type per reachable $def rather than inlining: a task output may
|
|
151
|
+
* reference itself (specs/recursive-type-inference.md) and inlining would not terminate. */
|
|
152
|
+
function declarations(site: Site, defs: Record<string, Schema>): string {
|
|
153
|
+
const used = new Set<string>();
|
|
154
|
+
const input = tsType(scriptInput(site, defs), used);
|
|
155
|
+
const output = tsType(site.output, used);
|
|
156
|
+
|
|
157
|
+
const emitted: string[] = [];
|
|
158
|
+
const done = new Set<string>();
|
|
159
|
+
while (true) {
|
|
160
|
+
const next = [...used].find((n) => !done.has(n));
|
|
161
|
+
if (next === undefined) break;
|
|
162
|
+
done.add(next);
|
|
163
|
+
const body = tsType(defs[next], used);
|
|
164
|
+
emitted.push(`export type ${identifier(next)} = ${body};`);
|
|
165
|
+
}
|
|
166
|
+
|
|
167
|
+
return [
|
|
168
|
+
"// Generated by genroc. Do not edit - regenerate with `genctl types`.",
|
|
169
|
+
`// ${site.process}${site.task ? ` / ${site.task}` : ""} (${site.pointer})`,
|
|
170
|
+
"",
|
|
171
|
+
...emitted,
|
|
172
|
+
emitted.length ? "" : "",
|
|
173
|
+
`export type Input = ${input};`,
|
|
174
|
+
"",
|
|
175
|
+
`export type Output = ${output};`,
|
|
176
|
+
"",
|
|
177
|
+
].join("\n");
|
|
178
|
+
}
|
|
179
|
+
|
|
180
|
+
/** Keyed by the script's PATH, not the task id: keyed by task, renaming a task would break
|
|
181
|
+
* the author's `import type` line with the error landing nowhere near the rename. */
|
|
182
|
+
function typesPathFor(scriptPath: string): string {
|
|
183
|
+
return scriptPath.replace(/\.[^.\/]+$/, "") + ".genroc.d.ts";
|
|
184
|
+
}
|
|
185
|
+
|
|
186
|
+
// ── typecheck ──────────────────────────────────────────────────────────────────
|
|
187
|
+
|
|
188
|
+
/** The nearest tsconfig above the script — the one the author's editor already reads. Two
|
|
189
|
+
* different configs mean a red editor over a clean apply, or the reverse. The walk stops at
|
|
190
|
+
* the project root: above it is not this project. */
|
|
191
|
+
async function nearestTsconfig(from: string, root: string): Promise<string | null> {
|
|
192
|
+
for (let dir = from; ; dir = dirname(dir)) {
|
|
193
|
+
const candidate = join(dir, "tsconfig.json");
|
|
194
|
+
if (await exists(candidate)) return candidate;
|
|
195
|
+
if (dir === root || dirname(dir) === dir) return null;
|
|
196
|
+
}
|
|
197
|
+
}
|
|
198
|
+
|
|
199
|
+
async function typecheck(root: string, sites: Site[]): Promise<void> {
|
|
200
|
+
const dir = join(root, ".genroc");
|
|
201
|
+
await write(join(dir, ".gitignore"), "*\n");
|
|
202
|
+
|
|
203
|
+
// One tsc per distinct base config: `extends` takes a single base, so merging two would
|
|
204
|
+
// check each script under the other author's options.
|
|
205
|
+
const groups = new Map<string, Site[]>();
|
|
206
|
+
for (const site of sites) {
|
|
207
|
+
const base = (await nearestTsconfig(dirname(site.path), root)) ?? "";
|
|
208
|
+
const group = groups.get(base);
|
|
209
|
+
if (group) group.push(site);
|
|
210
|
+
else groups.set(base, [site]);
|
|
211
|
+
}
|
|
212
|
+
|
|
213
|
+
let n = 0;
|
|
214
|
+
for (const [base, group] of groups) {
|
|
215
|
+
const config: Record<string, unknown> = {
|
|
216
|
+
...(base ? { extends: relative(dir, base) } : {}),
|
|
217
|
+
compilerOptions: {
|
|
218
|
+
noEmit: true,
|
|
219
|
+
strict: true,
|
|
220
|
+
skipLibCheck: true,
|
|
221
|
+
moduleDetection: "force",
|
|
222
|
+
module: "preserve",
|
|
223
|
+
target: "esnext",
|
|
224
|
+
// `lib` DESCRIBES the realm and is written after `extends` so a base cannot widen it:
|
|
225
|
+
// a worker thread has no document, whatever an author's config claims.
|
|
226
|
+
lib: ["esnext", "webworker"],
|
|
227
|
+
// `types` is the author's, and it is how a script opts into the node globals —
|
|
228
|
+
// the worker realm has them, so refusing the declarations would only lie. With no
|
|
229
|
+
// base config there is nothing to opt in with, so the default stays none.
|
|
230
|
+
...(base ? {} : { types: [] }),
|
|
231
|
+
},
|
|
232
|
+
files: group.flatMap((s) => [relative(dir, s.path), relative(dir, typesPathFor(s.path))]),
|
|
233
|
+
// `files` overrides the base's, but a base `include` survives beside it and would
|
|
234
|
+
// drag the author's whole tree in, to be checked under the worker lib.
|
|
235
|
+
include: [],
|
|
236
|
+
};
|
|
237
|
+
const configPath = join(dir, groups.size === 1 ? "tsconfig.json" : `tsconfig.${n++}.json`);
|
|
238
|
+
await write(configPath, JSON.stringify(config, null, 2));
|
|
239
|
+
await runTsc(root, configPath);
|
|
240
|
+
}
|
|
241
|
+
}
|
|
242
|
+
|
|
243
|
+
async function runTsc(root: string, configPath: string): Promise<void> {
|
|
244
|
+
const tsc = fileURLToPath(import.meta.resolve("typescript/bin/tsc"));
|
|
245
|
+
const proc = spawn(process.execPath, [tsc, "--noEmit", "-p", configPath], {
|
|
246
|
+
cwd: root,
|
|
247
|
+
stdio: ["ignore", "pipe", "pipe"],
|
|
248
|
+
});
|
|
249
|
+
let out = "";
|
|
250
|
+
let err = "";
|
|
251
|
+
proc.stdout.on("data", (c: Buffer) => (out += c));
|
|
252
|
+
proc.stderr.on("data", (c: Buffer) => (err += c));
|
|
253
|
+
const code = await new Promise<number>((resolve, reject) => {
|
|
254
|
+
proc.on("error", reject);
|
|
255
|
+
proc.on("close", (c) => resolve(c ?? 1));
|
|
256
|
+
});
|
|
257
|
+
if (code !== 0) {
|
|
258
|
+
// tsc reports on stdout; the exit code IS the type check, so this is the diagnostic
|
|
259
|
+
// genctl surfaces and the reason a failed import never produces a string.
|
|
260
|
+
die([out, err].filter(Boolean).join("\n").trimEnd());
|
|
261
|
+
}
|
|
262
|
+
}
|
|
263
|
+
|
|
264
|
+
// ── bundle ─────────────────────────────────────────────────────────────────────
|
|
265
|
+
|
|
266
|
+
/** Transpiles only. The typecheck above already ran over the author's OWN tsconfig, and a
|
|
267
|
+
* second opinion from a config they do not control could fail a build they cannot fix. */
|
|
268
|
+
const transpile: Plugin = {
|
|
269
|
+
name: "genroc-transpile",
|
|
270
|
+
transform(code, id) {
|
|
271
|
+
if (!id.endsWith(".ts") && !id.endsWith(".tsx")) return null;
|
|
272
|
+
const out = ts.transpileModule(code, {
|
|
273
|
+
fileName: id,
|
|
274
|
+
compilerOptions: {
|
|
275
|
+
target: ts.ScriptTarget.ESNext,
|
|
276
|
+
module: ts.ModuleKind.ESNext,
|
|
277
|
+
verbatimModuleSyntax: false,
|
|
278
|
+
jsx: id.endsWith(".tsx") ? ts.JsxEmit.ReactJSX : undefined,
|
|
279
|
+
},
|
|
280
|
+
});
|
|
281
|
+
return { code: out.outputText, map: out.sourceMapText ?? null };
|
|
282
|
+
},
|
|
283
|
+
};
|
|
284
|
+
|
|
285
|
+
const BUILTIN = new Set([...builtinModules, ...builtinModules.map((m) => `node:${m}`)]);
|
|
286
|
+
|
|
287
|
+
/** Resolves imports through TYPESCRIPT, using the same config the typecheck ran under, so a
|
|
288
|
+
* `paths` alias that compiles also bundles. Reimplementing `paths` here would be a second
|
|
289
|
+
* resolver to keep in agreement with tsc; this one cannot disagree.
|
|
290
|
+
* A package resolving to a `.d.ts` is declined — that is a type, not the implementation —
|
|
291
|
+
* which leaves node_modules to nodeResolve. */
|
|
292
|
+
function tsResolve(configPath: string | null): Plugin {
|
|
293
|
+
let options: ts.CompilerOptions = {};
|
|
294
|
+
if (configPath) {
|
|
295
|
+
const read = ts.readConfigFile(configPath, ts.sys.readFile);
|
|
296
|
+
options = ts.parseJsonConfigFileContent(read.config ?? {}, ts.sys, dirname(configPath)).options;
|
|
297
|
+
}
|
|
298
|
+
return {
|
|
299
|
+
name: "genroc-ts-resolve",
|
|
300
|
+
resolveId(source, importer) {
|
|
301
|
+
if (!importer || BUILTIN.has(source)) return null;
|
|
302
|
+
const { resolvedModule } = ts.resolveModuleName(source, importer, options, ts.sys);
|
|
303
|
+
if (!resolvedModule || resolvedModule.isExternalLibraryImport) return null;
|
|
304
|
+
return resolvedModule.resolvedFileName.endsWith(".d.ts") ? null : resolvedModule.resolvedFileName;
|
|
305
|
+
},
|
|
306
|
+
};
|
|
307
|
+
}
|
|
308
|
+
|
|
309
|
+
/** Bundles to CJS and wraps it as an async function BODY, which is what /eval compiles.
|
|
310
|
+
* The runtime stays unchanged: bundling is entirely the importer's job, and the string it
|
|
311
|
+
* produces is self-contained, so a definition version pins its code forever. */
|
|
312
|
+
async function bundle(site: Site, root: string): Promise<string> {
|
|
313
|
+
// Builtins are EXTERNALISED as `require` calls that worker.ts satisfies. Anything else
|
|
314
|
+
// unresolved is a REFUSAL, not an external: rollup's default is to leave it as a require
|
|
315
|
+
// of a module that will not be there, which bundles clean and fails at runtime.
|
|
316
|
+
const built = await rollup({
|
|
317
|
+
input: site.path,
|
|
318
|
+
external: (id) => BUILTIN.has(id),
|
|
319
|
+
plugins: [
|
|
320
|
+
tsResolve(await nearestTsconfig(dirname(site.path), root)),
|
|
321
|
+
nodeResolve({ extensions: [".ts", ".tsx", ".mjs", ".js", ".json"] }),
|
|
322
|
+
commonjs(),
|
|
323
|
+
// A `.json` import is a data file inlined at build time, which the previous bundler
|
|
324
|
+
// did natively; without it rollup hands the JSON to the JS parser.
|
|
325
|
+
json(),
|
|
326
|
+
transpile,
|
|
327
|
+
],
|
|
328
|
+
onwarn(warning) {
|
|
329
|
+
if (warning.code === "UNRESOLVED_IMPORT") {
|
|
330
|
+
die(`${site.path}: cannot resolve ${warning.exporter ?? "an import"} — is it installed?`);
|
|
331
|
+
}
|
|
332
|
+
},
|
|
333
|
+
}).catch((e: unknown) => die(`${site.path}: ${e instanceof Error ? e.message : String(e)}`));
|
|
334
|
+
|
|
335
|
+
const { output } = await built.generate({ format: "cjs", exports: "auto", inlineDynamicImports: true });
|
|
336
|
+
await built.close();
|
|
337
|
+
const cjs = output[0].code;
|
|
338
|
+
return [
|
|
339
|
+
"var module = { exports: {} }, exports = module.exports;",
|
|
340
|
+
cjs,
|
|
341
|
+
"var __genroc_main = module.exports.default ?? module.exports;",
|
|
342
|
+
'if (typeof __genroc_main !== "function") {',
|
|
343
|
+
` throw new Error(${JSON.stringify(`${site.path} has no default export function`)});`,
|
|
344
|
+
"}",
|
|
345
|
+
"return await __genroc_main(input);",
|
|
346
|
+
].join("\n");
|
|
347
|
+
}
|
|
348
|
+
|
|
349
|
+
// ── main ───────────────────────────────────────────────────────────────────────
|
|
350
|
+
|
|
351
|
+
const stdin: string = await new Promise((resolve, reject) => {
|
|
352
|
+
let raw = "";
|
|
353
|
+
process.stdin.setEncoding("utf8");
|
|
354
|
+
process.stdin.on("data", (c) => (raw += c));
|
|
355
|
+
process.stdin.on("end", () => resolve(raw));
|
|
356
|
+
process.stdin.on("error", reject);
|
|
357
|
+
});
|
|
358
|
+
const manifest = JSON.parse(stdin) as Manifest;
|
|
359
|
+
if (!manifest || !Array.isArray(manifest.sites)) die("stdin is not a genroc resolver manifest");
|
|
360
|
+
|
|
361
|
+
// One script at two sites with different input types is a refusal, not a union: the union
|
|
362
|
+
// is sound and would typecheck a body that is wrong at one of the sites.
|
|
363
|
+
const byPath = new Map<string, Site>();
|
|
364
|
+
for (const site of manifest.sites) {
|
|
365
|
+
const seen = byPath.get(site.path);
|
|
366
|
+
const defsOf = (x: Site) => (manifest.schemas[x.process]?.$defs ?? {}) as Record<string, Schema>;
|
|
367
|
+
if (
|
|
368
|
+
seen &&
|
|
369
|
+
JSON.stringify(scriptInput(seen, defsOf(seen))) !==
|
|
370
|
+
JSON.stringify(scriptInput(site, defsOf(site)))
|
|
371
|
+
) {
|
|
372
|
+
die(
|
|
373
|
+
`${site.path} is imported at ${seen.pointer} and ${site.pointer} with different input types.\n` +
|
|
374
|
+
"Split it into two scripts, or make the two call sites pass the same shape.",
|
|
375
|
+
);
|
|
376
|
+
}
|
|
377
|
+
byPath.set(site.path, site);
|
|
378
|
+
}
|
|
379
|
+
|
|
380
|
+
for (const site of byPath.values()) {
|
|
381
|
+
const defs = (manifest.schemas[site.process]?.$defs ?? {}) as Record<string, Schema>;
|
|
382
|
+
await write(typesPathFor(site.path), declarations(site, defs));
|
|
383
|
+
}
|
|
384
|
+
|
|
385
|
+
if (manifest.mode === "types") {
|
|
386
|
+
process.exit(0);
|
|
387
|
+
}
|
|
388
|
+
|
|
389
|
+
await typecheck(manifest.root, [...byPath.values()]);
|
|
390
|
+
|
|
391
|
+
const code: string[] = [];
|
|
392
|
+
for (const site of manifest.sites) {
|
|
393
|
+
code.push(await bundle(site, manifest.root));
|
|
394
|
+
}
|
|
395
|
+
process.stdout.write(JSON.stringify({ code }));
|
package/package.json
ADDED
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
{
|
|
2
|
+
"name": "@genroc/eval-node",
|
|
3
|
+
"version": "0.0.1",
|
|
4
|
+
"type": "module",
|
|
5
|
+
"engines": {
|
|
6
|
+
"node": ">=24"
|
|
7
|
+
},
|
|
8
|
+
"scripts": {
|
|
9
|
+
"work": "node worker.ts",
|
|
10
|
+
"typecheck": "tsc --noEmit"
|
|
11
|
+
},
|
|
12
|
+
"dependencies": {
|
|
13
|
+
"@rollup/plugin-commonjs": "^28.0.6",
|
|
14
|
+
"@rollup/plugin-json": "^6.1.0",
|
|
15
|
+
"@rollup/plugin-node-resolve": "^16.0.1",
|
|
16
|
+
"rollup": "^4.52.4",
|
|
17
|
+
"typescript": "^6.0.3"
|
|
18
|
+
},
|
|
19
|
+
"devDependencies": {
|
|
20
|
+
"@types/node": "^26.1.2"
|
|
21
|
+
},
|
|
22
|
+
"description": "genroc script tasks for Node: the author-time bundler and the queue worker",
|
|
23
|
+
"license": "Apache-2.0",
|
|
24
|
+
"repository": {
|
|
25
|
+
"type": "git",
|
|
26
|
+
"url": "git+https://github.com/genroc/genroc.git",
|
|
27
|
+
"directory": "eval-node"
|
|
28
|
+
},
|
|
29
|
+
"homepage": "https://genroc.org",
|
|
30
|
+
"bin": {
|
|
31
|
+
"genroc-import": "bin/import.mjs",
|
|
32
|
+
"genroc-eval-node": "bin/worker.mjs"
|
|
33
|
+
},
|
|
34
|
+
"files": [
|
|
35
|
+
"bin",
|
|
36
|
+
"*.ts",
|
|
37
|
+
"tsconfig.json",
|
|
38
|
+
"README.md"
|
|
39
|
+
]
|
|
40
|
+
}
|
package/realm.ts
ADDED
|
@@ -0,0 +1,132 @@
|
|
|
1
|
+
// The evaluation realm. One Worker per execution: a fresh global object per script, and a
|
|
2
|
+
// thread the host can kill mid-loop — the only thing that bounds a synchronous busy loop.
|
|
3
|
+
// eval.ts owns the budget and does the killing; nothing here knows about time.
|
|
4
|
+
//
|
|
5
|
+
// Everything that touches the script's VALUE lives on this side of the boundary — compiling,
|
|
6
|
+
// classifying, serialising — because this is the only realm the value exists in.
|
|
7
|
+
|
|
8
|
+
import { createRequire } from "node:module";
|
|
9
|
+
import { parentPort } from "node:worker_threads";
|
|
10
|
+
|
|
11
|
+
import type { EvalFailure, FailureKind, WorkerReply, WorkerRequest } from "./eval.ts";
|
|
12
|
+
|
|
13
|
+
const AsyncFunction = async function () {}.constructor as new (
|
|
14
|
+
...args: string[]
|
|
15
|
+
) => (...args: unknown[]) => Promise<unknown>;
|
|
16
|
+
|
|
17
|
+
const STRICT = '"use strict";\n';
|
|
18
|
+
|
|
19
|
+
// Bundled `node:*` imports survive as `require` calls — the importer externalises builtins and
|
|
20
|
+
// inlines everything else — and a function built by the AsyncFunction constructor has no
|
|
21
|
+
// `require` in scope. Passing one in is what makes an import of a builtin work at runtime rather than at
|
|
22
|
+
// typecheck only. Resolution is anchored here, which is right: only builtins reach it.
|
|
23
|
+
const scriptRequire = createRequire(import.meta.url);
|
|
24
|
+
const STACK_BYTES = 2_048;
|
|
25
|
+
|
|
26
|
+
/**
|
|
27
|
+
* Line offset the AsyncFunction preamble adds, measured rather than assumed: the generated
|
|
28
|
+
* wrapper's shape is engine-specific, and a hardcoded number silently misreports every
|
|
29
|
+
* script's error location the day it changes.
|
|
30
|
+
*/
|
|
31
|
+
const lineOffset: Promise<number> = (async () => {
|
|
32
|
+
// Same parameter list as a real compile: the preamble is what is being measured.
|
|
33
|
+
const probe = new AsyncFunction("input", "require", STRICT + "throw new Error('probe');");
|
|
34
|
+
try {
|
|
35
|
+
await probe();
|
|
36
|
+
return 0;
|
|
37
|
+
} catch (err) {
|
|
38
|
+
return reportedLine(err) - 1;
|
|
39
|
+
}
|
|
40
|
+
})();
|
|
41
|
+
|
|
42
|
+
// V8 marks a frame compiled by the AsyncFunction constructor with the site that CALLED the
|
|
43
|
+
// constructor, then the script's OWN position:
|
|
44
|
+
// at inner (eval at run (file:///…/worker.ts:107:10), <anonymous>:6:9)
|
|
45
|
+
// `eval at` is therefore what separates script frames from runner plumbing — matched without
|
|
46
|
+
// the function name, which is whatever encloses the `new AsyncFunction` below. The LAST such
|
|
47
|
+
// frame is the body's top level: frames interleave, since a script can throw inside a native
|
|
48
|
+
// callback.
|
|
49
|
+
const SCRIPT_FRAME = /\(eval at /;
|
|
50
|
+
// The trailing `<anonymous>:LINE:COL` — the script's position, after the host file's own.
|
|
51
|
+
const POSITION = /<anonymous>:(\d+):(\d+)\)?\s*$/;
|
|
52
|
+
// ` at name (` — absent on the top-level frame, which V8 names `eval`.
|
|
53
|
+
const FRAME_NAME = /^\s*at\s+(?:async\s+)?([^\s(]+)\s*\(/;
|
|
54
|
+
|
|
55
|
+
/** Line number of the throw as the engine reported it, or 1 if the stack is unreadable. */
|
|
56
|
+
function reportedLine(err: unknown): number {
|
|
57
|
+
const stack = err instanceof Error && typeof err.stack === "string" ? err.stack : "";
|
|
58
|
+
const frame = stack.split("\n").find((l) => SCRIPT_FRAME.test(l)) ?? "";
|
|
59
|
+
const m = frame.match(POSITION);
|
|
60
|
+
return m ? Number(m[1]) : 1;
|
|
61
|
+
}
|
|
62
|
+
|
|
63
|
+
/** Renumbers each script frame to the line the AUTHOR wrote and drops the runner's own.
|
|
64
|
+
* Rewriting the whole location is also what keeps the runner's path out of a script's
|
|
65
|
+
* stack — V8 puts it inside every compiled frame. */
|
|
66
|
+
function scriptStack(err: unknown, offset: number): string | undefined {
|
|
67
|
+
if (!(err instanceof Error) || typeof err.stack !== "string") return undefined;
|
|
68
|
+
const lines = err.stack.split("\n");
|
|
69
|
+
let boundary = -1;
|
|
70
|
+
for (let i = 0; i < lines.length; i++) if (SCRIPT_FRAME.test(lines[i]!)) boundary = i;
|
|
71
|
+
const frames = (boundary >= 0 ? lines.slice(0, boundary + 1) : lines.slice(0, 1))
|
|
72
|
+
.map((line) => {
|
|
73
|
+
const pos = line.match(POSITION);
|
|
74
|
+
if (!pos) return line; // the `Error: message` header and native frames, kept as-is
|
|
75
|
+
const name = line.match(FRAME_NAME)?.[1];
|
|
76
|
+
const at = name && name !== "eval" && name !== "anonymous" ? `at ${name} ` : "at ";
|
|
77
|
+
const indent = line.match(/^\s*/)![0];
|
|
78
|
+
return `${indent}${at}(script:${Math.max(1, Number(pos[1]) - offset)}:${pos[2]})`;
|
|
79
|
+
})
|
|
80
|
+
.join("\n");
|
|
81
|
+
return frames.length > STACK_BYTES ? frames.slice(0, STACK_BYTES) : frames;
|
|
82
|
+
}
|
|
83
|
+
|
|
84
|
+
function describe(err: unknown, kind: FailureKind, offset: number): EvalFailure {
|
|
85
|
+
if (err instanceof Error) {
|
|
86
|
+
return { kind, name: err.name, message: err.message, stack: scriptStack(err, offset) };
|
|
87
|
+
}
|
|
88
|
+
// A script may throw a non-Error (`throw {code: "x"}`), so name/message must not assume one.
|
|
89
|
+
return { kind, name: "Thrown", message: safeText(err) };
|
|
90
|
+
}
|
|
91
|
+
|
|
92
|
+
function safeText(v: unknown): string {
|
|
93
|
+
try {
|
|
94
|
+
return typeof v === "string" ? v : JSON.stringify(v) ?? String(v);
|
|
95
|
+
} catch {
|
|
96
|
+
return String(v);
|
|
97
|
+
}
|
|
98
|
+
}
|
|
99
|
+
|
|
100
|
+
async function run(req: WorkerRequest): Promise<WorkerReply> {
|
|
101
|
+
const offset = await lineOffset;
|
|
102
|
+
|
|
103
|
+
let fn: (...args: unknown[]) => Promise<unknown>;
|
|
104
|
+
try {
|
|
105
|
+
// No compile cache: the realm is discarded after this execution, so a cache in it could
|
|
106
|
+
// never be hit. Repeated compilation is the price of the fresh global object.
|
|
107
|
+
fn = new AsyncFunction("input", "require", STRICT + req.code);
|
|
108
|
+
} catch (err) {
|
|
109
|
+
return { ok: false, failure: describe(err, "compile_error", offset) };
|
|
110
|
+
}
|
|
111
|
+
|
|
112
|
+
let value: unknown;
|
|
113
|
+
try {
|
|
114
|
+
value = await fn(req.input, scriptRequire);
|
|
115
|
+
} catch (err) {
|
|
116
|
+
return { ok: false, failure: describe(err, "threw", offset) };
|
|
117
|
+
}
|
|
118
|
+
|
|
119
|
+
try {
|
|
120
|
+
// undefined stringifies to undefined, not "undefined"; an empty body is how genroc
|
|
121
|
+
// spells null, which is the right reading of a script that returned nothing.
|
|
122
|
+
return { ok: true, body: value === undefined ? "" : JSON.stringify(value) ?? "" };
|
|
123
|
+
} catch (err) {
|
|
124
|
+
return { ok: false, failure: describe(err, "nonserializable", offset) };
|
|
125
|
+
}
|
|
126
|
+
}
|
|
127
|
+
|
|
128
|
+
// Non-null because this module only ever runs as a worker entry point; a null port here
|
|
129
|
+
// would mean eval.ts loaded it as a plain module, which nothing does.
|
|
130
|
+
parentPort!.on("message", async (req: WorkerRequest) => {
|
|
131
|
+
parentPort!.postMessage(await run(req));
|
|
132
|
+
});
|
package/tsconfig.json
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
{
|
|
2
|
+
"compilerOptions": {
|
|
3
|
+
"lib": [
|
|
4
|
+
"ESNext"
|
|
5
|
+
],
|
|
6
|
+
"types": [
|
|
7
|
+
"node"
|
|
8
|
+
],
|
|
9
|
+
"target": "ESNext",
|
|
10
|
+
"module": "ESNext",
|
|
11
|
+
"moduleResolution": "bundler",
|
|
12
|
+
"moduleDetection": "force",
|
|
13
|
+
"allowImportingTsExtensions": true,
|
|
14
|
+
"verbatimModuleSyntax": true,
|
|
15
|
+
"strict": true,
|
|
16
|
+
"noUncheckedIndexedAccess": true,
|
|
17
|
+
"noEmit": true,
|
|
18
|
+
"skipLibCheck": true
|
|
19
|
+
},
|
|
20
|
+
"include": [
|
|
21
|
+
"**/*.ts"
|
|
22
|
+
]
|
|
23
|
+
}
|
package/worker.ts
ADDED
|
@@ -0,0 +1,300 @@
|
|
|
1
|
+
// The queue worker: claims parked `external` script tasks from genroc, evaluates each in its
|
|
2
|
+
// own realm, and answers. This is the whole genroc-facing half — eval.ts and realm.ts know
|
|
3
|
+
// nothing about the queue, which is what keeps the containment strategy swappable.
|
|
4
|
+
//
|
|
5
|
+
// See README.md for the contract, and specs/external-task-queue.md for the queue itself.
|
|
6
|
+
|
|
7
|
+
import { evaluate, type EvalRequest, type FailureKind } from "./eval.ts";
|
|
8
|
+
|
|
9
|
+
const SERVER = (process.env.GENROC_SERVER ?? "http://localhost:8448").replace(/\/$/, "");
|
|
10
|
+
const WORKER_ID = process.env.WORKER_ID ?? `evaluator-${process.pid}`;
|
|
11
|
+
// The credential, when the server runs with --auth token. A worker needs exactly the `worker`
|
|
12
|
+
// permission — the four queue verbs plus GET /api/objects — so mint it scoped rather than
|
|
13
|
+
// handing a worker an admin token: this is the credential most likely to sit on a machine you
|
|
14
|
+
// trust least. specs/api-auth.md §5.
|
|
15
|
+
//
|
|
16
|
+
// Sent as a header rather than in the URL because Node's fetch REFUSES a URL carrying
|
|
17
|
+
// credentials ("Request cannot be constructed from a URL that includes credentials"), so the
|
|
18
|
+
// basic-auth-in-the-URL trick that works for genctl is not available here.
|
|
19
|
+
const TOKEN = process.env.GENROC_TOKEN ?? "";
|
|
20
|
+
const authHeaders: Record<string, string> = TOKEN ? { authorization: `Bearer ${TOKEN}` } : {};
|
|
21
|
+
// Concurrency is the worker's to set, and that is the point of pulling: under the old fetch
|
|
22
|
+
// shape genroc decided how many scripts ran at once (--max-concurrent, default 200) and the
|
|
23
|
+
// evaluator accepted every one of them. Here it claims what it can run and no more, so a
|
|
24
|
+
// backlog is a queue rather than 200 threads fighting over a core.
|
|
25
|
+
const CONCURRENCY = Number(process.env.CONCURRENCY ?? 4);
|
|
26
|
+
const POLL_MS = Number(process.env.POLL_MS ?? 250);
|
|
27
|
+
// The visibility timeout. Short, and renewed while work is in flight: a worker that dies
|
|
28
|
+
// should return its task quickly rather than holding it for the whole budget.
|
|
29
|
+
const LEASE_MS = Number(process.env.LEASE_MS ?? 30_000);
|
|
30
|
+
const RENEW_MS = Math.max(1_000, Math.floor(LEASE_MS / 3));
|
|
31
|
+
const PROCESS_FILTER = process.env.PROCESS ?? "";
|
|
32
|
+
const TASK_FILTER = process.env.TASK ?? "";
|
|
33
|
+
|
|
34
|
+
type ObjectEntry = { path: (string | number)[]; ref: string; size: number };
|
|
35
|
+
|
|
36
|
+
type QueueTask = {
|
|
37
|
+
token: string;
|
|
38
|
+
process: string;
|
|
39
|
+
task_id: string;
|
|
40
|
+
input: unknown;
|
|
41
|
+
objects?: ObjectEntry[];
|
|
42
|
+
raises?: Record<string, unknown>;
|
|
43
|
+
};
|
|
44
|
+
|
|
45
|
+
// Values too large to ship inline are listed rather than carried, and a bundle is exactly that:
|
|
46
|
+
// one object shared by every instance of a definition version, fetched once instead of copied
|
|
47
|
+
// into each task. A ref is a content hash, so it is immutable — the cache never invalidates.
|
|
48
|
+
const objectCache = new Map<string, unknown>();
|
|
49
|
+
|
|
50
|
+
async function fetchObject(ref: string): Promise<unknown> {
|
|
51
|
+
const cached = objectCache.get(ref);
|
|
52
|
+
if (cached !== undefined) return cached;
|
|
53
|
+
// Left to throw on purpose: this runs while a task is IN FLIGHT, and a task whose input
|
|
54
|
+
// cannot be fetched must fail rather than silently run against a missing value. The caller
|
|
55
|
+
// releases the claim, so the task returns to the queue.
|
|
56
|
+
const res = await fetch(`${SERVER}/api/objects/${encodeURIComponent(ref)}`, { headers: authHeaders });
|
|
57
|
+
if (!res.ok) throw new Error(`fetch object ${ref}: HTTP ${res.status}`);
|
|
58
|
+
const { data } = (await res.json()) as { data: string };
|
|
59
|
+
let value: unknown;
|
|
60
|
+
try {
|
|
61
|
+
value = JSON.parse(data);
|
|
62
|
+
} catch {
|
|
63
|
+
value = data;
|
|
64
|
+
}
|
|
65
|
+
objectCache.set(ref, value);
|
|
66
|
+
return value;
|
|
67
|
+
}
|
|
68
|
+
|
|
69
|
+
/** Put each listed value back where its path says it belongs. Paths are arrays of keys, so this
|
|
70
|
+
* needs no parser: the whole reason they are not JSON Pointer strings. */
|
|
71
|
+
async function resolveObjects(job: QueueTask): Promise<unknown> {
|
|
72
|
+
let input: any = job.input;
|
|
73
|
+
for (const e of job.objects ?? []) {
|
|
74
|
+
const value = await fetchObject(e.ref);
|
|
75
|
+
if (e.path.length === 0) {
|
|
76
|
+
input = value;
|
|
77
|
+
continue;
|
|
78
|
+
}
|
|
79
|
+
// The path is rooted at the entry and starts with "input", which is the value being rebuilt.
|
|
80
|
+
const rest = e.path[0] === "input" ? e.path.slice(1) : e.path;
|
|
81
|
+
if (rest.length === 0) {
|
|
82
|
+
input = value;
|
|
83
|
+
continue;
|
|
84
|
+
}
|
|
85
|
+
let cur: any = input;
|
|
86
|
+
for (let i = 0; i < rest.length - 1; i++) cur = cur?.[rest[i]!];
|
|
87
|
+
if (cur) cur[rest[rest.length - 1]!] = value;
|
|
88
|
+
}
|
|
89
|
+
return input;
|
|
90
|
+
}
|
|
91
|
+
|
|
92
|
+
async function call(path: string, body: unknown): Promise<{ ok: boolean; status: number; data: any }> {
|
|
93
|
+
// A network error is a REPLY, not a throw. A worker outlives the server it polls — a
|
|
94
|
+
// restart, a rolling deploy, a container coming up before genroc is listening — and an
|
|
95
|
+
// unhandled rejection here kills it for a condition the next poll would clear. Status 0
|
|
96
|
+
// says "never reached the server", which is distinct from anything genroc answers.
|
|
97
|
+
let res: Response;
|
|
98
|
+
try {
|
|
99
|
+
res = await fetch(SERVER + path, {
|
|
100
|
+
method: "POST",
|
|
101
|
+
headers: { "content-type": "application/json", ...authHeaders },
|
|
102
|
+
body: JSON.stringify(body),
|
|
103
|
+
});
|
|
104
|
+
} catch (err) {
|
|
105
|
+
return { ok: false, status: 0, data: { error: `${SERVER} unreachable: ${(err as Error).message}` } };
|
|
106
|
+
}
|
|
107
|
+
const text = await res.text();
|
|
108
|
+
let data: any = null;
|
|
109
|
+
try {
|
|
110
|
+
data = text ? JSON.parse(text) : null;
|
|
111
|
+
} catch {
|
|
112
|
+
data = { error: text };
|
|
113
|
+
}
|
|
114
|
+
return { ok: res.ok, status: res.status, data };
|
|
115
|
+
}
|
|
116
|
+
|
|
117
|
+
/** The task input IS an EvalRequest: `code` required, `input` and `timeout_ms` optional. A task
|
|
118
|
+
* whose input is not that shape is the definition's fault, not the script's, and is reported
|
|
119
|
+
* as a compile_error — the nearest permanent kind, since no retry can fix the definition. */
|
|
120
|
+
function asEvalRequest(input: unknown): EvalRequest | string {
|
|
121
|
+
if (typeof input !== "object" || input === null) return "the task input is not an object";
|
|
122
|
+
const r = input as Record<string, unknown>;
|
|
123
|
+
if (typeof r.code !== "string") return "the task input has no `code` string";
|
|
124
|
+
return {
|
|
125
|
+
code: r.code,
|
|
126
|
+
input: r.input,
|
|
127
|
+
timeout_ms: typeof r.timeout_ms === "number" ? r.timeout_ms : undefined,
|
|
128
|
+
};
|
|
129
|
+
}
|
|
130
|
+
|
|
131
|
+
const inFlight = new Map<string, QueueTask>();
|
|
132
|
+
let running = true;
|
|
133
|
+
|
|
134
|
+
// Whether the last claim reached genroc. A worker polls several times a second, so an
|
|
135
|
+
// unreachable server would otherwise emit a line per poll — thousands during a restart, which
|
|
136
|
+
// buries the one line that mattered. Announce the TRANSITIONS instead: going away, and coming
|
|
137
|
+
// back. Silence in between is the report that nothing changed.
|
|
138
|
+
let serverReachable = true;
|
|
139
|
+
|
|
140
|
+
async function claim(n: number): Promise<QueueTask[]> {
|
|
141
|
+
const { ok, status, data } = await call("/api/external-tasks/claim", {
|
|
142
|
+
worker_id: WORKER_ID,
|
|
143
|
+
limit: n,
|
|
144
|
+
lease_ms: LEASE_MS,
|
|
145
|
+
...(PROCESS_FILTER ? { process: PROCESS_FILTER } : {}),
|
|
146
|
+
...(TASK_FILTER ? { task: TASK_FILTER } : {}),
|
|
147
|
+
});
|
|
148
|
+
if (!ok) {
|
|
149
|
+
// A credential problem is not transient, and polling through it looks like a healthy
|
|
150
|
+
// worker that never picks anything up — the worst shape for an operator to debug. Exit
|
|
151
|
+
// instead, so a supervisor restarts it and the failure is visible where it happened.
|
|
152
|
+
if (status === 401 || status === 403) {
|
|
153
|
+
console.error(
|
|
154
|
+
`claim rejected (${status}): ${JSON.stringify(data)}\n` +
|
|
155
|
+
`The server requires authentication. Set GENROC_TOKEN to a token with the 'worker' ` +
|
|
156
|
+
`permission — mint one with: genctl token create --perms worker --label evaluator -q`,
|
|
157
|
+
);
|
|
158
|
+
process.exit(1);
|
|
159
|
+
}
|
|
160
|
+
// status 0 is "never reached the server" (see call): a restart, a rolling deploy, a
|
|
161
|
+
// network blip. Not an error to act on — the next poll clears it — so it is reported once
|
|
162
|
+
// and then waited out.
|
|
163
|
+
if (status === 0) {
|
|
164
|
+
if (serverReachable) {
|
|
165
|
+
serverReachable = false;
|
|
166
|
+
console.error(
|
|
167
|
+
`genroc at ${SERVER} is unreachable — ${(data as { error?: string })?.error ?? ""}. ` +
|
|
168
|
+
`Still polling every ${POLL_MS}ms; work resumes when it comes back.`,
|
|
169
|
+
);
|
|
170
|
+
}
|
|
171
|
+
return [];
|
|
172
|
+
}
|
|
173
|
+
console.error(`claim failed: ${JSON.stringify(data)}`);
|
|
174
|
+
return [];
|
|
175
|
+
}
|
|
176
|
+
if (!serverReachable) {
|
|
177
|
+
serverReachable = true;
|
|
178
|
+
console.error(`genroc at ${SERVER} is reachable again — resuming.`);
|
|
179
|
+
}
|
|
180
|
+
return (data?.items ?? []) as QueueTask[];
|
|
181
|
+
}
|
|
182
|
+
|
|
183
|
+
async function release(token: string): Promise<void> {
|
|
184
|
+
const { ok, data } = await call("/api/external-tasks/release", { token });
|
|
185
|
+
if (!ok) console.error(`release failed: ${JSON.stringify(data)}`);
|
|
186
|
+
}
|
|
187
|
+
|
|
188
|
+
/** answer submits the outcome. A refusal is NOT retried with a different one: the definition
|
|
189
|
+
* declared a contract this worker does not satisfy (an undeclared code, a payload that does
|
|
190
|
+
* not fit `raises`), and guessing again would only pick a second wrong answer. Release it, so
|
|
191
|
+
* the task returns to the queue and an operator sees it waiting rather than silently gone. */
|
|
192
|
+
async function answer(token: string, outcome: Record<string, unknown>): Promise<void> {
|
|
193
|
+
const { ok, data } = await call("/api/external-tasks/resolve", { token, ...outcome });
|
|
194
|
+
if (ok) return;
|
|
195
|
+
console.error(`genroc refused the outcome for ${token}: ${JSON.stringify(data)}`);
|
|
196
|
+
await release(token);
|
|
197
|
+
}
|
|
198
|
+
|
|
199
|
+
async function run(job: QueueTask): Promise<void> {
|
|
200
|
+
let resolved: unknown;
|
|
201
|
+
try {
|
|
202
|
+
resolved = await resolveObjects(job);
|
|
203
|
+
} catch (err) {
|
|
204
|
+
// The values are there or they are not; this is the runner failing to read them, not the
|
|
205
|
+
// script failing, so hand the task back for someone else rather than reporting an outcome.
|
|
206
|
+
console.error(`resolving objects for ${job.token}: ${err instanceof Error ? err.message : String(err)}`);
|
|
207
|
+
await release(job.token);
|
|
208
|
+
return;
|
|
209
|
+
}
|
|
210
|
+
const req = asEvalRequest(resolved);
|
|
211
|
+
if (typeof req === "string") {
|
|
212
|
+
await answer(job.token, {
|
|
213
|
+
error: { code: "compile_error", message: req, data: { name: "BadTaskInput" } },
|
|
214
|
+
});
|
|
215
|
+
return;
|
|
216
|
+
}
|
|
217
|
+
|
|
218
|
+
let result;
|
|
219
|
+
try {
|
|
220
|
+
result = await evaluate(req);
|
|
221
|
+
} catch (err) {
|
|
222
|
+
// The RUNNER faulted, not the script — the one class where a retry can help. There is no
|
|
223
|
+
// error code for it on purpose: releasing the claim is how a queue spells "retryable", and
|
|
224
|
+
// it puts the task in front of a different worker instead of burning the definition's
|
|
225
|
+
// on_error budget on this one's bad day.
|
|
226
|
+
console.error(`evaluator fault on ${job.token}: ${err instanceof Error ? err.message : String(err)}`);
|
|
227
|
+
await release(job.token);
|
|
228
|
+
return;
|
|
229
|
+
}
|
|
230
|
+
|
|
231
|
+
if (result.ok) {
|
|
232
|
+
// `body` is JSON text produced inside the realm; an empty body is a script that returned
|
|
233
|
+
// nothing, which genroc reads as null.
|
|
234
|
+
await answer(job.token, { result: result.body === "" ? null : JSON.parse(result.body) });
|
|
235
|
+
return;
|
|
236
|
+
}
|
|
237
|
+
const f = result.failure;
|
|
238
|
+
await answer(job.token, {
|
|
239
|
+
error: {
|
|
240
|
+
// The failure KIND is the code, so an on_error rule branches on what went wrong without
|
|
241
|
+
// reading a payload. Every kind is permanent; see eval.ts.
|
|
242
|
+
code: f.kind satisfies FailureKind,
|
|
243
|
+
message: f.message,
|
|
244
|
+
data: { name: f.name, ...(f.stack ? { stack: f.stack } : {}) },
|
|
245
|
+
},
|
|
246
|
+
});
|
|
247
|
+
}
|
|
248
|
+
|
|
249
|
+
async function renewLoop(): Promise<void> {
|
|
250
|
+
while (running) {
|
|
251
|
+
await new Promise((r) => setTimeout(r, RENEW_MS));
|
|
252
|
+
const tokens = [...inFlight.keys()];
|
|
253
|
+
if (!tokens.length) continue;
|
|
254
|
+
const { ok, data } = await call("/api/external-tasks/renew", {
|
|
255
|
+
worker_id: WORKER_ID,
|
|
256
|
+
tokens,
|
|
257
|
+
lease_ms: LEASE_MS,
|
|
258
|
+
});
|
|
259
|
+
// A short count means a claim lapsed and was taken over. Nothing to do about it — the work
|
|
260
|
+
// continues and its answer will be refused — but say so, because it is the signal that
|
|
261
|
+
// LEASE_MS is too short for what these scripts actually take.
|
|
262
|
+
if (ok && data?.renewed < tokens.length) {
|
|
263
|
+
console.error(`renewed ${data.renewed}/${tokens.length} claims; a lease lapsed under load`);
|
|
264
|
+
}
|
|
265
|
+
}
|
|
266
|
+
}
|
|
267
|
+
|
|
268
|
+
async function pollLoop(): Promise<void> {
|
|
269
|
+
while (running) {
|
|
270
|
+
const free = CONCURRENCY - inFlight.size;
|
|
271
|
+
const jobs = free > 0 ? await claim(free) : [];
|
|
272
|
+
for (const job of jobs) {
|
|
273
|
+
inFlight.set(job.token, job);
|
|
274
|
+
void run(job).finally(() => inFlight.delete(job.token));
|
|
275
|
+
}
|
|
276
|
+
// Only idle when there was nothing to take: a full queue should be drained at the speed the
|
|
277
|
+
// realms allow, not at the poll interval.
|
|
278
|
+
if (jobs.length === 0) await new Promise((r) => setTimeout(r, POLL_MS));
|
|
279
|
+
}
|
|
280
|
+
}
|
|
281
|
+
|
|
282
|
+
async function shutdown(signal: string): Promise<void> {
|
|
283
|
+
if (!running) return;
|
|
284
|
+
running = false;
|
|
285
|
+
const tokens = [...inFlight.keys()];
|
|
286
|
+
console.log(`${signal}: releasing ${tokens.length} claim(s)`);
|
|
287
|
+
// Hand work back rather than letting it sit out its lease. The evaluations still running are
|
|
288
|
+
// abandoned, which is exactly what the release says: nobody answered.
|
|
289
|
+
await Promise.all(tokens.map(release));
|
|
290
|
+
process.exit(0);
|
|
291
|
+
}
|
|
292
|
+
|
|
293
|
+
process.on("SIGINT", () => void shutdown("SIGINT"));
|
|
294
|
+
process.on("SIGTERM", () => void shutdown("SIGTERM"));
|
|
295
|
+
|
|
296
|
+
console.log(
|
|
297
|
+
`evaluator worker ${WORKER_ID} polling ${SERVER} (concurrency=${CONCURRENCY}, lease=${LEASE_MS}ms)`,
|
|
298
|
+
);
|
|
299
|
+
void renewLoop();
|
|
300
|
+
void pollLoop();
|