@edgehero/pi-dispatch 1.9.0 → 1.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.env.example +9 -0
- package/package.json +7 -2
- package/src/backend-conformance.mjs +262 -0
- package/src/backend-local.mjs +222 -0
- package/src/backend-registry.mjs +157 -0
- package/src/backends.mjs +595 -0
- package/src/config.mjs +63 -1
- package/src/container-spec.mjs +226 -0
- package/src/docker-run.mjs +100 -106
- package/src/doctor.mjs +118 -0
- package/src/index.mjs +123 -10
- package/src/outbox.mjs +8 -0
- package/src/packages.mjs +7 -4
- package/src/processor.mjs +48 -11
- package/src/queue.mjs +14 -2
- package/src/sandbox-cli.mjs +36 -0
- package/src/schedules.mjs +1 -1
- package/src/start.mjs +118 -80
- package/src/triggers.mjs +120 -5
package/src/backends.mjs
ADDED
|
@@ -0,0 +1,595 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* THE BACKEND TABLE -- the one place that says where a job's container can run, and what each place is
|
|
3
|
+
* ABLE to guarantee (issue #227).
|
|
4
|
+
*
|
|
5
|
+
* A backend is where the box is built. Today there is one, `local`, which is the Docker daemon on the
|
|
6
|
+
* worker's own host and is what every deployment has always used. The table exists so a second one can be
|
|
7
|
+
* added without reading the worker's source, and so that adding one cannot quietly weaken a control.
|
|
8
|
+
*
|
|
9
|
+
* This module imports NOTHING, deliberately, for `forges.mjs`'s reason: it is a leaf, and `doctor`, the
|
|
10
|
+
* config loader and (later) the receiver all need to read a declaration without pulling the Docker
|
|
11
|
+
* implementation into their graph. A lookup returns `undefined` for an unknown name and the CALLER decides
|
|
12
|
+
* how loudly to fail, because the answer differs -- a config value refuses boot, a trigger field refuses
|
|
13
|
+
* the file.
|
|
14
|
+
*
|
|
15
|
+
* A DECLARATION IS A CAPABILITY, NOT A POSTURE. This is the distinction the first draft of this file got
|
|
16
|
+
* wrong, and it is worth stating first because everything else depends on it.
|
|
17
|
+
*
|
|
18
|
+
* `CONST-EGRESS-POLICY-IN-THE-ARGV` does not say egress is denied, and its revision row says exactly why
|
|
19
|
+
* it refuses to: "an operator can set `PI_EGRESS=0` ... so a 'shall' over denial would be the
|
|
20
|
+
* constraint-that-ships-unenforced `OQ-004` refused for in the first place." The rejected ID
|
|
21
|
+
* `CONST-EGRESS-DENIED-BY-DEFAULT` is named inside that entry so it is not re-proposed. A table that said
|
|
22
|
+
* `egress: enforced` as a flat statement of what a running job gets would be re-proposing it in one word --
|
|
23
|
+
* and in a vocabulary whose whole purpose is that the word is the thing that gets PRINTED.
|
|
24
|
+
*
|
|
25
|
+
* So a declaration answers "can this backend be asked for this, and does it build it itself?", never "is
|
|
26
|
+
* this deployment currently getting it?". Those are two axes and this table is one of them. A property a
|
|
27
|
+
* deployment switch gates carries `armedBy`, naming the switch, so a reader is never handed the capability
|
|
28
|
+
* word alone where it could be mistaken for the posture. Whether that switch is on is the deployment's own
|
|
29
|
+
* answer, which `egressArmed` owns; `doctor` joins the two and prints them together, and `unarmedFloor`
|
|
30
|
+
* below makes the boot refusal read the switch as well -- a floor asking for a gated property is not met by
|
|
31
|
+
* capability alone. `declarationOf` is the one definition of that join, so a consumer never has to
|
|
32
|
+
* reconstruct it and never prints a capability word bare.
|
|
33
|
+
*
|
|
34
|
+
* The capability axis is the one a floor compares against, which is what makes the axis the right choice
|
|
35
|
+
* rather than a convenient one: the refusal it has to produce is "this deployment ARMED egress and the
|
|
36
|
+
* backend it selected cannot do egress at all", and that question is answered by capability. A posture word
|
|
37
|
+
* could not express it, because a deployment that armed nothing needs no refusal.
|
|
38
|
+
*
|
|
39
|
+
* WHY DECLARING IS NOT CLAIMING.
|
|
40
|
+
*
|
|
41
|
+
* `CONST-EGRESS-POLICY-IN-THE-ARGV` states the objection this table has to answer, and it is worth quoting
|
|
42
|
+
* because it is easy to answer the weaker version by accident:
|
|
43
|
+
*
|
|
44
|
+
* "A control whose presence is unobservable to the thing that starts the containers is indistinguishable,
|
|
45
|
+
* from every angle this project can see, from no control at all -- and an operator who believes they have
|
|
46
|
+
* one is in a WORSE position than one who knows they do not, because the belief displaces the credential
|
|
47
|
+
* bound CONST-TOKEN-SCOPED-PER-JOB says is what actually bounds the damage."
|
|
48
|
+
*
|
|
49
|
+
* The tail is the operative half. The danger is not an unobservable control; it is a BELIEVED-IN one, which
|
|
50
|
+
* displaces the bound that is really holding. So a declaration is never a claim that a property holds. It
|
|
51
|
+
* is a claim about WHO IS ASSERTING IT, in three words that must stay distinguishable everywhere they are
|
|
52
|
+
* printed:
|
|
53
|
+
*
|
|
54
|
+
* enforced -- this worker CAN build it, in its own code, and a test in this repo reads it back off what
|
|
55
|
+
* was actually produced. The only word that means the property is ours.
|
|
56
|
+
* asserted -- something outside this worker provides it: an image's `USER`, a vendor's documentation.
|
|
57
|
+
* Unverifiable from here. `doctor` prints it differently for exactly that reason.
|
|
58
|
+
* absent -- not provided at all. A deployment whose configuration needs it is REFUSED rather than
|
|
59
|
+
* silently downgraded.
|
|
60
|
+
*
|
|
61
|
+
* `OQ-012` already draws this line for images -- "a required OCI label proves INTENT, not conformance" --
|
|
62
|
+
* and the same is true one level out. The value of the table is not that a vendor is verified; it is that a
|
|
63
|
+
* MISMATCH becomes a refusal instead of a silent downgrade. `backend-conformance.mjs` verifies THREE of the
|
|
64
|
+
* thirteen -- exit-code fidelity, the abort flag's independence from the code, and the read-only downgrade a
|
|
65
|
+
* copying runtime takes -- plus the shape of a bundle and the internal consistency of its declaration. The
|
|
66
|
+
* other ten need a live container on the target runtime, and the harness names each one and what it would
|
|
67
|
+
* take rather than passing them in silence. So a backend can still declare all of this and do most of it:
|
|
68
|
+
* these words are a contract with three of them checked, which is more than none and less than proof.
|
|
69
|
+
*
|
|
70
|
+
* THE PROPERTIES ARE NOT A RENDERING OF `ISOLATION_FLAGS`, and must not become one. That array is the
|
|
71
|
+
* literal, value-free, unconditional set the local argv splices in, and two tests assert every member of it
|
|
72
|
+
* reaches the sandbox argv AGAINST THE IMPORTED ARRAY. Issue #261 explicitly rejected "expressing isolation
|
|
73
|
+
* as semantic spec fields (`dropCapabilities`, `pidsLimit`), which is more portable", because it would
|
|
74
|
+
* retire those assertions. So a property here names a GUARANTEE an operator can reason about, and the flags
|
|
75
|
+
* that deliver it stay where they are.
|
|
76
|
+
*/
|
|
77
|
+
|
|
78
|
+
/**
|
|
79
|
+
* The exit codes a DOCKER-shaped runtime uses for "the runner never ran": 125 is `docker run` itself
|
|
80
|
+
* failing, 126 an entrypoint that is not executable, 127 an entrypoint that was not found. In all three the
|
|
81
|
+
* daemon never handed control to the runner, so nothing was spent -- which is what `container-never-started`
|
|
82
|
+
* means, and why they refund the budget slot instead of keeping it.
|
|
83
|
+
*
|
|
84
|
+
* HERE, in the leaf, rather than in `backend-local.mjs`, so `processor.mjs` can default to them without
|
|
85
|
+
* importing the local adapter and dragging `node:child_process` and the docker CLI machinery into its
|
|
86
|
+
* module graph. The local bundle declares the same constant; a second backend declares its own, or declares
|
|
87
|
+
* `[]` and normalises to the outcome itself.
|
|
88
|
+
*/
|
|
89
|
+
export const DOCKER_NEVER_STARTED_EXITS = Object.freeze([125, 126, 127]);
|
|
90
|
+
|
|
91
|
+
/** The three words. Ordered weakest-last so a floor can be expressed as "at least this". */
|
|
92
|
+
export const ENFORCED = "enforced";
|
|
93
|
+
export const ASSERTED = "asserted";
|
|
94
|
+
export const ABSENT = "absent";
|
|
95
|
+
|
|
96
|
+
/**
|
|
97
|
+
* Strength order, for the floor comparison. `absent` is 0 so an unknown or missing declaration sorts with
|
|
98
|
+
* it: a backend that declares nothing gets no benefit of the doubt, which is the polarity
|
|
99
|
+
* `dev.pi-dispatch.capabilities` already uses for images and for the same reason.
|
|
100
|
+
*/
|
|
101
|
+
const RANK = { [ABSENT]: 0, [ASSERTED]: 1, [ENFORCED]: 2 };
|
|
102
|
+
|
|
103
|
+
/**
|
|
104
|
+
* Whether `have` is at least as strong as `want`.
|
|
105
|
+
*
|
|
106
|
+
* ASYMMETRIC ABOUT AN UNKNOWN WORD, and the asymmetry is the whole subtlety of this function. On the HAVE
|
|
107
|
+
* side an unknown word ranks 0, so a backend declaring gibberish gets no credit -- a failure, as intended.
|
|
108
|
+
* On the WANT side the same rule INVERTS: a floor of `{ egress: "enfroced" }` ranks 0 and is therefore met
|
|
109
|
+
* by everything, so the typo yields the open posture while an operator believes they have a floor.
|
|
110
|
+
*
|
|
111
|
+
* This function does not and cannot fix that, because "met by everything" is also the correct answer for a
|
|
112
|
+
* floor that genuinely asks for `absent`. `isDeclaration` is the guard, and whatever parses a floor must
|
|
113
|
+
* call it -- `PI_EGRESS` refuses a third value for exactly this reason. `shortfall` below does call it.
|
|
114
|
+
*/
|
|
115
|
+
export function meets(have, want) {
|
|
116
|
+
return (RANK[have] ?? 0) >= (RANK[want] ?? 0);
|
|
117
|
+
}
|
|
118
|
+
|
|
119
|
+
/**
|
|
120
|
+
* Is `word` one of the three? `Object.hasOwn`, never `in` or a truthy index, so `"constructor"` and
|
|
121
|
+
* `"toString"` are not declarations.
|
|
122
|
+
*/
|
|
123
|
+
export function isDeclaration(word) {
|
|
124
|
+
// `typeof` first: `Object.hasOwn` coerces via ToPropertyKey, so `["enforced"]` and
|
|
125
|
+
// `{ toString: () => "absent" }` would otherwise be declarations. A floor value that is not a string is
|
|
126
|
+
// a malformed floor, and this function is what stops one being ranked.
|
|
127
|
+
return typeof word === "string" && Object.hasOwn(RANK, word);
|
|
128
|
+
}
|
|
129
|
+
|
|
130
|
+
/**
|
|
131
|
+
* The properties every backend declares, each with the question an operator is really asking. A bare
|
|
132
|
+
* property name is the un-actionable amber this project's design rejects elsewhere.
|
|
133
|
+
*
|
|
134
|
+
* A CLOSED LIST, because a backend that simply omits one would otherwise be admitted for it. That rule is
|
|
135
|
+
* only sound if the list is COMPLETE, so it is derived from what the specs say the container boundary
|
|
136
|
+
* provides rather than from what the local implementation happens to have flags for. The five properties
|
|
137
|
+
* after `localFolders` exist because without them a backend could declare every other word `enforced`,
|
|
138
|
+
* be fully conformant, and still bind-mount the docker socket, reuse one container across mutually
|
|
139
|
+
* untrusting issue authors, or have no way to stop a runaway job.
|
|
140
|
+
*
|
|
141
|
+
* `armedBy` names the deployment switch that gates a property, or null when nothing does. A capability is
|
|
142
|
+
* not a posture (see the header), and a reader must never be handed one where they could take it for the
|
|
143
|
+
* other.
|
|
144
|
+
*/
|
|
145
|
+
const PROPERTIES_TABLE = {
|
|
146
|
+
isolation: {
|
|
147
|
+
question: "capabilities dropped, no-new-privileges, pid and memory bounds on the job container",
|
|
148
|
+
armedBy: null,
|
|
149
|
+
},
|
|
150
|
+
ephemeral: {
|
|
151
|
+
// CONST-ISOLATION-CONTAINER-PER-JOB's core sentence, and NOT covered by `isolation`: a backend
|
|
152
|
+
// reusing one long-lived container satisfies every flag above word for word and still leaks state
|
|
153
|
+
// between mutually untrusting issue authors, which is the whole reason that constraint exists.
|
|
154
|
+
question: "one container per job, destroyed after it, never reused across jobs",
|
|
155
|
+
armedBy: null,
|
|
156
|
+
},
|
|
157
|
+
mountSet: {
|
|
158
|
+
// The Acceptance clause of CONST-ISOLATION-CONTAINER-PER-JOB: "the agent has no filesystem path to
|
|
159
|
+
// the host outside the declared mounts", and it names PI_SESSIONS_DIR as never bind-mounted into any
|
|
160
|
+
// container. Without this property a backend could declare everything else enforced and mount the
|
|
161
|
+
// docker socket, which is not a weakened boundary but no boundary at all.
|
|
162
|
+
question: "only the declared mounts exist; no docker socket, no home dir, no shared session store",
|
|
163
|
+
armedBy: null,
|
|
164
|
+
},
|
|
165
|
+
egress: {
|
|
166
|
+
// ARMED, not unconditional, and the table would be re-proposing a rejected constitution entry if it
|
|
167
|
+
// said otherwise. See the header.
|
|
168
|
+
question: "the job reaches only what the allowlist proxy permits (CONST-EGRESS-POLICY-IN-THE-ARGV)",
|
|
169
|
+
armedBy: "PI_EGRESS",
|
|
170
|
+
},
|
|
171
|
+
jobToJobIsolation: {
|
|
172
|
+
// The measured half of REQ-EGRESS-ALLOWLIST. `egress` is about OUTBOUND reach; this is about whether
|
|
173
|
+
// two jobs can reach each other, which a backend could fail while passing the egress question by
|
|
174
|
+
// routing every job through the proxy on one shared segment. Also armed: with PI_EGRESS=0 both job
|
|
175
|
+
// containers sit on docker's default bridge, where egress.mjs records they can reach each other by
|
|
176
|
+
// IP today.
|
|
177
|
+
question: "two jobs cannot reach each other, structurally rather than by policy",
|
|
178
|
+
armedBy: "PI_EGRESS",
|
|
179
|
+
},
|
|
180
|
+
imagePinning: {
|
|
181
|
+
question: "the image cannot be fetched at run time, so a typo'd name is unreachable (--pull=never)",
|
|
182
|
+
armedBy: null,
|
|
183
|
+
},
|
|
184
|
+
exitCodes: {
|
|
185
|
+
// Narrowed deliberately. An earlier draft also claimed "un-interleaved stdout arrives undistorted",
|
|
186
|
+
// which this project does not have: run-container.mjs tees two independently buffered pipes into one
|
|
187
|
+
// sink, run-history.mjs reads only the last 8KB, and `OQ-003` records that anything sharing the
|
|
188
|
+
// container's stdout can land a partial write inside a runner line. The integer is the part that is
|
|
189
|
+
// actually guaranteed, and it is the part every retry decision rests on.
|
|
190
|
+
question: "the container's integer exit code reaches the processor unmodified (INT-RUNNER-EXIT-CODE-PROTOCOL)",
|
|
191
|
+
armedBy: null,
|
|
192
|
+
},
|
|
193
|
+
abortable: {
|
|
194
|
+
// REQ-JOB-TIMEOUT-30M. Declared two slices before the seam existed, on purpose: `stopContainer` was
|
|
195
|
+
// hard-wired in index.mjs's abort path, so a second backend could have passed every other check with
|
|
196
|
+
// no way to stop a runaway container and nothing in this table would have moved. A gap that is
|
|
197
|
+
// declared is a refusal; a gap that is unnamed is a surprise. The seam landed in slice 4.
|
|
198
|
+
question: "a running job can be stopped on the 30-minute timeout or a shutdown, and the abort is distinguishable from a crash",
|
|
199
|
+
armedBy: null,
|
|
200
|
+
},
|
|
201
|
+
readOnlyJobInputs: {
|
|
202
|
+
question: "/job is read-only by the kernel, not by convention (INT-CONTAINER-JOB-INPUTS)",
|
|
203
|
+
armedBy: null,
|
|
204
|
+
},
|
|
205
|
+
nonRoot: {
|
|
206
|
+
question: "the agent runs as a non-root user",
|
|
207
|
+
armedBy: null,
|
|
208
|
+
},
|
|
209
|
+
secretsCustody: {
|
|
210
|
+
// Stated as what the code actually earns and a test can read back, rather than as "values stay on the
|
|
211
|
+
// operator's host", which is a topological fact about `remote: false` and not something the worker
|
|
212
|
+
// builds. The host-side argv exposure under a default `hidepid` is real, pre-existing, disclosed in
|
|
213
|
+
// SECURITY.md, and deliberately outside this sentence rather than denied by it. WHETHER THE VALUES
|
|
214
|
+
// CROSS A NETWORK is deliberately not asked here either -- that is `credentialTransit`, which answers
|
|
215
|
+
// it honestly, and folding the two would let this one's verified half carry the other's unverified.
|
|
216
|
+
question: "no resolved secret VALUE reaches a log, a run record or a forge comment",
|
|
217
|
+
armedBy: null,
|
|
218
|
+
},
|
|
219
|
+
credentialTransit: {
|
|
220
|
+
// CONST-TOKEN-SCOPED-PER-JOB is what CONST-EGRESS-POLICY-IN-THE-ARGV calls "what actually bounds the
|
|
221
|
+
// damage", and the first draft of this table had no property for it at all. A remote backend must
|
|
222
|
+
// ship the provider key and the per-job forge token to a daemon it does not own; that is a different
|
|
223
|
+
// question from secretsCustody, which is about this host's own logs and records.
|
|
224
|
+
question: "the provider key and the per-job forge token reach the container without crossing a network this deployment does not own",
|
|
225
|
+
armedBy: null,
|
|
226
|
+
},
|
|
227
|
+
localFolders: {
|
|
228
|
+
question: "a local folder can be bind-mounted and edited in place (DES-WORKER-ON-HOST finding 2)",
|
|
229
|
+
armedBy: null,
|
|
230
|
+
},
|
|
231
|
+
};
|
|
232
|
+
|
|
233
|
+
/**
|
|
234
|
+
* FROZEN for the reason `BACKENDS` is, and it is the sharper of the two. `isProperty` reads THIS object,
|
|
235
|
+
* and `isProperty` is the whole of `shortfall`'s validation gate -- so an unfrozen `PROPERTIES` means one
|
|
236
|
+
* assignment can delete a property (every floor naming it then throws "unknown property"), add one
|
|
237
|
+
* (a floor naming it validates and ranks against `absent`), or null out an `armedBy` so the capability word
|
|
238
|
+
* is printable bare again. That last one is the exact fix this table just made, undone from inside.
|
|
239
|
+
*/
|
|
240
|
+
export const PROPERTIES = Object.freeze(
|
|
241
|
+
Object.fromEntries(Object.entries(PROPERTIES_TABLE).map(([name, entry]) => [name, Object.freeze({ ...entry })])),
|
|
242
|
+
);
|
|
243
|
+
|
|
244
|
+
export const PROPERTY_NAMES = Object.freeze(Object.keys(PROPERTIES));
|
|
245
|
+
|
|
246
|
+
/**
|
|
247
|
+
* One property's declaration on one backend, joined with what qualifies it: `{ property, word, armedBy,
|
|
248
|
+
* question }`. `undefined` for an unknown backend or property.
|
|
249
|
+
*
|
|
250
|
+
* Exists because `declares` is a bare `{ property: word }` map, and a consumer that prints a word without
|
|
251
|
+
* its `armedBy` reintroduces the defect the `armedBy` field was added to fix -- a capability read as a
|
|
252
|
+
* posture. One definition of the join, so the first consumer does not have to remember to write it.
|
|
253
|
+
*/
|
|
254
|
+
export function declarationOf(name, property) {
|
|
255
|
+
const entry = backendFor(name);
|
|
256
|
+
if (!entry || !isProperty(property)) return undefined;
|
|
257
|
+
const word = entry.declares[property];
|
|
258
|
+
return {
|
|
259
|
+
property,
|
|
260
|
+
word,
|
|
261
|
+
armedBy: PROPERTIES[property].armedBy,
|
|
262
|
+
question: PROPERTIES[property].question,
|
|
263
|
+
// Only meaningful for an asserted word, and null otherwise rather than absent, so a consumer that
|
|
264
|
+
// prints it unconditionally renders nothing rather than "undefined".
|
|
265
|
+
assertedBy: word === ASSERTED ? (entry.asserts?.[property] ?? null) : null,
|
|
266
|
+
};
|
|
267
|
+
}
|
|
268
|
+
|
|
269
|
+
/** Is `name` one of the closed list? `Object.hasOwn`, so `"toString"` is not a property. */
|
|
270
|
+
export function isProperty(name) {
|
|
271
|
+
return Object.hasOwn(PROPERTIES, name);
|
|
272
|
+
}
|
|
273
|
+
|
|
274
|
+
const BACKENDS_TABLE = {
|
|
275
|
+
local: {
|
|
276
|
+
/**
|
|
277
|
+
* The Docker daemon on the worker's own host -- what every deployment has always used, and what
|
|
278
|
+
* `DES-WORKER-ON-HOST` chose deliberately: the worker runs on the host and shells out to the real
|
|
279
|
+
* `docker` CLI, inheriting its cross-platform path translation rather than reimplementing it.
|
|
280
|
+
*/
|
|
281
|
+
describe: "the Docker daemon on this worker's host",
|
|
282
|
+
remote: false,
|
|
283
|
+
declares: {
|
|
284
|
+
// Built by this worker's own argv, in `ISOLATION_FLAGS`, and asserted back member-by-member
|
|
285
|
+
// against the imported array by two tests. `dockerArgsFromSpec` additionally refuses a
|
|
286
|
+
// `dockerExtra` carrying a flag that would supersede one of them -- membership in an argv is not
|
|
287
|
+
// effectiveness of that argv, and without that guard those two assertions would pass on an argv
|
|
288
|
+
// with no boundary left. It is a deny-list, so it NARROWS that gap rather than closing it; what
|
|
289
|
+
// closes it today is that the only production caller passes fixed literals.
|
|
290
|
+
isolation: ENFORCED,
|
|
291
|
+
// `--rm` leads ISOLATION_FLAGS and the container name carries the job id, so no container is
|
|
292
|
+
// reachable to reuse even in principle.
|
|
293
|
+
ephemeral: ENFORCED,
|
|
294
|
+
// The mount list is built by `containerSpec` from a fixed set of named host paths and nothing
|
|
295
|
+
// else. There is no pass-through, and the shared session store under PI_SESSIONS_DIR is never
|
|
296
|
+
// among them -- only this job's own copy.
|
|
297
|
+
mountSet: ENFORCED,
|
|
298
|
+
// CAPABILITY, gated by PI_EGRESS. When armed: `--network=pi-job-<id>-net`, `--internal`, created
|
|
299
|
+
// before the spawn and removed in a finally. When PI_EGRESS=0 the flag is absent and the job is
|
|
300
|
+
// on docker's default bridge -- which is why this word is `armedBy` rather than flat.
|
|
301
|
+
egress: ENFORCED,
|
|
302
|
+
// Same switch, and the same reason: an `--internal` network holding exactly two endpoints is what
|
|
303
|
+
// makes job-to-job structurally impossible. Unarmed, egress.mjs records the opposite is true.
|
|
304
|
+
jobToJobIsolation: ENFORCED,
|
|
305
|
+
// `--pull=never`, which makes the fetch branch unreachable rather than merely unlikely. Every
|
|
306
|
+
// path that starts a container from the job image carries it, doctor's probes included.
|
|
307
|
+
imagePinning: ENFORCED,
|
|
308
|
+
// `spawn` on the CLI: the integer comes from the `close` event and is passed to `decideRetry`
|
|
309
|
+
// unmodified.
|
|
310
|
+
exitCodes: ENFORCED,
|
|
311
|
+
// `docker stop` on the name, and the abort FLAG rather than the code is the discriminator -- a
|
|
312
|
+
// worker SIGKILL and a kernel OOM both surface as 137. The worker also BOUNDS the wait after the
|
|
313
|
+
// abort, so a stop that does not take frees the slot instead of holding it forever.
|
|
314
|
+
abortable: ENFORCED,
|
|
315
|
+
// `-v <host>:/job:ro` -- the kernel enforces it. `verify-image.sh` proves it with a live write
|
|
316
|
+
// attempt rather than trusting the flag.
|
|
317
|
+
readOnlyJobInputs: ENFORCED,
|
|
318
|
+
// ASSERTED, not enforced, and this is the honest one. Non-root is `USER pi` in the IMAGE, not in
|
|
319
|
+
// the worker's argv -- SECURITY.md says so in terms: "Non-root is not in that argv." An
|
|
320
|
+
// operator-built image can run as root and nothing here would refuse it (`OQ-012`).
|
|
321
|
+
nonRoot: ASSERTED,
|
|
322
|
+
// Traced end to end: the resolver logs a stderr BYTE COUNT and never the bytes, the refusal
|
|
323
|
+
// comment carries neither the reference nor the resolver's path nor a byte of what it printed,
|
|
324
|
+
// and `buildRecord` is an explicit literal with no spread. Where the values GO is
|
|
325
|
+
// `credentialTransit`'s question, not this one's.
|
|
326
|
+
secretsCustody: ENFORCED,
|
|
327
|
+
// ASSERTED, and this is the second honest one. The intent is that the daemon is on this host, so
|
|
328
|
+
// nothing leaves it -- but every spawn is `docker` with the worker's own environment inherited,
|
|
329
|
+
// and `DOCKER_HOST=tcp://...` or a `docker context` redirects that connection to another machine
|
|
330
|
+
// with the provider key and the per-job forge token riding along as `-e NAME=VALUE`. `DOCKER_HOST`
|
|
331
|
+
// appears NOWHERE in this repository: no code sets it, no check refuses it, no test reads it back.
|
|
332
|
+
// By this file's own definition of `enforced` -- "this worker CAN build it, in its own code, and a
|
|
333
|
+
// test in this repo reads it back" -- there is no such code, so the word would be exactly the
|
|
334
|
+
// overclaim `nonRoot` avoids one property up. A boot check on DOCKER_HOST would earn `enforced`.
|
|
335
|
+
credentialTransit: ASSERTED,
|
|
336
|
+
// The whole reason `DES-WORKER-ON-HOST` reversed the containerised worker.
|
|
337
|
+
localFolders: ENFORCED,
|
|
338
|
+
},
|
|
339
|
+
/**
|
|
340
|
+
* WHO is asserting each `asserted` property. Required for every property this backend declares
|
|
341
|
+
* ASSERTED and meaningless for the others, which a test pins both ways.
|
|
342
|
+
*
|
|
343
|
+
* Exists because "asserted" alone is not actionable: it tells an operator the worker is not the one
|
|
344
|
+
* providing the property without telling them who is, so they cannot go and check. `doctor` prints
|
|
345
|
+
* this beside the word, which is what lets the claim "asserted names who is asserting it" be true
|
|
346
|
+
* rather than aspirational. For a vendor adapter this is where "the vendor's documentation" goes.
|
|
347
|
+
*/
|
|
348
|
+
asserts: {
|
|
349
|
+
nonRoot: "the job image's USER directive (this repo's builds `USER pi`; an operator-built image may not)",
|
|
350
|
+
credentialTransit: "the docker endpoint DOCKER_HOST resolves to, which is this host unless something redirects it",
|
|
351
|
+
},
|
|
352
|
+
},
|
|
353
|
+
};
|
|
354
|
+
|
|
355
|
+
/**
|
|
356
|
+
* FROZEN, deeply. `makeLocalBackend` hands `BACKENDS[name].declares` out on the bundle it returns, so
|
|
357
|
+
* without this a consumer holding a backend could assign one word and silently rewrite what every later
|
|
358
|
+
* reader -- `doctor`, the boot refusal, the receiver -- is told about that backend. A shared mutable
|
|
359
|
+
* security declaration is the believed-in control this file exists to prevent, arriving from inside.
|
|
360
|
+
*/
|
|
361
|
+
export const BACKENDS = Object.freeze(
|
|
362
|
+
Object.fromEntries(
|
|
363
|
+
Object.entries(BACKENDS_TABLE).map(([name, entry]) => [name, Object.freeze({ ...entry, declares: Object.freeze({ ...entry.declares }) })]),
|
|
364
|
+
),
|
|
365
|
+
);
|
|
366
|
+
|
|
367
|
+
export const BACKEND_NAMES = Object.freeze(Object.keys(BACKENDS));
|
|
368
|
+
|
|
369
|
+
/** The default, and the name a deployment that has never heard of this table is running. */
|
|
370
|
+
export const DEFAULT_BACKEND = "local";
|
|
371
|
+
|
|
372
|
+
/**
|
|
373
|
+
* One backend's entry, or `undefined`. The caller decides how loudly an unknown name fails.
|
|
374
|
+
*
|
|
375
|
+
* `Object.hasOwn` rather than a bare index, because the callers this module's header describes are written
|
|
376
|
+
* as `if (!backendFor(name)) refuse()` -- and a bare index walks the prototype chain, so `constructor` and
|
|
377
|
+
* `toString` would return a truthy function and walk straight past that guard.
|
|
378
|
+
*/
|
|
379
|
+
export function backendFor(name) {
|
|
380
|
+
const key = name ?? DEFAULT_BACKEND;
|
|
381
|
+
return Object.hasOwn(BACKENDS, key) ? BACKENDS[key] : undefined;
|
|
382
|
+
}
|
|
383
|
+
|
|
384
|
+
/**
|
|
385
|
+
* Which declared properties fall short of `want`, as `[{ property, have, want }]`. Empty means it meets it.
|
|
386
|
+
*
|
|
387
|
+
* Returns the SHORTFALL rather than a boolean because every caller has to name what is missing: a refusal
|
|
388
|
+
* that says only "this backend does not meet the floor" sends an operator to read a table, and the whole
|
|
389
|
+
* point of the three words is that the failing one can be printed.
|
|
390
|
+
*
|
|
391
|
+
* THROWS on a floor this module cannot read: an unknown property name, or a value that is not one of the
|
|
392
|
+
* three words. Both are the same failure -- a typo that would otherwise be INVISIBLE. Iterating the closed
|
|
393
|
+
* property list would silently drop `{ egres: "enforced" }`, and ranking an unknown word would silently
|
|
394
|
+
* admit `{ egress: "enfroced" }`; either way the caller gets `[]` back, which is indistinguishable from a
|
|
395
|
+
* satisfied floor, and an operator believes they have a bound they do not have. A config error at boot is
|
|
396
|
+
* the loud, correct end of that, and it is why this validates rather than tolerates.
|
|
397
|
+
*/
|
|
398
|
+
export function shortfall(name, want = {}) {
|
|
399
|
+
const entry = backendFor(name);
|
|
400
|
+
const out = [];
|
|
401
|
+
// `?? {}` as well as the parameter default: the default only fires for `undefined`, and a floor that
|
|
402
|
+
// arrived as JSON `null` is the same "asked for nothing" case, not a TypeError.
|
|
403
|
+
for (const property of Object.keys(want ?? {})) {
|
|
404
|
+
if (!isProperty(property)) {
|
|
405
|
+
throw new Error(`backend floor: unknown property ${JSON.stringify(property)} (known: ${PROPERTY_NAMES.join(", ")})`);
|
|
406
|
+
}
|
|
407
|
+
const need = (want ?? {})[property];
|
|
408
|
+
if (!isDeclaration(need)) {
|
|
409
|
+
throw new Error(`backend floor: ${property} must be one of ${ENFORCED}, ${ASSERTED}, ${ABSENT}; got ${JSON.stringify(need)}`);
|
|
410
|
+
}
|
|
411
|
+
const have = entry?.declares?.[property] ?? ABSENT;
|
|
412
|
+
if (!meets(have, need)) out.push({ property, have, want: need });
|
|
413
|
+
}
|
|
414
|
+
return out;
|
|
415
|
+
}
|
|
416
|
+
|
|
417
|
+
/**
|
|
418
|
+
* `PI_BACKENDS` -- which backends this deployment blesses, comma separated. Unset means `[local]`, which is
|
|
419
|
+
* what every deployment that has never heard of this table is already running.
|
|
420
|
+
*
|
|
421
|
+
* ENV-ONLY, never the settings overlay and never the deployment pointer, on `config.mjs`'s rule for
|
|
422
|
+
* `PI_SECRET_RESOLVER_ROOTS`: "a bound that can be widened from the surface it bounds is not a bound". The
|
|
423
|
+
* pointer needs no change to enforce that -- `POINTER_ENV_ALLOWLIST` is an ALLOWLIST (of the path and URL
|
|
424
|
+
* variables `resolvePaths` reads), so a name absent from it is refused by omission.
|
|
425
|
+
*
|
|
426
|
+
* An unknown name is REFUSED rather than dropped. Dropping it would leave an operator who misspelled their
|
|
427
|
+
* one entry with a silently empty set, and a deployment that blesses nothing is a deployment where every
|
|
428
|
+
* trigger naming a backend is refused for a reason that names the trigger rather than the typo.
|
|
429
|
+
*
|
|
430
|
+
* Throws a plain Error; `config.mjs` re-tags it as a config error, which is `egressArmed`'s arrangement and
|
|
431
|
+
* for its reason: this module imports nothing, so it cannot reach for that tagger itself.
|
|
432
|
+
*/
|
|
433
|
+
export function parseBackendList(raw) {
|
|
434
|
+
const names = (raw ?? "")
|
|
435
|
+
.split(",")
|
|
436
|
+
.map((s) => s.trim())
|
|
437
|
+
.filter((s) => s.length > 0);
|
|
438
|
+
if (names.length === 0) return [DEFAULT_BACKEND];
|
|
439
|
+
for (const name of names) {
|
|
440
|
+
if (!Object.hasOwn(BACKENDS, name)) {
|
|
441
|
+
throw new Error(`PI_BACKENDS names an unknown backend ${JSON.stringify(name)} (known: ${BACKEND_NAMES.join(", ")})`);
|
|
442
|
+
}
|
|
443
|
+
}
|
|
444
|
+
// Deduplicated, order preserved: the first entry is what a deployment means by "the default one".
|
|
445
|
+
const unique = [...new Set(names)];
|
|
446
|
+
// The DEFAULT backend must stay in the set. A trigger that names no venue is dispatched to
|
|
447
|
+
// `backends[0]`, and the boot registry refuses a default it does not hold -- so a set excluding it would
|
|
448
|
+
// describe a deployment whose unflagged triggers, which is nearly all of them, have nowhere to run.
|
|
449
|
+
if (!unique.includes(DEFAULT_BACKEND)) {
|
|
450
|
+
throw new Error(`PI_BACKENDS must include ${JSON.stringify(DEFAULT_BACKEND)}: a trigger that names no backend is dispatched there, so a set without it leaves every unflagged trigger nowhere to run`);
|
|
451
|
+
}
|
|
452
|
+
return unique;
|
|
453
|
+
}
|
|
454
|
+
|
|
455
|
+
/**
|
|
456
|
+
* `PI_BACKEND_FLOOR` -- the minimum every blessed backend must declare, as `property=word` pairs, comma
|
|
457
|
+
* separated. Example: `egress=enforced,nonRoot=asserted`. Unset means no floor.
|
|
458
|
+
*
|
|
459
|
+
* PARSED HERE, IN THE LEAF, which is `egressArmed`'s pattern and its reason: `doctor`, `up` and the worker
|
|
460
|
+
* must not be able to disagree about what the floor says. One parse, one answer, and a typo throws at the
|
|
461
|
+
* one place that reads the string rather than defaulting three consumers to three different open postures.
|
|
462
|
+
*
|
|
463
|
+
* Every part is validated and NOTHING is skipped. A pair with no `=`, an unknown property name, an unknown
|
|
464
|
+
* word: each throws. That strictness is the whole point of the variable existing -- `shortfall` returning
|
|
465
|
+
* `[]` is indistinguishable from a satisfied floor, so a floor this module cannot read must never be
|
|
466
|
+
* allowed to become one it reads as empty. `PI_EGRESS` refuses a third value for the same reason.
|
|
467
|
+
*/
|
|
468
|
+
export function parseBackendFloor(raw) {
|
|
469
|
+
const floor = {};
|
|
470
|
+
for (const pair of (raw ?? "").split(",").map((s) => s.trim()).filter((s) => s.length > 0)) {
|
|
471
|
+
const at = pair.indexOf("=");
|
|
472
|
+
if (at <= 0) {
|
|
473
|
+
throw new Error(`PI_BACKEND_FLOOR entry ${JSON.stringify(pair)} must be property=word (e.g. egress=enforced)`);
|
|
474
|
+
}
|
|
475
|
+
const property = pair.slice(0, at).trim();
|
|
476
|
+
const word = pair.slice(at + 1).trim();
|
|
477
|
+
if (!isProperty(property)) {
|
|
478
|
+
throw new Error(`PI_BACKEND_FLOOR names an unknown property ${JSON.stringify(property)} (known: ${PROPERTY_NAMES.join(", ")})`);
|
|
479
|
+
}
|
|
480
|
+
if (!isDeclaration(word)) {
|
|
481
|
+
throw new Error(`PI_BACKEND_FLOOR: ${property} must be one of ${ENFORCED}, ${ASSERTED}, ${ABSENT}; got ${JSON.stringify(word)}`);
|
|
482
|
+
}
|
|
483
|
+
if (Object.hasOwn(floor, property)) {
|
|
484
|
+
// Last-wins would be a silent choice between two things an operator wrote down deliberately.
|
|
485
|
+
throw new Error(`PI_BACKEND_FLOOR names ${property} twice`);
|
|
486
|
+
}
|
|
487
|
+
floor[property] = word;
|
|
488
|
+
}
|
|
489
|
+
return floor;
|
|
490
|
+
}
|
|
491
|
+
|
|
492
|
+
/**
|
|
493
|
+
* Which floored properties this deployment is NOT getting because their switch is off, as
|
|
494
|
+
* `[{ property, want, armedBy }]`.
|
|
495
|
+
*
|
|
496
|
+
* THE FLOOR IS NOT MET BY CAPABILITY ALONE, and reading it that way was the defect this function exists to
|
|
497
|
+
* close. `shortfall` compares a floor against what a backend CAN do, which is right for the question "could
|
|
498
|
+
* this deployment's jobs ever get this here". It is not the question an operator asks by writing a floor.
|
|
499
|
+
* `local` declares `egress: enforced` whether or not `PI_EGRESS` is armed, so `PI_BACKEND_FLOOR=egress=enforced`
|
|
500
|
+
* on a `PI_EGRESS=0` deployment passed `shortfall` and booted -- and `doctor` then printed "this deployment
|
|
501
|
+
* is not getting it" two lines above "PI_BACKEND_FLOOR holds". Every job ran on docker's default bridge
|
|
502
|
+
* while the operator had asked, in writing, for the opposite.
|
|
503
|
+
*
|
|
504
|
+
* That is the believed-in control `CONST-EGRESS-POLICY-IN-THE-ARGV` describes, arriving through the very
|
|
505
|
+
* mechanism added to discharge it, so the floor reads the switch too. Anything above `absent` on a gated
|
|
506
|
+
* property requires the switch to be ON: a floor asking for `absent` is asking for nothing and is met.
|
|
507
|
+
*
|
|
508
|
+
* `switches` maps an `armedBy` name to whether it is armed. `undefined` means "not known" and is treated as
|
|
509
|
+
* NOT armed, on this file's standing polarity: a thing that cannot be shown to be on gets no credit.
|
|
510
|
+
*/
|
|
511
|
+
export function unarmedFloor(floor, switches = {}) {
|
|
512
|
+
const out = [];
|
|
513
|
+
for (const [property, want] of Object.entries(floor ?? {})) {
|
|
514
|
+
if (!isProperty(property) || want === ABSENT) continue;
|
|
515
|
+
const armedBy = PROPERTIES[property].armedBy;
|
|
516
|
+
if (armedBy && switches[armedBy] !== true) out.push({ property, want, armedBy });
|
|
517
|
+
}
|
|
518
|
+
return out;
|
|
519
|
+
}
|
|
520
|
+
|
|
521
|
+
/**
|
|
522
|
+
* Every reason the blessed set fails the floor, as `[{ backend, property, have, want }]`. Empty means the
|
|
523
|
+
* deployment is admissible.
|
|
524
|
+
*
|
|
525
|
+
* WHY EVERY BACKEND RATHER THAN THE SELECTED ONE: a floor is a statement about where this deployment's jobs
|
|
526
|
+
* may run, and any blessed backend is somewhere they may run. Checking only the default would let an
|
|
527
|
+
* operator bless a backend that fails the floor and reach it from a trigger, which is the floor widened
|
|
528
|
+
* from the surface it bounds.
|
|
529
|
+
*/
|
|
530
|
+
export function floorShortfall(names, floor) {
|
|
531
|
+
const out = [];
|
|
532
|
+
for (const backend of names ?? []) {
|
|
533
|
+
for (const miss of shortfall(backend, floor)) out.push({ backend, ...miss });
|
|
534
|
+
}
|
|
535
|
+
return out;
|
|
536
|
+
}
|
|
537
|
+
|
|
538
|
+
/**
|
|
539
|
+
* Every reason this deployment's configuration is inadmissible, as a list of operator-facing messages.
|
|
540
|
+
* Empty means it may boot.
|
|
541
|
+
*
|
|
542
|
+
* THE LADDERS LIVE HERE, IN THE LEAF, rather than in `config.mjs`, for the reason the parsers do: a rule
|
|
543
|
+
* reachable only through `loadConfig` can only be tested through whatever backend names `parseBackendList`
|
|
544
|
+
* currently accepts -- which today is exactly one. Three separate mutations of these rules survived a
|
|
545
|
+
* mutation pass for precisely that reason: the tests could not construct a deployment that violated them.
|
|
546
|
+
* As a pure function over an explicit backend list, each rule can be driven against a backend that fails
|
|
547
|
+
* it, so the rule is pinned rather than merely present. `config.mjs` re-tags these as config errors.
|
|
548
|
+
*
|
|
549
|
+
* Three questions, in the order an operator can act on them:
|
|
550
|
+
*
|
|
551
|
+
* 1. Does every blessed backend clear the floor the operator wrote? Checked against EVERY member rather
|
|
552
|
+
* than the default alone, because any blessed backend is somewhere this deployment's jobs may run.
|
|
553
|
+
* 2. Is the operator's floor asking for something they have themselves switched off? Capability is not
|
|
554
|
+
* posture, and a floor met only in principle is the belief this whole vocabulary exists to prevent.
|
|
555
|
+
* 3. Has the deployment armed a control its backend cannot provide at all? Implied rather than written:
|
|
556
|
+
* arming egress IS asking for egress, whatever the floor says.
|
|
557
|
+
*/
|
|
558
|
+
export function backendRefusals({ backends = [], backendFloor = {}, egress = false } = {}) {
|
|
559
|
+
const out = [];
|
|
560
|
+
|
|
561
|
+
const misses = floorShortfall(backends, backendFloor);
|
|
562
|
+
if (misses.length > 0) {
|
|
563
|
+
const lines = misses.map((m) => ` ${m.backend}: ${m.property} is ${m.have}, PI_BACKEND_FLOOR wants ${m.want}`);
|
|
564
|
+
out.push(`PI_BACKEND_FLOOR is not met by every backend in PI_BACKENDS:\n${lines.join("\n")}`);
|
|
565
|
+
}
|
|
566
|
+
|
|
567
|
+
const unarmed = unarmedFloor(backendFloor, { PI_EGRESS: egress });
|
|
568
|
+
if (unarmed.length > 0) {
|
|
569
|
+
const lines = unarmed.map((u) => ` ${u.property}=${u.want} requires ${u.armedBy}, which is off`);
|
|
570
|
+
out.push(
|
|
571
|
+
`PI_BACKEND_FLOOR asks for something this deployment has switched off:\n${lines.join("\n")}\n` +
|
|
572
|
+
"Arm the switch, or lower that entry to `absent` if you did not mean to require it.",
|
|
573
|
+
);
|
|
574
|
+
}
|
|
575
|
+
|
|
576
|
+
// EVERY property the armed switch gates, not just `egress`. `armedBy` is a general field and
|
|
577
|
+
// `jobToJobIsolation` carries the same switch -- the table calls it "the measured half of
|
|
578
|
+
// REQ-EGRESS-ALLOWLIST" -- so naming one property here would have let a backend that cannot keep two
|
|
579
|
+
// jobs apart be blessed under an armed policy with nothing said. Hardcoding one variable is the defect
|
|
580
|
+
// `armedBy` was added to prevent, and it had crept back in one function over.
|
|
581
|
+
const switches = { PI_EGRESS: egress };
|
|
582
|
+
for (const property of PROPERTY_NAMES) {
|
|
583
|
+
const armedBy = PROPERTIES[property].armedBy;
|
|
584
|
+
if (!armedBy || switches[armedBy] !== true) continue;
|
|
585
|
+
const unarmable = floorShortfall(backends, { [property]: ASSERTED });
|
|
586
|
+
if (unarmable.length === 0) continue;
|
|
587
|
+
const names = unarmable.map((m) => m.backend);
|
|
588
|
+
out.push(
|
|
589
|
+
`${armedBy} is armed but ${names.join(", ")} ${names.length === 1 ? "declares" : "declare"} ${property} absent, so that control cannot exist there. ` +
|
|
590
|
+
`Set ${armedBy}=0 to run without it, or remove that backend from PI_BACKENDS.`,
|
|
591
|
+
);
|
|
592
|
+
}
|
|
593
|
+
|
|
594
|
+
return out;
|
|
595
|
+
}
|