@parcha/pi-durable-disk 0.1.0-beta.15 → 0.1.0-beta.16
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +45 -0
- package/README.md +49 -6
- package/dist/hosts/daytona.d.ts +25 -0
- package/dist/hosts/daytona.js +59 -1
- package/dist/index.d.ts +3 -3
- package/dist/index.js +1 -1
- package/dist/place.d.ts +51 -5
- package/dist/place.js +341 -8
- package/dist/run.js +11 -2
- package/dist/status.d.ts +21 -0
- package/dist/status.js +79 -0
- package/dist/supervise.d.ts +20 -13
- package/dist/supervise.js +4 -17
- package/package.json +2 -2
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,51 @@
|
|
|
2
2
|
|
|
3
3
|
## Unreleased
|
|
4
4
|
|
|
5
|
+
## 0.1.0-beta.16 (2026-10-18)
|
|
6
|
+
|
|
7
|
+
- **A place's outcome carries the models the run's live parts chose (`PlaceOutcome.models`)**
|
|
8
|
+
- `PlaceOutcome.models?: Record<string, string>`: the role-to-model ids (`lead`, `eyes`, ...) the app reported, on any
|
|
9
|
+
outcome status, so a supervisor's manifest names the models of a remote run as of a local one.
|
|
10
|
+
- `archilPlace` reads it from `run.json`'s final status detail (`detail.models`); `localPlace` from the instance's
|
|
11
|
+
`{"outcome":...}` line, or from `run.json` without one.
|
|
12
|
+
- Both keep it only when it is a plain object whose values are all strings. A missing or malformed one is left out, never
|
|
13
|
+
replaced by `{}`. Before, `localPlace` passed the outcome line's `models` through unchecked, a malformed one included,
|
|
14
|
+
and `archilPlace` never carried it.
|
|
15
|
+
- **`archilPlace.recordHome`: a runner on a box records into the run's disk, and the place brings the recording home**
|
|
16
|
+
- `Place.recordHome` (`PlaceRecordHome`): `dir(attempt)` is where that attempt's runner records, and
|
|
17
|
+
`bringHome(into, { final })` brings what it recorded into the supervisor's recording directory. `archilPlace` has
|
|
18
|
+
it when its control can list (`RecordControl`, as `archilControl` does); `localPlace` has none, since its runner
|
|
19
|
+
writes the supervisor's directory itself.
|
|
20
|
+
- `dir(attempt)` is `<root>/record/<attempt>`, inside the run's directory. `bringHome` reads it over S3 (list and get,
|
|
21
|
+
never the run's claim). Blobs come first, hash-checked and written whole; then the new whole lines of each
|
|
22
|
+
`.spool/*.jsonl` segment, with an offset per file name kept as what `into` already holds, so it is idempotent across
|
|
23
|
+
calls and processes. Carriers that overlap on one destination append each line once: one call at a time per
|
|
24
|
+
destination in a place, and a lock file per spool file across places and processes (a gone holder's is taken over;
|
|
25
|
+
one held past 5 s is `CARRY_FAILED`). Every write is carried to its last byte.
|
|
26
|
+
- With `ensure.create`, each start first makes the EMPTY `signals/`, `record/<attempt>/.spool/` and
|
|
27
|
+
`record/<attempt>/blobs/` over the control, owned by the run user (mode 0755), while no delegation holds the run. A
|
|
28
|
+
directory made by path on an Archil mount was seen to refuse file creates. Every file in them stays the runner's; a
|
|
29
|
+
refused put fails the start with `DIRS_REFUSED`.
|
|
30
|
+
- Box receipts: `HostDriver.box(handle)` (`HostBox`) and `PlaceStatus.boxes`. `archilPlace.status()` lists a receipt
|
|
31
|
+
for each box it started, released ones included: provider, kind, `from`, `to` and, when it has a price, the published
|
|
32
|
+
list price per second (`usdPerSecond`), the page it was read from (`source`) and the date read (`priced`). An unpriced
|
|
33
|
+
box has none of the three, never 0, and no field holds the provider's box id. `daytonaHost` reports its sandboxes, priced at `DAYTONA_LIST_PRICES`
|
|
34
|
+
(https://www.daytona.io/pricing, read 2026-10-16), with the "first 5 GiB free" applied to disk (the footnote's
|
|
35
|
+
placement is ambiguous). A box of unknown size has no price, nor does a box with a GPU, which the page prices per
|
|
36
|
+
model (`daytonaUsdPerSecond`). `localPlace` reports none.
|
|
37
|
+
- Failures carry codes (`PlaceError`). `BLOB_MISMATCH` is thrown after everything else came home. `CARRY_FAILED` is
|
|
38
|
+
thrown for a list or get that fails, and on `final` for a line whose blob never landed. `BOX_GONE` is never thrown
|
|
39
|
+
by this place: the recording lives on the disk, which outlives the box.
|
|
40
|
+
- **A seed of any length stays in the supervisor's start grace**
|
|
41
|
+
- While a new run seeds it has no `run.json` heartbeat, and `ensureRunning` keeps a start in its grace only while the
|
|
42
|
+
start mark (`runs/<id>/start.json`) is younger than the grace (90 s by default). A longer seed was taken as a failed
|
|
43
|
+
start: revoked, then seeded again.
|
|
44
|
+
- Now, while the seed runs, the instance rewrites the mark's `at` every heartbeat through its claim (fsync and rename).
|
|
45
|
+
It does so only when a supervisor's mark for generation 1 is there, keeping its generation, `by`, `failures` and
|
|
46
|
+
`lastExit`. The rewrite runs on a thread of its own, one at a time, so a seed that blocks its thread (a synchronous
|
|
47
|
+
import) stays covered. It stops when the seed ends, before the open goes on, and `run.json`'s heartbeat takes over.
|
|
48
|
+
- A seeder that dies stops rewriting, and the grace runs out as for any start. The mark's format does not change.
|
|
49
|
+
|
|
5
50
|
## 0.1.0-beta.15 (2026-10-16)
|
|
6
51
|
|
|
7
52
|
- **`HostDriver.kill`: take an instance away as a crash would, and `kills` says what that takes**
|
package/README.md
CHANGED
|
@@ -422,7 +422,13 @@ except after exit 65, 70, 75 or 76. `localHost({ mode: "child" })` runs a detach
|
|
|
422
422
|
without that guarantee.
|
|
423
423
|
|
|
424
424
|
`daytonaHost({ client, snapshot, ... })` runs each instance in a Daytona sandbox of its own (library only: `supervise`
|
|
425
|
-
starts local units, so pass the driver to `ensureRunning`).
|
|
425
|
+
starts local units, so pass the driver to `ensureRunning`). Its `box(handle)` is that sandbox's receipt: its size
|
|
426
|
+
(vCPUs, memory, disk), when it was made and when the driver deleted it, and its list price per second at
|
|
427
|
+
`DAYTONA_LIST_PRICES`. Those are Daytona's published rates, with the page and the date they were read
|
|
428
|
+
(https://www.daytona.io/pricing, 2026-10-16): $0.0504 per vCPU-hour, $0.0162 per GiB-hour of memory and $0.000108 per
|
|
429
|
+
GiB-hour of disk past the first 5 GiB, billed per second. The page's "first 5 free" footnote is applied to disk only (its
|
|
430
|
+
placement is ambiguous). A box of unknown size gets no price, and neither does a box with a GPU: the page prices GPUs per
|
|
431
|
+
model, which the driver doesn't map (`daytonaUsdPerSecond`). The box comes from a snapshot that holds Node, the archil client
|
|
426
432
|
(`libfuse2`), this package, the run user and the sudoers drop-in, or from a default image that the caller's `prepare` installs
|
|
427
433
|
them into. A container sandbox has no systemd (PID 1 is the Daytona daemon), so an in-box launcher keeps the instance
|
|
428
434
|
alive in place and never restarts it after exit 0, 65, 70, 75 or 76. Every incarnation of an attempt gets the attempt's
|
|
@@ -757,10 +763,14 @@ await openDurableRun(ref, {
|
|
|
757
763
|
- **When it fails.** A seed that throws fails the open with its own error. A process killed while it seeds leaves the run
|
|
758
764
|
unpublished in the same way: no `run.json`, the claim let go. The next open seeds again, so a seed overwrites what an
|
|
759
765
|
earlier attempt wrote.
|
|
760
|
-
- **How long it may take.**
|
|
761
|
-
`startGraceMs`; default the lease expiry, 90 s)
|
|
762
|
-
|
|
763
|
-
|
|
766
|
+
- **How long it may take.** A new run has no `run.json` heartbeat while it seeds. A supervisor keeps a start in its grace
|
|
767
|
+
(`--start-grace`, `startGraceMs`; default the lease expiry, 90 s) only while its start mark (`start.json`) is younger
|
|
768
|
+
than the grace. So while the seed runs, the instance rewrites the mark's time every heartbeat, through its claim, when a
|
|
769
|
+
supervisor's mark for generation 1 is there. It does so on a thread of its own, so a seed that holds its thread (a
|
|
770
|
+
synchronous import through `DatabaseSync`) is covered too. A live seed of any length stays `starting`. A seeder that dies stops
|
|
771
|
+
rewriting, the grace runs out as for any start, and the claim is revoked. If the claim is revoked while the seed still
|
|
772
|
+
runs, its next write or its barrier is refused, the instance is fenced (exit 75) with nothing published, and the next
|
|
773
|
+
start seeds again.
|
|
764
774
|
- **On a local directory** (`localRun`) it is the same. As with `work/`, the files it writes are the operating system's
|
|
765
775
|
to flush; a SQLite database it commits with `synchronous = FULL` is durable when the commit returns.
|
|
766
776
|
|
|
@@ -887,7 +897,35 @@ id and directory as the supervisor computed them.
|
|
|
887
897
|
- A recovery is the takeover: the killed holder's delegations are revoked whatever its lease says, and the next
|
|
888
898
|
instance starts. It reports `revoked` and `ms`. A newer incarnation than the one the place killed (another
|
|
889
899
|
supervisor's) gets the usual lease and start-grace checks instead, so a live one is `HELD`, not revoked.
|
|
890
|
-
- `status()` reads `run.json` over S3 and lists the run's delegations
|
|
900
|
+
- `status()` reads `run.json` over S3 and lists the run's delegations, plus, when its host driver makes boxes
|
|
901
|
+
(`daytonaHost`), a receipt for each box the place started, released ones included (`boxes`, `HostBox`). A receipt
|
|
902
|
+
has the provider, the kind, `from` and `to` (null while it lives). It also has the published list price per second
|
|
903
|
+
(`usdPerSecond`), the page that price was read from (`source`) and the ISO date it was read (`priced`). A box with
|
|
904
|
+
no price has none of those three, never a price of 0. No field holds the provider's id of the box.
|
|
905
|
+
- `recordHome` (when the control can list, as `archilControl`'s can): an attempt's runner records into
|
|
906
|
+
`recordHome.dir(attempt)`, `<root>/record/<attempt>`, in the run's directory, so its delegation covers it and
|
|
907
|
+
`deleteRunTree` removes it. `bringHome(into, { final })` reads that directory through the control (S3 list and get:
|
|
908
|
+
nothing takes the run's claim) and brings the recording into the supervisor's `into`.
|
|
909
|
+
- Blobs come first, each checked against the hash its name carries and written whole (a temporary file, fsync,
|
|
910
|
+
rename). One that doesn't match is skipped; everything else still comes home, then the call throws
|
|
911
|
+
`BLOB_MISMATCH`.
|
|
912
|
+
- Then the new whole lines of each `.spool/*.jsonl` segment, with an offset per file name: what `into` already
|
|
913
|
+
holds of it. So a second call, or a call from another process, brings nothing twice. A line waits while a blob it
|
|
914
|
+
names isn't home yet. On `final` it comes home anyway, and the call throws `CARRY_FAILED` naming the blob.
|
|
915
|
+
- Carriers that overlap on one `into` never append at once. The place runs one call at a time per destination, and
|
|
916
|
+
each append happens under a lock file beside its spool file (`.<name>.lock`, naming its holder's pid), after the
|
|
917
|
+
destination's size is read again, so only the lines past it are appended. A lock whose holder is gone is taken
|
|
918
|
+
over; one held past 5 s is `CARRY_FAILED`. A second `into` gets every line: what the place fetched for one
|
|
919
|
+
destination says nothing of another.
|
|
920
|
+
- A list or get that fails is `CARRY_FAILED`.
|
|
921
|
+
- The recording is on the disk, not in the box, so a final call after a box kill still brings home everything the
|
|
922
|
+
box fsynced: this place never throws `BOX_GONE`.
|
|
923
|
+
- With `ensure.create` (the run user's uid and gid), the place makes the EMPTY directories the runner writes into
|
|
924
|
+
at each start: `signals/`, `record/<attempt>/.spool/` and `record/<attempt>/blobs/`. It makes them over the
|
|
925
|
+
control, owned by the run user, mode 0755, in ensureRunning's window where no delegation holds the run. A
|
|
926
|
+
directory a process made by path on an Archil mount was seen to refuse file creates afterwards (ENOENT), and one
|
|
927
|
+
made over the control didn't. Every file in them stays the runner's. A refused put fails the start
|
|
928
|
+
(`DIRS_REFUSED`, as the cause of `HOST_START_FAILED`), and without an owner nothing is made.
|
|
891
929
|
- `requestFile(attempt)` is `<root>/signals/kill-<attempt>.json`, in the run's directory and outside `work/`, and
|
|
892
930
|
`signals(attempt)` reads `runs/<id>/signals/kill-<attempt>.json` over S3 every 500 ms (`signalPollMs`). A runner on
|
|
893
931
|
a box of its own can't reach the supervisor's files, so its request travels through the disk. The runner must write
|
|
@@ -912,6 +950,11 @@ when the run is `done` and that detail carries a record, `failed` for anything e
|
|
|
912
950
|
run its instance left `running`, `paused` or `sleeping`). So an app that runs on a box and wants its supervisor to have
|
|
913
951
|
its record puts the record there, in its final `setStatus("done", { record })`.
|
|
914
952
|
|
|
953
|
+
Either way the outcome may carry `models`, the role-to-model ids the run's live parts chose (`lead`, `eyes`, ...), on any
|
|
954
|
+
status, a failed run's too: from the outcome line's `models`, or from the final detail's (`setStatus("done", { record,
|
|
955
|
+
models })`). Both places keep it only when it is a plain object whose values are all strings; a missing or malformed one
|
|
956
|
+
is left out, never replaced by `{}`.
|
|
957
|
+
|
|
915
958
|
## Tests
|
|
916
959
|
|
|
917
960
|
```sh
|
package/dist/hosts/daytona.d.ts
CHANGED
|
@@ -7,12 +7,37 @@ export type SandboxInfo = {
|
|
|
7
7
|
id: string;
|
|
8
8
|
name: string;
|
|
9
9
|
state?: string;
|
|
10
|
+
/** The box's size as Daytona allocates it: vCPUs, GPUs, memory and disk (the API's GB). */
|
|
11
|
+
cpu?: number;
|
|
12
|
+
gpu?: number;
|
|
13
|
+
memory?: number;
|
|
14
|
+
disk?: number;
|
|
15
|
+
createdAt?: string;
|
|
10
16
|
snapshot?: string;
|
|
11
17
|
labels?: Record<string, string>;
|
|
12
18
|
target?: string;
|
|
13
19
|
toolboxProxyUrl?: string;
|
|
14
20
|
errorReason?: string;
|
|
15
21
|
};
|
|
22
|
+
/**
|
|
23
|
+
* Daytona's published list prices, as its pricing page gave them on `readOn`: billed per second, quoted per hour, for
|
|
24
|
+
* each vCPU, each GiB of memory and each GiB of disk. The page's footnote "Price per GiB after first 5 free" is applied
|
|
25
|
+
* to storage (disk) only: its placement is ambiguous and could be read as covering memory too, so this is a choice.
|
|
26
|
+
* The page lists GPU rates per GPU model; this driver does not map a box's GPU to a model, so a GPU box is unpriced.
|
|
27
|
+
*/
|
|
28
|
+
export declare const DAYTONA_LIST_PRICES: {
|
|
29
|
+
readonly page: "https://www.daytona.io/pricing";
|
|
30
|
+
readonly readOn: "2026-10-16";
|
|
31
|
+
readonly vcpuPerHour: 0.0504;
|
|
32
|
+
readonly memoryGiBPerHour: 0.0162;
|
|
33
|
+
readonly diskGiBPerHour: 0.000108;
|
|
34
|
+
readonly freeDiskGiB: 5;
|
|
35
|
+
};
|
|
36
|
+
/**
|
|
37
|
+
* A box's list price per second at DAYTONA_LIST_PRICES, from its size; null when its size is unknown or it has a GPU
|
|
38
|
+
* (priced per GPU model on the page, which this driver does not map).
|
|
39
|
+
*/
|
|
40
|
+
export declare function daytonaUsdPerSecond(size: Pick<SandboxInfo, "cpu" | "gpu" | "memory" | "disk">): number | null;
|
|
16
41
|
export type CreateSandboxBody = {
|
|
17
42
|
name: string;
|
|
18
43
|
snapshot: string;
|
package/dist/hosts/daytona.js
CHANGED
|
@@ -21,6 +21,32 @@ import { TERMINAL_EXITS } from "./local-host.js";
|
|
|
21
21
|
export const DAYTONA_API_URL = "https://app.daytona.io/api";
|
|
22
22
|
export const LABEL_FLEET = "pda-fleet";
|
|
23
23
|
export const LABEL_RUN = "pda-run";
|
|
24
|
+
/**
|
|
25
|
+
* Daytona's published list prices, as its pricing page gave them on `readOn`: billed per second, quoted per hour, for
|
|
26
|
+
* each vCPU, each GiB of memory and each GiB of disk. The page's footnote "Price per GiB after first 5 free" is applied
|
|
27
|
+
* to storage (disk) only: its placement is ambiguous and could be read as covering memory too, so this is a choice.
|
|
28
|
+
* The page lists GPU rates per GPU model; this driver does not map a box's GPU to a model, so a GPU box is unpriced.
|
|
29
|
+
*/
|
|
30
|
+
export const DAYTONA_LIST_PRICES = {
|
|
31
|
+
page: "https://www.daytona.io/pricing",
|
|
32
|
+
readOn: "2026-10-16",
|
|
33
|
+
vcpuPerHour: 0.0504,
|
|
34
|
+
memoryGiBPerHour: 0.0162,
|
|
35
|
+
diskGiBPerHour: 0.000108,
|
|
36
|
+
freeDiskGiB: 5,
|
|
37
|
+
};
|
|
38
|
+
/**
|
|
39
|
+
* A box's list price per second at DAYTONA_LIST_PRICES, from its size; null when its size is unknown or it has a GPU
|
|
40
|
+
* (priced per GPU model on the page, which this driver does not map).
|
|
41
|
+
*/
|
|
42
|
+
export function daytonaUsdPerSecond(size) {
|
|
43
|
+
const { cpu, memory, disk, gpu } = size;
|
|
44
|
+
const count = (n) => typeof n === "number" && Number.isFinite(n) && n >= 0;
|
|
45
|
+
if (!count(cpu) || !count(memory) || !count(disk) || (gpu !== undefined && gpu !== 0))
|
|
46
|
+
return null;
|
|
47
|
+
const p = DAYTONA_LIST_PRICES;
|
|
48
|
+
return (cpu * p.vcpuPerHour + memory * p.memoryGiBPerHour + Math.max(0, disk - p.freeDiskGiB) * p.diskGiBPerHour) / 3600;
|
|
49
|
+
}
|
|
24
50
|
export class DaytonaApiError extends PdaError {
|
|
25
51
|
/** The HTTP status, null when no response arrived (network failure or timeout). */
|
|
26
52
|
status;
|
|
@@ -256,6 +282,13 @@ export function daytonaHost(opts) {
|
|
|
256
282
|
throw new DaytonaHostError("START_FAILED", `${what} in ${box.name} exited ${r.exitCode}: ${r.result.trim().split("\n").at(-1)?.slice(0, 300) ?? ""}`);
|
|
257
283
|
return r;
|
|
258
284
|
}
|
|
285
|
+
/** Each box this driver made, for its receipt: what Daytona said of it, when it was made, and when it ended. */
|
|
286
|
+
const made = new Map();
|
|
287
|
+
const ended = (id) => {
|
|
288
|
+
const m = made.get(id);
|
|
289
|
+
if (m && m.to === null)
|
|
290
|
+
m.to = new Date(now()).toISOString();
|
|
291
|
+
};
|
|
259
292
|
return {
|
|
260
293
|
fleet,
|
|
261
294
|
mountRoot,
|
|
@@ -279,8 +312,10 @@ export function daytonaHost(opts) {
|
|
|
279
312
|
...(opts.network?.allowList ? { networkAllowList: opts.network.allowList } : {}),
|
|
280
313
|
};
|
|
281
314
|
let box = await createOrAdopt(body, ref.id);
|
|
315
|
+
made.set(box.id, { info: box, from: box.createdAt ?? new Date(t0).toISOString(), to: null });
|
|
282
316
|
try {
|
|
283
317
|
box = await waitStarted(box, t0 + startTimeoutMs);
|
|
318
|
+
made.get(box.id).info = { ...made.get(box.id).info, ...box };
|
|
284
319
|
const prepared = now();
|
|
285
320
|
if (opts.prepare)
|
|
286
321
|
await opts.prepare(box, client);
|
|
@@ -308,7 +343,9 @@ export function daytonaHost(opts) {
|
|
|
308
343
|
return handle;
|
|
309
344
|
}
|
|
310
345
|
catch (err) {
|
|
346
|
+
// The box was made and is deleted now: its receipt ends here, deleted or not (a delete that failed is the sweep's).
|
|
311
347
|
await client.remove(box.id).catch(() => { });
|
|
348
|
+
ended(box.id);
|
|
312
349
|
if (err instanceof DaytonaHostError)
|
|
313
350
|
throw err;
|
|
314
351
|
throw new DaytonaHostError("START_FAILED", `starting an instance in ${name} failed: ${err.message}`, { cause: err });
|
|
@@ -324,8 +361,10 @@ export function daytonaHost(opts) {
|
|
|
324
361
|
catch (err) {
|
|
325
362
|
throw new DaytonaHostError("DAYTONA_API_FAILED", `reading sandbox ${h.name}: ${err.message}`, { cause: err });
|
|
326
363
|
}
|
|
327
|
-
if (!box)
|
|
364
|
+
if (!box) {
|
|
365
|
+
ended(h.sandboxId);
|
|
328
366
|
return "gone";
|
|
367
|
+
}
|
|
329
368
|
// The label, not the id in a handle, decides whose box this is: a handle can never reach another fleet's box.
|
|
330
369
|
if (box.labels?.[LABEL_FLEET] !== fleet)
|
|
331
370
|
return "unknown";
|
|
@@ -366,6 +405,7 @@ export function daytonaHost(opts) {
|
|
|
366
405
|
throw new DaytonaHostError("STOP_FAILED", `deleting sandbox ${h.name} failed: ${err.message}`, { cause: first });
|
|
367
406
|
}
|
|
368
407
|
}
|
|
408
|
+
ended(id);
|
|
369
409
|
},
|
|
370
410
|
async kill(h) {
|
|
371
411
|
if (!mine(h))
|
|
@@ -394,6 +434,24 @@ export function daytonaHost(opts) {
|
|
|
394
434
|
throw new DaytonaHostError("KILL_FAILED", `deleting sandbox ${h.name} failed: ${err.message}`, { cause: first });
|
|
395
435
|
}
|
|
396
436
|
}
|
|
437
|
+
ended(id);
|
|
438
|
+
},
|
|
439
|
+
box(h) {
|
|
440
|
+
if (!mine(h))
|
|
441
|
+
return null;
|
|
442
|
+
const m = made.get(h.sandboxId);
|
|
443
|
+
if (!m)
|
|
444
|
+
return null;
|
|
445
|
+
const { cpu, gpu, memory, disk } = m.info;
|
|
446
|
+
const usdPerSecond = daytonaUsdPerSecond(m.info);
|
|
447
|
+
const size = [cpu, memory, disk].every((n) => typeof n === "number") ? ` ${cpu} vCPU, ${memory} GiB memory, ${disk} GiB disk${gpu ? `, ${gpu} GPU` : ""}` : "";
|
|
448
|
+
return {
|
|
449
|
+
provider: "daytona",
|
|
450
|
+
kind: `sandbox${size}`,
|
|
451
|
+
from: m.from,
|
|
452
|
+
to: m.to,
|
|
453
|
+
...(usdPerSecond === null ? {} : { usdPerSecond, source: DAYTONA_LIST_PRICES.page, priced: DAYTONA_LIST_PRICES.readOn }),
|
|
454
|
+
};
|
|
397
455
|
},
|
|
398
456
|
};
|
|
399
457
|
}
|
package/dist/index.d.ts
CHANGED
|
@@ -18,17 +18,17 @@ export type { ArchilStore, Fence, FencedDatabase, Profile, StoreFencedError } fr
|
|
|
18
18
|
export { AppError } from "./app.js";
|
|
19
19
|
export type { AppContext, AppModule, AppOptions } from "./app.js";
|
|
20
20
|
export { archilControl, deleteRunTree, ensureRunning, readRunStatus, SuperviseError } from "./supervise.js";
|
|
21
|
-
export type { ArchilControl, CheckControl, CheckOptions, DiskClient, EnsureOptions, EnsureResult, HostDriver, HostHandle, HostStatus, Json, StartAttempt, StartReason, StonithOutcome, SuperviseErrorCode, SupervisorControl, TokenUser } from "./supervise.js";
|
|
21
|
+
export type { ArchilControl, CheckControl, CheckOptions, DiskClient, EnsureOptions, EnsureResult, HostBox, HostDriver, HostHandle, HostStatus, Json, StartAttempt, StartReason, StonithOutcome, SuperviseErrorCode, SupervisorControl, TokenUser } from "./supervise.js";
|
|
22
22
|
export { localHost, LocalHostError } from "./hosts/local-host.js";
|
|
23
23
|
export type { LocalHostOptions, Ran, Runner } from "./hosts/local-host.js";
|
|
24
24
|
export { dockerHost, DockerHostError } from "./hosts/docker.js";
|
|
25
25
|
export type { ContainerInfo, DockerHostOptions, DockerRan, DockerRunner } from "./hosts/docker.js";
|
|
26
|
-
export { DaytonaApiError, daytonaHost, DaytonaHostError, daytonaRest } from "./hosts/daytona.js";
|
|
26
|
+
export { DAYTONA_LIST_PRICES, DaytonaApiError, daytonaHost, DaytonaHostError, daytonaRest, daytonaUsdPerSecond } from "./hosts/daytona.js";
|
|
27
27
|
export type { CreateSandboxBody, DaytonaClient, DaytonaHostErrorCode, DaytonaHostOptions, DaytonaRestOptions, ExecResult, SandboxInfo } from "./hosts/daytona.js";
|
|
28
28
|
export { requestRun, ServeConfigError, ServeError, serveRun } from "./serve.js";
|
|
29
29
|
export type { RequestOptions, RunResponse, RunServer, ServeConfigErrorCode, ServeErrorCode, ServeOptions } from "./serve.js";
|
|
30
30
|
export { archilPlace, localPlace, PlaceError } from "./place.js";
|
|
31
|
-
export type { ArchilPlaceOptions, LocalPlaceOptions, Place, PlaceContext, PlaceErrorCode, PlaceExit, PlaceKill, PlaceOutcome, PlacePaths, PlaceRecovery, PlaceRunner, PlaceSignal, PlaceStatus, RunnerOrder } from "./place.js";
|
|
31
|
+
export type { ArchilPlaceOptions, LocalPlaceOptions, Place, PlaceContext, PlaceErrorCode, PlaceExit, PlaceKill, PlaceOutcome, PlacePaths, PlaceRecordHome, PlaceRecovery, PlaceRunner, PlaceSignal, PlaceStatus, RecordControl, RunnerOrder } from "./place.js";
|
|
32
32
|
export { busyState, drain, leaseParkTarget, watchParking } from "./park.js";
|
|
33
33
|
export type { BusyState, DrainOptions, Parking, ParkOptions, ParkResult, ParkTarget } from "./park.js";
|
|
34
34
|
export { fork, ForkError, forkLocal } from "./fork.js";
|
package/dist/index.js
CHANGED
|
@@ -17,7 +17,7 @@ export { AppError } from "./app.js";
|
|
|
17
17
|
export { archilControl, deleteRunTree, ensureRunning, readRunStatus, SuperviseError } from "./supervise.js";
|
|
18
18
|
export { localHost, LocalHostError } from "./hosts/local-host.js";
|
|
19
19
|
export { dockerHost, DockerHostError } from "./hosts/docker.js";
|
|
20
|
-
export { DaytonaApiError, daytonaHost, DaytonaHostError, daytonaRest } from "./hosts/daytona.js";
|
|
20
|
+
export { DAYTONA_LIST_PRICES, DaytonaApiError, daytonaHost, DaytonaHostError, daytonaRest, daytonaUsdPerSecond } from "./hosts/daytona.js";
|
|
21
21
|
// Serve: one HTTP request runs one turn.
|
|
22
22
|
export { requestRun, ServeConfigError, ServeError, serveRun } from "./serve.js";
|
|
23
23
|
// Places: where a run lives for a supervisor that kills it and brings it back (agentrun's reference place seam).
|
package/dist/place.d.ts
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
import type { Delegation } from "disk";
|
|
2
2
|
import { PdaError } from "./errors.js";
|
|
3
3
|
import { type RunRecord } from "./status.js";
|
|
4
|
-
import { type EnsureOptions, type HostDriver, type SupervisorControl } from "./supervise.js";
|
|
4
|
+
import { type EnsureOptions, type HostBox, type HostDriver, type SupervisorControl } from "./supervise.js";
|
|
5
5
|
export type PlaceKill = "process" | "box";
|
|
6
6
|
/** The run's paths in the place: its id, its directory, and the `work` and `store` under it. */
|
|
7
7
|
export type PlacePaths = {
|
|
@@ -12,7 +12,8 @@ export type PlacePaths = {
|
|
|
12
12
|
};
|
|
13
13
|
/**
|
|
14
14
|
* How the run stood when a runner ended: the instance's own report, or `run.json` as it left it. `status` is the run's
|
|
15
|
-
* word: `complete` when it delivered, else `failed`.
|
|
15
|
+
* word: `complete` when it delivered, else `failed`. `models`, on any status: the role-to-model ids the run's live parts
|
|
16
|
+
* chose (`lead`, `eyes`, ...), when the app reported a plain object of strings; absent otherwise, never `{}` in its place.
|
|
16
17
|
*/
|
|
17
18
|
export type PlaceOutcome = {
|
|
18
19
|
status: string;
|
|
@@ -21,6 +22,7 @@ export type PlaceOutcome = {
|
|
|
21
22
|
message?: string;
|
|
22
23
|
record?: unknown;
|
|
23
24
|
storeSha256?: string;
|
|
25
|
+
models?: Record<string, string>;
|
|
24
26
|
};
|
|
25
27
|
/** How a runner ended. `exit`: the exit code or signal (local), or the host's status (`stopped`, `failed`, `gone`) or `SIGKILL` when the place killed it (archil). */
|
|
26
28
|
export type PlaceExit = {
|
|
@@ -42,9 +44,15 @@ export type RunnerOrder = {
|
|
|
42
44
|
stdin?: string;
|
|
43
45
|
};
|
|
44
46
|
/** What the place reads of the run, the way it reads it: `run.json`, and on a disk the run's delegations. */
|
|
47
|
+
/**
|
|
48
|
+
* What the place reads of the run, the way it reads it: `run.json`; on a disk, the run's delegations; and the boxes the
|
|
49
|
+
* place made for its runners, each as a receipt (`HostBox`: provider, kind, from, to, list price per second and where
|
|
50
|
+
* that price was read), when its host driver makes boxes. A box the place released stays listed with its end.
|
|
51
|
+
*/
|
|
45
52
|
export type PlaceStatus = {
|
|
46
53
|
record: RunRecord | null;
|
|
47
54
|
delegations?: Pick<Delegation, "clientId" | "inodeId" | "path" | "isOrphaned" | "isPending">[];
|
|
55
|
+
boxes?: HostBox[];
|
|
48
56
|
};
|
|
49
57
|
/** What a recovery did. A takeover on a disk lists what it revoked, and `ms` is the whole of it (revoke and start). */
|
|
50
58
|
export type PlaceRecovery = {
|
|
@@ -91,6 +99,23 @@ export type PlaceContext = {
|
|
|
91
99
|
destroyBoxes?: () => void | Promise<void>;
|
|
92
100
|
requestFile?: (attempt: number) => string;
|
|
93
101
|
};
|
|
102
|
+
/**
|
|
103
|
+
* Where an attempt's runner records (its RECORD_DIR, in the place) and how the place brings that recording home. For a
|
|
104
|
+
* place whose runners cannot write the supervisor's recording directory.
|
|
105
|
+
*/
|
|
106
|
+
export interface PlaceRecordHome {
|
|
107
|
+
/** The directory, in the place, attempt `attempt`'s runner records into. */
|
|
108
|
+
dir(attempt: number): string;
|
|
109
|
+
/**
|
|
110
|
+
* Brings home into `into` what the runners recorded: blobs first, then the new whole lines of each `.spool/*.jsonl`.
|
|
111
|
+
* Idempotent; `final` is the call after the runner ended, when nothing is still landing.
|
|
112
|
+
*/
|
|
113
|
+
bringHome(into: string, options?: {
|
|
114
|
+
final?: boolean;
|
|
115
|
+
}): Promise<void>;
|
|
116
|
+
/** How far the place's clock is from the supervisor's, when it knows. */
|
|
117
|
+
readonly clockSkewMs?: number;
|
|
118
|
+
}
|
|
94
119
|
export interface Place {
|
|
95
120
|
/** `local` or `archil`, for the recording's lines. */
|
|
96
121
|
readonly name: string;
|
|
@@ -122,8 +147,10 @@ export interface Place {
|
|
|
122
147
|
move?(to: Place): Promise<Record<string, unknown>>;
|
|
123
148
|
/** Lets the run go: stops what still runs (its commands too) and removes what the place made for it (tokens), never the run's directory. */
|
|
124
149
|
release(): Promise<void>;
|
|
150
|
+
/** A place whose runners record where the supervisor cannot read says where, and brings the recording home. */
|
|
151
|
+
readonly recordHome?: PlaceRecordHome;
|
|
125
152
|
}
|
|
126
|
-
export type PlaceErrorCode = "INVALID_ARGUMENT" | "HELD" | "START_TIMED_OUT" | "STATUS_FAILED" | "KILL_UNSUPPORTED" | "RELEASE_FAILED";
|
|
153
|
+
export type PlaceErrorCode = "INVALID_ARGUMENT" | "HELD" | "START_TIMED_OUT" | "STATUS_FAILED" | "KILL_UNSUPPORTED" | "RELEASE_FAILED" | "BOX_GONE" | "CARRY_FAILED" | "BLOB_MISMATCH" | "DIRS_REFUSED";
|
|
127
154
|
export declare class PlaceError extends PdaError {
|
|
128
155
|
constructor(code: PlaceErrorCode, message: string, options?: {
|
|
129
156
|
cause?: unknown;
|
|
@@ -150,8 +177,11 @@ export declare function localPlace(options: LocalPlaceOptions): (ctx: PlaceConte
|
|
|
150
177
|
export interface ArchilPlaceOptions {
|
|
151
178
|
disk: string;
|
|
152
179
|
region: string;
|
|
153
|
-
/**
|
|
154
|
-
|
|
180
|
+
/**
|
|
181
|
+
* The disk's control API (`archilControl`). The place holds it; no instance ever does. With `listObjects` (as
|
|
182
|
+
* `archilControl` has), the place has `recordHome`.
|
|
183
|
+
*/
|
|
184
|
+
control: SupervisorControl & Partial<RecordControl>;
|
|
155
185
|
/**
|
|
156
186
|
* The driver that runs the instance (`localHost`, `dockerHost`, `daytonaHost`), set up with the app the instance runs.
|
|
157
187
|
* Its `kill` and `kills` are the place's.
|
|
@@ -175,6 +205,22 @@ export interface ArchilPlaceOptions {
|
|
|
175
205
|
*/
|
|
176
206
|
signalFailMs?: number;
|
|
177
207
|
}
|
|
208
|
+
/** The listing `recordHome` reads the run's recordings with: S3 ListObjects, each object's size when the API gives it. */
|
|
209
|
+
export interface RecordControl {
|
|
210
|
+
listObjects(prefix: string, options?: {
|
|
211
|
+
recursive?: boolean;
|
|
212
|
+
}): Promise<{
|
|
213
|
+
objects: {
|
|
214
|
+
key: string;
|
|
215
|
+
size?: number;
|
|
216
|
+
}[];
|
|
217
|
+
}>;
|
|
218
|
+
}
|
|
219
|
+
/**
|
|
220
|
+
* Every byte of `bytes` to `fd`: a write may take fewer bytes than it was given, so it goes on from where the last one
|
|
221
|
+
* stopped. A write that takes none fails, as one that throws (a full disk) does.
|
|
222
|
+
*/
|
|
223
|
+
export declare function writeAll(fd: number, bytes: Uint8Array, write?: (fd: number, bytes: Uint8Array, offset: number, length: number) => number): void;
|
|
178
224
|
/**
|
|
179
225
|
* The run on an Archil disk, its instance run by a host driver. `token(attempt)` mints the attempt's mount token through
|
|
180
226
|
* the place's control, and the runner it is handed to mounts with it. A start is `ensureRunning` (it creates the run's
|
package/dist/place.js
CHANGED
|
@@ -5,8 +5,9 @@
|
|
|
5
5
|
// place seam (examples/reference/place.ts) structurally. The supervisor reads nothing under the run's directory: the
|
|
6
6
|
// outcome, the run's record and its delegations come through the place.
|
|
7
7
|
import { spawn } from "node:child_process";
|
|
8
|
-
import {
|
|
9
|
-
import {
|
|
8
|
+
import { createHash, randomUUID } from "node:crypto";
|
|
9
|
+
import { closeSync, existsSync, fsyncSync, mkdirSync, openSync, readdirSync, readFileSync, renameSync, rmSync, statSync, writeSync } from "node:fs";
|
|
10
|
+
import { dirname, extname, join, posix, resolve } from "node:path";
|
|
10
11
|
import { fileURLToPath } from "node:url";
|
|
11
12
|
import { DEFAULT_MOUNT_ROOT, findDelegations, mintMountToken, pathlessResolver, removeMountToken, runPath } from "./claim.js";
|
|
12
13
|
import { reapCommands } from "./commands.js";
|
|
@@ -48,6 +49,7 @@ function outcomeOf(record) {
|
|
|
48
49
|
return undefined;
|
|
49
50
|
const detail = record.detail !== null && typeof record.detail === "object" && !Array.isArray(record.detail) ? record.detail : {};
|
|
50
51
|
const delivered = record.status === "done" && "record" in detail && detail.record !== null && detail.record !== undefined;
|
|
52
|
+
const models = modelsOf(detail.models);
|
|
51
53
|
return {
|
|
52
54
|
status: delivered ? "complete" : "failed",
|
|
53
55
|
generation: record.generation,
|
|
@@ -55,8 +57,22 @@ function outcomeOf(record) {
|
|
|
55
57
|
...(typeof detail.message === "string" ? { message: detail.message } : {}),
|
|
56
58
|
...("record" in detail ? { record: detail.record } : {}),
|
|
57
59
|
...(typeof detail.storeSha256 === "string" ? { storeSha256: detail.storeSha256 } : {}),
|
|
60
|
+
...(models ? { models } : {}),
|
|
58
61
|
};
|
|
59
62
|
}
|
|
63
|
+
/**
|
|
64
|
+
* The role-to-model ids an outcome carries: a copy of `value` when it is a plain object whose values are all strings,
|
|
65
|
+
* else undefined, so a malformed report is left out and never stands in as `{}`.
|
|
66
|
+
*/
|
|
67
|
+
function modelsOf(value) {
|
|
68
|
+
if (value === null || typeof value !== "object" || Array.isArray(value))
|
|
69
|
+
return undefined;
|
|
70
|
+
const proto = Object.getPrototypeOf(value);
|
|
71
|
+
if (proto !== Object.prototype && proto !== null)
|
|
72
|
+
return undefined;
|
|
73
|
+
const entries = Object.entries(value);
|
|
74
|
+
return entries.every(([, id]) => typeof id === "string") ? Object.fromEntries(entries) : undefined;
|
|
75
|
+
}
|
|
60
76
|
const pathsUnder = (id, root) => ({ id, root, work: join(root, "work"), store: join(root, "store") });
|
|
61
77
|
/**
|
|
62
78
|
* The run on a directory of this machine: each runner is `pi-durable-disk run --dir <root> --id <id> --app <app>`, a
|
|
@@ -281,13 +297,20 @@ function requestIn(text) {
|
|
|
281
297
|
return undefined;
|
|
282
298
|
}
|
|
283
299
|
}
|
|
284
|
-
/**
|
|
300
|
+
/**
|
|
301
|
+
* The `outcome` of a `{"outcome":{...}}` line with at least a status; undefined for any other line. Its `models` stays
|
|
302
|
+
* only when it is a plain object of strings (`modelsOf`), as `run.json`'s does.
|
|
303
|
+
*/
|
|
285
304
|
function outcomeLine(line) {
|
|
286
305
|
if (!line.startsWith("{"))
|
|
287
306
|
return undefined;
|
|
288
307
|
try {
|
|
289
308
|
const parsed = JSON.parse(line);
|
|
290
|
-
|
|
309
|
+
if (!parsed.outcome || typeof parsed.outcome !== "object" || typeof parsed.outcome.status !== "string")
|
|
310
|
+
return undefined;
|
|
311
|
+
const { models: reported, ...outcome } = parsed.outcome;
|
|
312
|
+
const models = modelsOf(reported);
|
|
313
|
+
return models ? { ...outcome, models } : outcome;
|
|
291
314
|
}
|
|
292
315
|
catch {
|
|
293
316
|
return undefined;
|
|
@@ -295,11 +318,263 @@ function outcomeLine(line) {
|
|
|
295
318
|
}
|
|
296
319
|
/** Where in the run's directory an Archil place's runners write their kill requests. */
|
|
297
320
|
const REQUESTS = "signals";
|
|
298
|
-
|
|
321
|
+
/** Where in the run's directory an Archil place's runners record, one directory per attempt. */
|
|
322
|
+
const RECORDS = "record";
|
|
323
|
+
const BLOB_NAME = /^([0-9a-f]{64})\.[A-Za-z0-9]{1,16}$/;
|
|
324
|
+
const SPOOL_NAME = /^[A-Za-z0-9][A-Za-z0-9._-]{0,200}\.jsonl$/;
|
|
325
|
+
const NEWLINE = 0x0a;
|
|
326
|
+
/** A message for a recording's line: at most 240 characters, so it never carries more than the line can hold. */
|
|
327
|
+
const said = (text) => text.slice(0, MESSAGE_KEEP);
|
|
328
|
+
/**
|
|
329
|
+
* Every byte of `bytes` to `fd`: a write may take fewer bytes than it was given, so it goes on from where the last one
|
|
330
|
+
* stopped. A write that takes none fails, as one that throws (a full disk) does.
|
|
331
|
+
*/
|
|
332
|
+
export function writeAll(fd, bytes, write = writeSync) {
|
|
333
|
+
for (let at = 0; at < bytes.length;) {
|
|
334
|
+
const n = write(fd, bytes, at, bytes.length - at);
|
|
335
|
+
if (!(n > 0))
|
|
336
|
+
throw new PlaceError("CARRY_FAILED", `a write took no bytes, ${bytes.length - at} of ${bytes.length} left`);
|
|
337
|
+
at += n;
|
|
338
|
+
}
|
|
339
|
+
}
|
|
340
|
+
/** `bytes` into `file` whole: a temporary file, fsync, rename, then the directory's fsync. */
|
|
341
|
+
function writeWhole(file, bytes) {
|
|
342
|
+
const dir = dirname(file);
|
|
343
|
+
mkdirSync(dir, { recursive: true });
|
|
344
|
+
const tmp = join(dir, `.${randomUUID()}.tmp`);
|
|
345
|
+
const fd = openSync(tmp, "wx", 0o644);
|
|
346
|
+
try {
|
|
347
|
+
writeAll(fd, bytes);
|
|
348
|
+
fsyncSync(fd);
|
|
349
|
+
}
|
|
350
|
+
catch (error) {
|
|
351
|
+
closeSync(fd);
|
|
352
|
+
rmSync(tmp, { force: true });
|
|
353
|
+
throw error;
|
|
354
|
+
}
|
|
355
|
+
closeSync(fd);
|
|
356
|
+
renameSync(tmp, file);
|
|
357
|
+
syncDir(dir);
|
|
358
|
+
}
|
|
359
|
+
/** `bytes` at the end of `file`, fsynced; a file it creates is made durable in its directory too. */
|
|
360
|
+
function appendDurably(file, bytes) {
|
|
361
|
+
const dir = dirname(file);
|
|
362
|
+
const created = !existsSync(file);
|
|
363
|
+
mkdirSync(dir, { recursive: true });
|
|
364
|
+
const fd = openSync(file, "a", 0o644);
|
|
365
|
+
try {
|
|
366
|
+
writeAll(fd, bytes);
|
|
367
|
+
fsyncSync(fd);
|
|
368
|
+
}
|
|
369
|
+
finally {
|
|
370
|
+
closeSync(fd);
|
|
371
|
+
}
|
|
372
|
+
if (created)
|
|
373
|
+
syncDir(dir);
|
|
374
|
+
}
|
|
375
|
+
/** How long a carrier waits for another one appending to the same destination file. */
|
|
376
|
+
const CARRY_LOCK_WAIT_MS = 5_000;
|
|
377
|
+
/** A lock file with no holder written yet (its taker is between the create and the write) counts as held this long. */
|
|
378
|
+
const CARRY_LOCK_UNNAMED_MS = 10_000;
|
|
379
|
+
const alive = (pid) => {
|
|
380
|
+
try {
|
|
381
|
+
process.kill(pid, 0);
|
|
382
|
+
return true;
|
|
383
|
+
}
|
|
384
|
+
catch (error) {
|
|
385
|
+
return error.code === "EPERM";
|
|
386
|
+
}
|
|
387
|
+
};
|
|
388
|
+
/**
|
|
389
|
+
* `fn` while holding `lock`, a file created exclusively that names its holder's pid, so two carriers on one destination
|
|
390
|
+
* (two places, two processes) never append at once. A lock whose holder is gone (its process ended) is taken over; one a
|
|
391
|
+
* live holder keeps past CARRY_LOCK_WAIT_MS is CARRY_FAILED. `fn` is synchronous, so the lock is held only for it.
|
|
392
|
+
*/
|
|
393
|
+
async function withCarryLock(lock, fn) {
|
|
394
|
+
for (const t0 = Date.now();;) {
|
|
395
|
+
let fd;
|
|
396
|
+
try {
|
|
397
|
+
fd = openSync(lock, "wx", 0o644);
|
|
398
|
+
}
|
|
399
|
+
catch (error) {
|
|
400
|
+
if (error.code !== "EEXIST")
|
|
401
|
+
throw error;
|
|
402
|
+
}
|
|
403
|
+
if (fd !== undefined) {
|
|
404
|
+
// Once created, the lock goes on every way out: a failed write of the pid or close of the file included.
|
|
405
|
+
try {
|
|
406
|
+
try {
|
|
407
|
+
writeAll(fd, new TextEncoder().encode(String(process.pid)));
|
|
408
|
+
}
|
|
409
|
+
finally {
|
|
410
|
+
closeSync(fd);
|
|
411
|
+
}
|
|
412
|
+
fn();
|
|
413
|
+
}
|
|
414
|
+
finally {
|
|
415
|
+
rmSync(lock, { force: true });
|
|
416
|
+
}
|
|
417
|
+
return;
|
|
418
|
+
}
|
|
419
|
+
let holder = 0;
|
|
420
|
+
let ageMs = 0;
|
|
421
|
+
try {
|
|
422
|
+
holder = Number(readFileSync(lock, "utf8").trim());
|
|
423
|
+
ageMs = Date.now() - statSync(lock).mtimeMs;
|
|
424
|
+
}
|
|
425
|
+
catch (error) {
|
|
426
|
+
// Released meanwhile: take it on the next turn. Any other failure to read it (its mode, the disk) is the carry's.
|
|
427
|
+
if (error.code === "ENOENT" && Date.now() - t0 <= CARRY_LOCK_WAIT_MS)
|
|
428
|
+
continue;
|
|
429
|
+
throw new PlaceError("CARRY_FAILED", said(`reading the carry lock ${lock} failed: ${error?.message ?? error}`), { cause: error });
|
|
430
|
+
}
|
|
431
|
+
const gone = Number.isSafeInteger(holder) && holder > 0 ? !alive(holder) : ageMs > CARRY_LOCK_UNNAMED_MS;
|
|
432
|
+
if (gone) {
|
|
433
|
+
rmSync(lock, { force: true });
|
|
434
|
+
continue;
|
|
435
|
+
}
|
|
436
|
+
if (Date.now() - t0 > CARRY_LOCK_WAIT_MS)
|
|
437
|
+
throw new PlaceError("CARRY_FAILED", said(`another carrier (pid ${holder || "unnamed"}) holds ${lock} past ${CARRY_LOCK_WAIT_MS} ms`));
|
|
438
|
+
await new Promise((resolve) => setTimeout(resolve, 20));
|
|
439
|
+
}
|
|
440
|
+
}
|
|
441
|
+
function syncDir(dir) {
|
|
442
|
+
const fd = openSync(dir, "r");
|
|
443
|
+
try {
|
|
444
|
+
fsyncSync(fd);
|
|
445
|
+
}
|
|
446
|
+
finally {
|
|
447
|
+
closeSync(fd);
|
|
448
|
+
}
|
|
449
|
+
}
|
|
450
|
+
/** The blob hashes (SHA-256, hex) a recording's line names in its `blobs`; none for a line that is not JSON or names none. */
|
|
451
|
+
function blobsOf(line) {
|
|
452
|
+
try {
|
|
453
|
+
const v = JSON.parse(new TextDecoder().decode(line));
|
|
454
|
+
return Array.isArray(v?.blobs) ? v.blobs.filter((b) => typeof b === "string") : [];
|
|
455
|
+
}
|
|
456
|
+
catch {
|
|
457
|
+
return [];
|
|
458
|
+
}
|
|
459
|
+
}
|
|
460
|
+
/**
|
|
461
|
+
* Brings home into `into` what the runners recorded under `prefix` (`runs/<id>/record/`), read through the control (S3
|
|
462
|
+
* ListObjects and GetObject: nothing takes the run's claim). It keeps no state: a blob is home when `into/blobs/<name>`
|
|
463
|
+
* exists, and a spool file's carried bytes are what `into/.spool/<name>` holds, so a second call, or a call from another
|
|
464
|
+
* process, carries nothing twice. Carriers that overlap on one destination never append at once: the place runs one call
|
|
465
|
+
* at a time per destination, and every append happens under a lock file beside its spool file (`.<name>.lock`, naming
|
|
466
|
+
* its holder's pid), where the destination's size is read again so only the lines past it are appended. A lock whose
|
|
467
|
+
* holder is gone is taken over; one held past 5 s is CARRY_FAILED. Every write is carried to its last byte.
|
|
468
|
+
* - Blobs first: each one not yet home is fetched, checked against the hash its name carries, and written whole into
|
|
469
|
+
* `into/blobs`. One whose bytes do not match is skipped; everything else is brought home, then the call throws
|
|
470
|
+
* BLOB_MISMATCH, and the next call tries that blob again.
|
|
471
|
+
* - Then lines: the new whole lines of each `.spool/<name>.jsonl`, the recorder's segments (`<stream>.<n>.jsonl`), each
|
|
472
|
+
* with its own offset: what `into/.spool/<name>` already holds. A file is fetched only when it grew since this place
|
|
473
|
+
* last fetched it (the SDK has no ranged read). A line whose blobs are not home yet waits for the next call, so no line arrives before its blob; on
|
|
474
|
+
* `final` it is carried anyway, and its missing blobs are CARRY_FAILED once everything else is home.
|
|
475
|
+
* - A listing or fetch that fails is CARRY_FAILED. The recording is on the run's disk, not in the box, so a lost box
|
|
476
|
+
* costs only what it had not fsynced: this place never throws BOX_GONE.
|
|
477
|
+
*/
|
|
478
|
+
async function carryHome(control, prefix, into, final, fetched) {
|
|
479
|
+
const fetch = async (key) => {
|
|
480
|
+
try {
|
|
481
|
+
return await control.getObject(key);
|
|
482
|
+
}
|
|
483
|
+
catch (error) {
|
|
484
|
+
throw new PlaceError("CARRY_FAILED", said(`reading ${key.slice(prefix.length)} of the run's recording failed: ${error?.message ?? error}`), { cause: error });
|
|
485
|
+
}
|
|
486
|
+
};
|
|
487
|
+
let objects;
|
|
488
|
+
try {
|
|
489
|
+
objects = (await control.listObjects(prefix, { recursive: true })).objects;
|
|
490
|
+
}
|
|
491
|
+
catch (error) {
|
|
492
|
+
throw new PlaceError("CARRY_FAILED", said(`listing the run's recording failed: ${error?.message ?? error}`), { cause: error });
|
|
493
|
+
}
|
|
494
|
+
// `<attempt>/blobs/<name>` and `<attempt>/.spool/<name>.jsonl`; anything else (`started`, a box's temporary file) is not carried.
|
|
495
|
+
const parts = (key) => key.slice(prefix.length).split("/");
|
|
496
|
+
const blobs = objects.filter((o) => {
|
|
497
|
+
const p = parts(o.key);
|
|
498
|
+
return p.length === 3 && p[1] === "blobs" && BLOB_NAME.test(p[2]);
|
|
499
|
+
});
|
|
500
|
+
const spools = objects.filter((o) => {
|
|
501
|
+
const p = parts(o.key);
|
|
502
|
+
return p.length === 3 && p[1] === ".spool" && SPOOL_NAME.test(p[2]);
|
|
503
|
+
});
|
|
504
|
+
const home = (name) => existsSync(join(into, "blobs", name));
|
|
505
|
+
const mismatched = [];
|
|
506
|
+
for (const blob of blobs) {
|
|
507
|
+
const name = parts(blob.key)[2];
|
|
508
|
+
if (home(name))
|
|
509
|
+
continue;
|
|
510
|
+
const bytes = await fetch(blob.key);
|
|
511
|
+
if (createHash("sha256").update(bytes).digest("hex") !== BLOB_NAME.exec(name)[1]) {
|
|
512
|
+
mismatched.push(name);
|
|
513
|
+
continue;
|
|
514
|
+
}
|
|
515
|
+
writeWhole(join(into, "blobs", name), bytes);
|
|
516
|
+
}
|
|
517
|
+
// A line names its blobs by hash; the file is `<hash>.<ext>`.
|
|
518
|
+
const homeHashes = new Set(existsSync(join(into, "blobs")) ? readdirSync(join(into, "blobs")).map((n) => BLOB_NAME.exec(n)?.[1]).filter((h) => h !== undefined) : []);
|
|
519
|
+
const waiting = new Set();
|
|
520
|
+
for (const spool of spools) {
|
|
521
|
+
const name = parts(spool.key)[2];
|
|
522
|
+
const dest = join(into, ".spool", name);
|
|
523
|
+
const carried = existsSync(dest) ? statSync(dest).size : 0;
|
|
524
|
+
// Not fetched again while its size is what this place last fetched and nothing in it waited for a blob (a partial
|
|
525
|
+
// last line keeps it bigger than what is home). Never skipped on `final`. Only a saving: what is home is still what
|
|
526
|
+
// the destination holds.
|
|
527
|
+
// Per destination: what this place fetched for one `into` says nothing of another.
|
|
528
|
+
const cacheKey = `${resolve(into)}\0${spool.key}`;
|
|
529
|
+
if (spool.size !== undefined && (spool.size <= carried || (!final && fetched.get(cacheKey) === spool.size)))
|
|
530
|
+
continue;
|
|
531
|
+
const bytes = await fetch(spool.key);
|
|
532
|
+
if (bytes.length <= carried)
|
|
533
|
+
continue;
|
|
534
|
+
let heldForBlob = false;
|
|
535
|
+
let end = carried;
|
|
536
|
+
for (let at = carried; at < bytes.length;) {
|
|
537
|
+
const nl = bytes.indexOf(NEWLINE, at);
|
|
538
|
+
if (nl < 0)
|
|
539
|
+
break; // a line still being written
|
|
540
|
+
const missing = blobsOf(bytes.subarray(at, nl)).filter((hash) => !homeHashes.has(hash));
|
|
541
|
+
if (missing.length > 0 && !final) {
|
|
542
|
+
heldForBlob = true;
|
|
543
|
+
break;
|
|
544
|
+
}
|
|
545
|
+
for (const b of missing)
|
|
546
|
+
waiting.add(b);
|
|
547
|
+
end = nl + 1;
|
|
548
|
+
at = nl + 1;
|
|
549
|
+
}
|
|
550
|
+
if (end > carried) {
|
|
551
|
+
// Another carrier on this destination (another place, another process) may have appended since `carried` was read.
|
|
552
|
+
// Under the file's lock, what the destination holds now decides: only the lines past it are appended.
|
|
553
|
+
mkdirSync(dirname(dest), { recursive: true });
|
|
554
|
+
await withCarryLock(join(dirname(dest), `.${name}.lock`), () => {
|
|
555
|
+
const now = existsSync(dest) ? statSync(dest).size : 0;
|
|
556
|
+
if (end > now)
|
|
557
|
+
appendDurably(dest, bytes.subarray(now, end));
|
|
558
|
+
});
|
|
559
|
+
}
|
|
560
|
+
if (heldForBlob)
|
|
561
|
+
fetched.delete(cacheKey);
|
|
562
|
+
else
|
|
563
|
+
fetched.set(cacheKey, bytes.length);
|
|
564
|
+
}
|
|
565
|
+
if (mismatched.length > 0) {
|
|
566
|
+
throw new PlaceError("BLOB_MISMATCH", said(`${mismatched.length} blob(s) of the run's recording do not hash to their names: ${mismatched.join(", ")}`));
|
|
567
|
+
}
|
|
568
|
+
if (final && waiting.size > 0) {
|
|
569
|
+
throw new PlaceError("CARRY_FAILED", said(`${waiting.size} blob(s) the run's recording names never reached its disk: ${[...waiting].join(", ")}`));
|
|
570
|
+
}
|
|
571
|
+
}
|
|
572
|
+
const countFrom1 = (attempt) => {
|
|
299
573
|
if (!Number.isSafeInteger(attempt) || attempt < 1)
|
|
300
574
|
throw new PlaceError("INVALID_ARGUMENT", `attempt ${attempt} is not a count from 1`);
|
|
301
|
-
return
|
|
575
|
+
return attempt;
|
|
302
576
|
};
|
|
577
|
+
const requestName = (attempt) => `kill-${countFrom1(attempt)}.json`;
|
|
303
578
|
/**
|
|
304
579
|
* The run on an Archil disk, its instance run by a host driver. `token(attempt)` mints the attempt's mount token through
|
|
305
580
|
* the place's control, and the runner it is handed to mounts with it. A start is `ensureRunning` (it creates the run's
|
|
@@ -330,8 +605,40 @@ export function archilPlace(options) {
|
|
|
330
605
|
const paths = { id: ref.id, root, work: posix.join(root, "work"), store: posix.join(root, "store") };
|
|
331
606
|
const minted = new Map();
|
|
332
607
|
const handles = [];
|
|
608
|
+
/** Every handle the place started, released or not: the boxes it reports. */
|
|
609
|
+
const started = [];
|
|
333
610
|
const killed = new Map();
|
|
334
611
|
let resolver = pathlessResolver(control);
|
|
612
|
+
const records = `${runPath(ref.id)}/${RECORDS}/`;
|
|
613
|
+
const recording = typeof options.control.listObjects === "function";
|
|
614
|
+
const owner = options.ensure?.create;
|
|
615
|
+
/**
|
|
616
|
+
* Before attempt `gen`'s runner starts: the EMPTY directories it writes into, `signals/` and, with `recordHome`,
|
|
617
|
+
* `record/<gen>/.spool/` and `record/<gen>/blobs/`, made over the control (PutObject with the owner, as createRunDir
|
|
618
|
+
* does: an exec mkdir would make them root's) and owned by the run user (`ensure.create`), in ensureRunning's window
|
|
619
|
+
* where no delegation holds the run (a held run refuses them: DIRS_REFUSED). A directory a process
|
|
620
|
+
* made by path on an Archil mount was seen to refuse file creates afterwards (ENOENT), and one made over the control
|
|
621
|
+
* does not. Every file in them stays the runner's (its kill request, its started, its spool), and a runner that
|
|
622
|
+
* makes them itself finds them made. Without an owner nothing is made: a directory the run user does not own would
|
|
623
|
+
* refuse the runner's files.
|
|
624
|
+
*/
|
|
625
|
+
const makeDirs = async (gen) => {
|
|
626
|
+
if (!owner)
|
|
627
|
+
return;
|
|
628
|
+
const run = `${runPath(ref.id)}/`;
|
|
629
|
+
const dirs = [`${run}${REQUESTS}/`, ...(recording ? [`${run}${RECORDS}/`, `${run}${RECORDS}/${gen}/`, `${run}${RECORDS}/${gen}/.spool/`, `${run}${RECORDS}/${gen}/blobs/`] : [])];
|
|
630
|
+
for (const dir of dirs) {
|
|
631
|
+
try {
|
|
632
|
+
if (!(await control.headObject(dir)))
|
|
633
|
+
await control.putObject(dir, "", { uid: owner.uid, gid: owner.gid, mode: owner.mode ?? 0o755 });
|
|
634
|
+
}
|
|
635
|
+
catch (error) {
|
|
636
|
+
throw new PlaceError("DIRS_REFUSED", said(`making ${dir.slice(run.length)} in the run over the control was refused: ${error?.message ?? error}`), { cause: error });
|
|
637
|
+
}
|
|
638
|
+
}
|
|
639
|
+
};
|
|
640
|
+
const fetchedSizes = new Map();
|
|
641
|
+
const carrying = new Map();
|
|
335
642
|
const handleOf = new Map();
|
|
336
643
|
const mint = async (attempt) => {
|
|
337
644
|
const t = await mintMountToken(control, { run: { id: ref.id, attempt }, prefix: options.ensure?.tokenPrefix, ttl: options.ensure?.tokenTtl ?? TOKEN_TTL });
|
|
@@ -385,7 +692,11 @@ export function archilPlace(options) {
|
|
|
385
692
|
const m = given && !given.removed ? given : await mint(order.gen);
|
|
386
693
|
// The driver gets the order's environment with each start; everything else is the host's own.
|
|
387
694
|
const driver = {
|
|
388
|
-
|
|
695
|
+
// Inside ensureRunning, after its revoke and start mark: no delegation holds the run, so S3 takes the writes.
|
|
696
|
+
start: async (r, token, attempt) => {
|
|
697
|
+
await makeDirs(order.gen);
|
|
698
|
+
return host.start(r, token, { attempt: attempt?.attempt ?? order.gen, env: order.env });
|
|
699
|
+
},
|
|
389
700
|
status: (h) => host.status(h),
|
|
390
701
|
stop: (h) => host.stop(h),
|
|
391
702
|
};
|
|
@@ -414,6 +725,7 @@ export function archilPlace(options) {
|
|
|
414
725
|
if (result.adopted)
|
|
415
726
|
await drop(m).catch(() => { });
|
|
416
727
|
handles.push(result.handle);
|
|
728
|
+
started.push(result.handle);
|
|
417
729
|
const w = watch(result.handle);
|
|
418
730
|
const mark = result.startMark ? await readStartMark(control, ref.id).catch(() => null) : null;
|
|
419
731
|
const runner = { epoch: mark?.generation ?? order.gen, exited: w.exited };
|
|
@@ -460,6 +772,26 @@ export function archilPlace(options) {
|
|
|
460
772
|
paths,
|
|
461
773
|
kills: host.kill && host.kills ? [host.kills] : [],
|
|
462
774
|
token: async (attempt) => (await mint(attempt)).token,
|
|
775
|
+
...(recording
|
|
776
|
+
? {
|
|
777
|
+
recordHome: {
|
|
778
|
+
// In the run's directory, so the runner's delegation covers it and deleteRunTree removes it.
|
|
779
|
+
dir: (attempt) => posix.join(root, RECORDS, String(countFrom1(attempt))),
|
|
780
|
+
// One carry at a time per destination in this place; another place or process is held off by the file lock.
|
|
781
|
+
bringHome: (into, carry = {}) => {
|
|
782
|
+
const key = resolve(into);
|
|
783
|
+
const run = (carrying.get(key) ?? Promise.resolve()).catch(() => { }).then(() => carryHome(control, records, into, carry.final === true, fetchedSizes));
|
|
784
|
+
const settled = run.then(() => { }, () => { });
|
|
785
|
+
carrying.set(key, settled);
|
|
786
|
+
void settled.then(() => {
|
|
787
|
+
if (carrying.get(key) === settled)
|
|
788
|
+
carrying.delete(key);
|
|
789
|
+
});
|
|
790
|
+
return run;
|
|
791
|
+
},
|
|
792
|
+
},
|
|
793
|
+
}
|
|
794
|
+
: {}),
|
|
463
795
|
// The request goes in the run's directory, outside work/ where the agent's tools are confined, and the place reads
|
|
464
796
|
// it over S3: a runner on a box of its own cannot reach the supervisor's files.
|
|
465
797
|
requestFile: (attempt) => posix.join(root, REQUESTS, requestName(attempt)),
|
|
@@ -491,7 +823,8 @@ export function archilPlace(options) {
|
|
|
491
823
|
throw error;
|
|
492
824
|
}),
|
|
493
825
|
]);
|
|
494
|
-
|
|
826
|
+
const boxes = host.box ? started.map((h) => host.box(h)).filter((b) => b !== null) : undefined;
|
|
827
|
+
return { record, delegations: delegations.map(({ clientId, inodeId, path, isOrphaned, isPending }) => ({ clientId, inodeId, path, isOrphaned, isPending })), ...(boxes ? { boxes } : {}) };
|
|
495
828
|
},
|
|
496
829
|
recover: async (last, kind, order) => {
|
|
497
830
|
if (killed.has(last))
|
package/dist/run.js
CHANGED
|
@@ -15,7 +15,7 @@ import { archilEnv } from "./env.js";
|
|
|
15
15
|
import { EXIT_FENCED, FencedError, HeldError, ownWriteFenced, PdaError } from "./errors.js";
|
|
16
16
|
import { openArchilStore } from "./store.js";
|
|
17
17
|
import { LeaseWatchdog } from "./watchdog.js";
|
|
18
|
-
import { currentHolder, DEFAULT_HEARTBEAT_MS, DEFAULT_LEASE_EXPIRY_MS, DEFAULT_LEASE_MARGIN_MS, openClaimDir, persistRecord, readRunRecord, RUN_JSON, RunRecordError, writeRunRecord, } from "./status.js";
|
|
18
|
+
import { currentHolder, DEFAULT_HEARTBEAT_MS, DEFAULT_LEASE_EXPIRY_MS, DEFAULT_LEASE_MARGIN_MS, keepStartMark, openClaimDir, persistRecord, readRunRecord, RUN_JSON, RunRecordError, writeRunRecord, } from "./status.js";
|
|
19
19
|
/** The same-host owner lock under the run's root. */
|
|
20
20
|
export const OWNER_LOCK = "owner.lock";
|
|
21
21
|
/** pi's store under the claim's `store/` directory. */
|
|
@@ -358,7 +358,16 @@ class Lease {
|
|
|
358
358
|
if (seed !== undefined && previous === undefined) {
|
|
359
359
|
await step("seed", async () => {
|
|
360
360
|
await this.#dir.assertMounted();
|
|
361
|
-
|
|
361
|
+
// While a new run seeds it has no `run.json` heartbeat, and a supervisor keeps a start in its grace only while its
|
|
362
|
+
// start mark is younger than the grace: the mark is kept fresh until the seed ends, through the held claim directory.
|
|
363
|
+
// A seeder that dies stops, and the grace runs out as for any start.
|
|
364
|
+
const stop = keepStartMark(this.#dir.at, this.#lease.heartbeatMs);
|
|
365
|
+
try {
|
|
366
|
+
await seed({ root: claim.root, storePath: join(claim.store, STORE_FILE), workDir: claim.work });
|
|
367
|
+
}
|
|
368
|
+
finally {
|
|
369
|
+
await stop();
|
|
370
|
+
}
|
|
362
371
|
await this.#dir.assertMounted();
|
|
363
372
|
await claim.barrier();
|
|
364
373
|
});
|
package/dist/status.d.ts
CHANGED
|
@@ -81,6 +81,27 @@ export declare const persistRecord: PersistRecord;
|
|
|
81
81
|
* `PersistRecord`.
|
|
82
82
|
*/
|
|
83
83
|
export declare function writeRunRecord(root: string | ClaimDir, record: RunRecord, persist?: PersistRecord, signal?: AbortSignal): Promise<void>;
|
|
84
|
+
/**
|
|
85
|
+
* The supervisor's record of its last start of a run, in the run's directory: the generation the instance will write,
|
|
86
|
+
* when, by whom, and the failed starts right before it (`failures`; `lastExit`, what the decision saw of the last one).
|
|
87
|
+
*/
|
|
88
|
+
export declare const START_MARK = "start.json";
|
|
89
|
+
export type StartMark = {
|
|
90
|
+
generation: number;
|
|
91
|
+
at: string;
|
|
92
|
+
by: string | null;
|
|
93
|
+
failures: number;
|
|
94
|
+
lastExit: string | null;
|
|
95
|
+
};
|
|
96
|
+
/** A start mark's text as a mark, or null when it is not one (an unreadable mark gives no grace). */
|
|
97
|
+
export declare function parseStartMark(text: string): StartMark | null;
|
|
98
|
+
/**
|
|
99
|
+
* Keeps a supervisor's start mark for generation 1 in `dir` fresh: its `at` rewritten every `everyMs` (temp file, fsync,
|
|
100
|
+
* rename, fsync the directory), the rest of it as the supervisor wrote it; nothing is made when there is none. On a thread of
|
|
101
|
+
* its own, so a caller whose thread is blocked (a synchronous import) is kept fresh too. Best effort: a refused write leaves
|
|
102
|
+
* the mark as it was. Returns the stop, which resolves once the thread has exited, so no write outlives it.
|
|
103
|
+
*/
|
|
104
|
+
export declare function keepStartMark(dir: string, everyMs: number): () => Promise<void>;
|
|
84
105
|
/** The FencedError code of a claim root that is no longer the mount the instance opened (unmounted, lazily or not). */
|
|
85
106
|
export declare const CLAIM_UNMOUNTED = "CLAIM_UNMOUNTED";
|
|
86
107
|
/** The file system type of an Archil mount in the mount table. */
|
package/dist/status.js
CHANGED
|
@@ -6,6 +6,7 @@ import { constants } from "node:fs";
|
|
|
6
6
|
import { open, readFile, rename, rm, stat } from "node:fs/promises";
|
|
7
7
|
import { hostname } from "node:os";
|
|
8
8
|
import { join } from "node:path";
|
|
9
|
+
import { Worker } from "node:worker_threads";
|
|
9
10
|
import { FencedError, ownWriteFenced, PdaError } from "./errors.js";
|
|
10
11
|
import { thisBoot } from "./process-table.js";
|
|
11
12
|
export const RUN_JSON = "run.json";
|
|
@@ -158,6 +159,84 @@ export async function writeRunRecord(root, record, persist = persistRecord, sign
|
|
|
158
159
|
throw ownWriteFenced(join(typeof root === "string" ? root : root.root, RUN_JSON), error);
|
|
159
160
|
}
|
|
160
161
|
}
|
|
162
|
+
// ---- the supervisor's start mark --------------------------------------------------------------------------------------
|
|
163
|
+
/**
|
|
164
|
+
* The supervisor's record of its last start of a run, in the run's directory: the generation the instance will write,
|
|
165
|
+
* when, by whom, and the failed starts right before it (`failures`; `lastExit`, what the decision saw of the last one).
|
|
166
|
+
*/
|
|
167
|
+
export const START_MARK = "start.json";
|
|
168
|
+
/** A start mark's text as a mark, or null when it is not one (an unreadable mark gives no grace). */
|
|
169
|
+
export function parseStartMark(text) {
|
|
170
|
+
try {
|
|
171
|
+
const v = JSON.parse(text);
|
|
172
|
+
if (!Number.isSafeInteger(v.generation) || v.generation < 1 || typeof v.at !== "string" || !Number.isFinite(Date.parse(v.at)))
|
|
173
|
+
return null;
|
|
174
|
+
const failures = Number.isSafeInteger(v.failures) && v.failures >= 0 ? v.failures : 0;
|
|
175
|
+
const lastExit = typeof v.lastExit === "string" ? v.lastExit : null;
|
|
176
|
+
return { generation: v.generation, at: v.at, by: typeof v.by === "string" ? v.by : null, failures, lastExit };
|
|
177
|
+
}
|
|
178
|
+
catch {
|
|
179
|
+
return null;
|
|
180
|
+
}
|
|
181
|
+
}
|
|
182
|
+
// The keeper's code is plain CommonJS in a string, started with `eval: true` and no inherited Node flags, as the lease
|
|
183
|
+
// watchdog's is (src/watchdog.ts says why). One refresh at a time, all synchronous: a stop (the message) is handled only
|
|
184
|
+
// between two refreshes, so none is still writing once the thread has exited.
|
|
185
|
+
const KEEPER_SOURCE = `"use strict";
|
|
186
|
+
const fs = require("node:fs");
|
|
187
|
+
const path = require("node:path");
|
|
188
|
+
const { parentPort, workerData } = require("node:worker_threads");
|
|
189
|
+
const { dir, name, everyMs } = workerData;
|
|
190
|
+
const file = path.join(dir, name);
|
|
191
|
+
function refresh() {
|
|
192
|
+
try {
|
|
193
|
+
const mark = JSON.parse(fs.readFileSync(file, "utf8"));
|
|
194
|
+
if (mark === null || typeof mark !== "object" || mark.generation !== 1 || typeof mark.at !== "string" || !Number.isFinite(Date.parse(mark.at))) return;
|
|
195
|
+
const temp = file + ".tmp";
|
|
196
|
+
const fd = fs.openSync(temp, "w", 0o644);
|
|
197
|
+
try {
|
|
198
|
+
fs.writeSync(fd, JSON.stringify({ ...mark, at: new Date().toISOString() }));
|
|
199
|
+
fs.fsyncSync(fd);
|
|
200
|
+
} finally {
|
|
201
|
+
fs.closeSync(fd);
|
|
202
|
+
}
|
|
203
|
+
fs.renameSync(temp, file);
|
|
204
|
+
const handle = fs.openSync(dir, "r");
|
|
205
|
+
try {
|
|
206
|
+
fs.fsyncSync(handle);
|
|
207
|
+
} finally {
|
|
208
|
+
fs.closeSync(handle);
|
|
209
|
+
}
|
|
210
|
+
} catch {
|
|
211
|
+
// absent, unreadable or refused
|
|
212
|
+
}
|
|
213
|
+
}
|
|
214
|
+
refresh();
|
|
215
|
+
const timer = setInterval(refresh, everyMs);
|
|
216
|
+
parentPort.once("message", () => {
|
|
217
|
+
clearInterval(timer);
|
|
218
|
+
parentPort.close();
|
|
219
|
+
});
|
|
220
|
+
`;
|
|
221
|
+
/**
|
|
222
|
+
* Keeps a supervisor's start mark for generation 1 in `dir` fresh: its `at` rewritten every `everyMs` (temp file, fsync,
|
|
223
|
+
* rename, fsync the directory), the rest of it as the supervisor wrote it; nothing is made when there is none. On a thread of
|
|
224
|
+
* its own, so a caller whose thread is blocked (a synchronous import) is kept fresh too. Best effort: a refused write leaves
|
|
225
|
+
* the mark as it was. Returns the stop, which resolves once the thread has exited, so no write outlives it.
|
|
226
|
+
*/
|
|
227
|
+
export function keepStartMark(dir, everyMs) {
|
|
228
|
+
const worker = new Worker(KEEPER_SOURCE, { eval: true, workerData: { dir, name: START_MARK, everyMs }, execArgv: [] });
|
|
229
|
+
worker.unref();
|
|
230
|
+
worker.on("error", () => {
|
|
231
|
+
// the thread failed to start or threw: nothing more is refreshed, and the grace runs out as for any start
|
|
232
|
+
});
|
|
233
|
+
const exited = new Promise((resolve) => worker.once("exit", () => resolve()));
|
|
234
|
+
return () => {
|
|
235
|
+
worker.ref();
|
|
236
|
+
worker.postMessage("stop");
|
|
237
|
+
return exited;
|
|
238
|
+
};
|
|
239
|
+
}
|
|
161
240
|
// ---- the claim directory ----------------------------------------------------------------------------------------------
|
|
162
241
|
/** The FencedError code of a claim root that is no longer the mount the instance opened (unmounted, lazily or not). */
|
|
163
242
|
export const CLAIM_UNMOUNTED = "CLAIM_UNMOUNTED";
|
package/dist/supervise.d.ts
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
import type { Delegation } from "disk";
|
|
2
2
|
import { type ArchilHost, type ControlApi, type PathlessResolver, type RunRef } from "./claim.js";
|
|
3
3
|
import { PdaError } from "./errors.js";
|
|
4
|
-
import { type RunRecord } from "./status.js";
|
|
4
|
+
import { type RunRecord, type StartMark } from "./status.js";
|
|
5
5
|
export type Json = null | boolean | number | string | Json[] | {
|
|
6
6
|
[key: string]: Json;
|
|
7
7
|
};
|
|
@@ -21,6 +21,22 @@ export type StartAttempt = {
|
|
|
21
21
|
attempt: number;
|
|
22
22
|
env?: Record<string, string>;
|
|
23
23
|
};
|
|
24
|
+
/**
|
|
25
|
+
* A box a driver made for an instance, as a receipt: who provided it, what kind, when it was made and when it ended
|
|
26
|
+
* (null while it lives), and its published list price per second (`usdPerSecond`), with the page that price was read
|
|
27
|
+
* from (`source`) and the ISO date it was read (`priced`). A box with no price (a kind the provider's page does not
|
|
28
|
+
* price, or of unknown size) has none of the three: never a price of 0. No field holds the provider's id of the box:
|
|
29
|
+
* the supervisor labels boxes itself and computes their seconds and cost.
|
|
30
|
+
*/
|
|
31
|
+
export type HostBox = {
|
|
32
|
+
provider: string;
|
|
33
|
+
kind: string;
|
|
34
|
+
from: string;
|
|
35
|
+
to: string | null;
|
|
36
|
+
usdPerSecond?: number;
|
|
37
|
+
source?: string;
|
|
38
|
+
priced?: string;
|
|
39
|
+
};
|
|
24
40
|
/** Compute is disposable and pluggable: a driver is these three calls. */
|
|
25
41
|
export interface HostDriver {
|
|
26
42
|
/**
|
|
@@ -41,6 +57,8 @@ export interface HostDriver {
|
|
|
41
57
|
kill?(handle: HostHandle): Promise<void>;
|
|
42
58
|
/** What `kill` takes: `process`, the instance's processes while its machine stays (a restart policy may bring them back), or `box`, its machine. */
|
|
43
59
|
readonly kills?: "process" | "box";
|
|
60
|
+
/** Optional, for a driver whose instances get machines of their own: the box `handle`'s instance runs in, as a receipt. */
|
|
61
|
+
box?(handle: HostHandle): HostBox | null;
|
|
44
62
|
}
|
|
45
63
|
/** The control-plane calls the supervisor uses; an archil `Disk` satisfies it. */
|
|
46
64
|
export interface SupervisorControl extends ControlApi {
|
|
@@ -227,18 +245,7 @@ export type EnsureResult =
|
|
|
227
245
|
* supervisors are safe: the mount admits one claimant and the other instance exits 76.
|
|
228
246
|
*/
|
|
229
247
|
export declare function ensureRunning(ref: RunRef, host: HostDriver, options: EnsureOptions): Promise<EnsureResult>;
|
|
230
|
-
|
|
231
|
-
* The supervisor's record of its last start of a run: the generation the instance will write, when, by whom, and the
|
|
232
|
-
* failed starts right before it (`failures`; `lastExit`, what the decision saw of the last one).
|
|
233
|
-
*/
|
|
234
|
-
export declare const START_MARK = "start.json";
|
|
235
|
-
export type StartMark = {
|
|
236
|
-
generation: number;
|
|
237
|
-
at: string;
|
|
238
|
-
by: string | null;
|
|
239
|
-
failures: number;
|
|
240
|
-
lastExit: string | null;
|
|
241
|
-
};
|
|
248
|
+
export { START_MARK, type StartMark } from "./status.js";
|
|
242
249
|
/** The grace of a start that follows `failures` failed ones: the base doubled per failure, capped, never below the base. */
|
|
243
250
|
export declare function startGrace(baseMs: number, failures: number, maxMs?: number): number;
|
|
244
251
|
/**
|
package/dist/supervise.js
CHANGED
|
@@ -8,7 +8,7 @@ import { hostname } from "node:os";
|
|
|
8
8
|
import { join } from "node:path";
|
|
9
9
|
import { acquire, createRunDir, findDelegations, matchDelegations, pathlessResolver, revokeBestEffort, revokeCompanions, mintMountToken, MOUNT_TOKEN_TTL, parseTokenNickname, removeMountToken, runPath, unmountClaim, } from "./claim.js";
|
|
10
10
|
import { FencedError, HeldError, PdaError } from "./errors.js";
|
|
11
|
-
import { parseRunRecord, RUN_JSON } from "./status.js";
|
|
11
|
+
import { parseRunRecord, parseStartMark, RUN_JSON, START_MARK } from "./status.js";
|
|
12
12
|
export const CONTROL_TIMEOUT_MS = 10_000;
|
|
13
13
|
/** The same control API with every call bounded: one that does not settle in `ms` fails as CONTROL_API_FAILED. */
|
|
14
14
|
export function withTimeouts(control, ms) {
|
|
@@ -160,11 +160,8 @@ export async function ensureRunning(ref, host, options) {
|
|
|
160
160
|
await revokeListed(opts.control, ref.id, held, all);
|
|
161
161
|
return start("lease-expired", held, stonith);
|
|
162
162
|
}
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
* failed starts right before it (`failures`; `lastExit`, what the decision saw of the last one).
|
|
166
|
-
*/
|
|
167
|
-
export const START_MARK = "start.json";
|
|
163
|
+
// The start mark (`start.json`): its name, shape and parser live in status.ts, which the instance also reads it with.
|
|
164
|
+
export { START_MARK } from "./status.js";
|
|
168
165
|
/** The grace of a start that follows `failures` failed ones: the base doubled per failure, capped, never below the base. */
|
|
169
166
|
export function startGrace(baseMs, failures, maxMs = START_BACKOFF_MAX_MS) {
|
|
170
167
|
if (baseMs <= 0)
|
|
@@ -193,17 +190,7 @@ export async function readStartMark(control, id) {
|
|
|
193
190
|
return null;
|
|
194
191
|
throw new SuperviseError("CONTROL_API_FAILED", `GetObject ${runPath(id)}/${START_MARK} failed`, { cause: err });
|
|
195
192
|
}
|
|
196
|
-
|
|
197
|
-
const v = JSON.parse(text);
|
|
198
|
-
if (!Number.isSafeInteger(v.generation) || v.generation < 1 || typeof v.at !== "string" || !Number.isFinite(Date.parse(v.at)))
|
|
199
|
-
return null;
|
|
200
|
-
const failures = Number.isSafeInteger(v.failures) && v.failures >= 0 ? v.failures : 0;
|
|
201
|
-
const lastExit = typeof v.lastExit === "string" ? v.lastExit : null;
|
|
202
|
-
return { generation: v.generation, at: v.at, by: typeof v.by === "string" ? v.by : null, failures, lastExit };
|
|
203
|
-
}
|
|
204
|
-
catch {
|
|
205
|
-
return null;
|
|
206
|
-
}
|
|
193
|
+
return parseStartMark(text);
|
|
207
194
|
}
|
|
208
195
|
/** Only a run with no run.json and no holder gets here, so the directory is either absent or idle. */
|
|
209
196
|
async function ensureRunDir(control, id, owner) {
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@parcha/pi-durable-disk",
|
|
3
|
-
"version": "0.1.0-beta.
|
|
3
|
+
"version": "0.1.0-beta.16",
|
|
4
4
|
"description": "A durable pi host on a network disk (Archil): one owner per run, a fence, durable commits, and a supervisor.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"pi",
|
|
@@ -56,7 +56,7 @@
|
|
|
56
56
|
"test": "node --test --test-timeout=240000 'test/*.test.ts' 'test/examples/*.test.ts'",
|
|
57
57
|
"test:acceptance": "node --test --test-concurrency=1 'test/acceptance/*.test.ts'",
|
|
58
58
|
"test:live": "node --test --test-concurrency=1 'test/live/*.test.ts'",
|
|
59
|
-
"test:live:claim": "node --test --test-concurrency=1 test/live/claim.live.test.ts",
|
|
59
|
+
"test:live:claim": "node --test --test-concurrency=1 test/live/claim.live.test.ts test/live/record-home.live.test.ts",
|
|
60
60
|
"test:live:env": "node --test --test-concurrency=1 test/live/env.live.test.ts",
|
|
61
61
|
"test:live:lifecycle": "node --test --test-concurrency=1 test/live/lifecycle.live.test.ts",
|
|
62
62
|
"test:live:run": "node --test --test-concurrency=1 test/live/run.live.test.ts",
|