@parcha/pi-durable-disk 0.1.0-beta.15 → 0.1.0-beta.16

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -2,6 +2,51 @@
2
2
 
3
3
  ## Unreleased
4
4
 
5
+ ## 0.1.0-beta.16 (2026-10-18)
6
+
7
+ - **A place's outcome carries the models the run's live parts chose (`PlaceOutcome.models`)**
8
+ - `PlaceOutcome.models?: Record<string, string>`: the role-to-model ids (`lead`, `eyes`, ...) the app reported, on any
9
+ outcome status, so a supervisor's manifest names the models of a remote run as of a local one.
10
+ - `archilPlace` reads it from `run.json`'s final status detail (`detail.models`); `localPlace` from the instance's
11
+ `{"outcome":...}` line, or from `run.json` without one.
12
+ - Both keep it only when it is a plain object whose values are all strings. A missing or malformed one is left out, never
13
+ replaced by `{}`. Before, `localPlace` passed the outcome line's `models` through unchecked, a malformed one included,
14
+ and `archilPlace` never carried it.
15
+ - **`archilPlace.recordHome`: a runner on a box records into the run's disk, and the place brings the recording home**
16
+ - `Place.recordHome` (`PlaceRecordHome`): `dir(attempt)` is where that attempt's runner records, and
17
+ `bringHome(into, { final })` brings what it recorded into the supervisor's recording directory. `archilPlace` has
18
+ it when its control can list (`RecordControl`, as `archilControl` does); `localPlace` has none, since its runner
19
+ writes the supervisor's directory itself.
20
+ - `dir(attempt)` is `<root>/record/<attempt>`, inside the run's directory. `bringHome` reads it over S3 (list and get,
21
+ never the run's claim). Blobs come first, hash-checked and written whole; then the new whole lines of each
22
+ `.spool/*.jsonl` segment, with an offset per file name kept as what `into` already holds, so it is idempotent across
23
+ calls and processes. Carriers that overlap on one destination append each line once: one call at a time per
24
+ destination in a place, and a lock file per spool file across places and processes (a gone holder's is taken over;
25
+ one held past 5 s is `CARRY_FAILED`). Every write is carried to its last byte.
26
+ - With `ensure.create`, each start first makes the EMPTY `signals/`, `record/<attempt>/.spool/` and
27
+ `record/<attempt>/blobs/` over the control, owned by the run user (mode 0755), while no delegation holds the run. A
28
+ directory made by path on an Archil mount was seen to refuse file creates. Every file in them stays the runner's; a
29
+ refused put fails the start with `DIRS_REFUSED`.
30
+ - Box receipts: `HostDriver.box(handle)` (`HostBox`) and `PlaceStatus.boxes`. `archilPlace.status()` lists a receipt
31
+ for each box it started, released ones included: provider, kind, `from`, `to` and, when it has a price, the published
32
+ list price per second (`usdPerSecond`), the page it was read from (`source`) and the date read (`priced`). An unpriced
33
+ box has none of the three, never 0, and no field holds the provider's box id. `daytonaHost` reports its sandboxes, priced at `DAYTONA_LIST_PRICES`
34
+ (https://www.daytona.io/pricing, read 2026-10-16), with the "first 5 GiB free" applied to disk (the footnote's
35
+ placement is ambiguous). A box of unknown size has no price, nor does a box with a GPU, which the page prices per
36
+ model (`daytonaUsdPerSecond`). `localPlace` reports none.
37
+ - Failures carry codes (`PlaceError`). `BLOB_MISMATCH` is thrown after everything else came home. `CARRY_FAILED` is
38
+ thrown for a list or get that fails, and on `final` for a line whose blob never landed. `BOX_GONE` is never thrown
39
+ by this place: the recording lives on the disk, which outlives the box.
40
+ - **A seed of any length stays in the supervisor's start grace**
41
+ - While a new run seeds it has no `run.json` heartbeat, and `ensureRunning` keeps a start in its grace only while the
42
+ start mark (`runs/<id>/start.json`) is younger than the grace (90 s by default). A longer seed was taken as a failed
43
+ start: revoked, then seeded again.
44
+ - Now, while the seed runs, the instance rewrites the mark's `at` every heartbeat through its claim (fsync and rename).
45
+ It does so only when a supervisor's mark for generation 1 is there, keeping its generation, `by`, `failures` and
46
+ `lastExit`. The rewrite runs on a thread of its own, one at a time, so a seed that blocks its thread (a synchronous
47
+ import) stays covered. It stops when the seed ends, before the open goes on, and `run.json`'s heartbeat takes over.
48
+ - A seeder that dies stops rewriting, and the grace runs out as for any start. The mark's format does not change.
49
+
5
50
  ## 0.1.0-beta.15 (2026-10-16)
6
51
 
7
52
  - **`HostDriver.kill`: take an instance away as a crash would, and `kills` says what that takes**
package/README.md CHANGED
@@ -422,7 +422,13 @@ except after exit 65, 70, 75 or 76. `localHost({ mode: "child" })` runs a detach
422
422
  without that guarantee.
423
423
 
424
424
  `daytonaHost({ client, snapshot, ... })` runs each instance in a Daytona sandbox of its own (library only: `supervise`
425
- starts local units, so pass the driver to `ensureRunning`). The box comes from a snapshot that holds Node, the archil client
425
+ starts local units, so pass the driver to `ensureRunning`). Its `box(handle)` is that sandbox's receipt: its size
426
+ (vCPUs, memory, disk), when it was made and when the driver deleted it, and its list price per second at
427
+ `DAYTONA_LIST_PRICES`. Those are Daytona's published rates, with the page and the date they were read
428
+ (https://www.daytona.io/pricing, 2026-10-16): $0.0504 per vCPU-hour, $0.0162 per GiB-hour of memory and $0.000108 per
429
+ GiB-hour of disk past the first 5 GiB, billed per second. The page's "first 5 free" footnote is applied to disk only (its
430
+ placement is ambiguous). A box of unknown size gets no price, and neither does a box with a GPU: the page prices GPUs per
431
+ model, which the driver doesn't map (`daytonaUsdPerSecond`). The box comes from a snapshot that holds Node, the archil client
426
432
  (`libfuse2`), this package, the run user and the sudoers drop-in, or from a default image that the caller's `prepare` installs
427
433
  them into. A container sandbox has no systemd (PID 1 is the Daytona daemon), so an in-box launcher keeps the instance
428
434
  alive in place and never restarts it after exit 0, 65, 70, 75 or 76. Every incarnation of an attempt gets the attempt's
@@ -757,10 +763,14 @@ await openDurableRun(ref, {
757
763
  - **When it fails.** A seed that throws fails the open with its own error. A process killed while it seeds leaves the run
758
764
  unpublished in the same way: no `run.json`, the claim let go. The next open seeds again, so a seed overwrites what an
759
765
  earlier attempt wrote.
760
- - **How long it may take.** Under a supervisor, a seed must end within the start grace (`--start-grace`,
761
- `startGraceMs`; default the lease expiry, 90 s). Past it, `ensureRunning` takes the start as failed and revokes the
762
- claim. The seed's next write or its barrier is then refused, the instance is fenced (exit 75) with nothing published,
763
- and the next start seeds again. A deployment whose seeds can take longer raises the start grace.
766
+ - **How long it may take.** A new run has no `run.json` heartbeat while it seeds. A supervisor keeps a start in its grace
767
+ (`--start-grace`, `startGraceMs`; default the lease expiry, 90 s) only while its start mark (`start.json`) is younger
768
+ than the grace. So while the seed runs, the instance rewrites the mark's time every heartbeat, through its claim, when a
769
+ supervisor's mark for generation 1 is there. It does so on a thread of its own, so a seed that holds its thread (a
770
+ synchronous import through `DatabaseSync`) is covered too. A live seed of any length stays `starting`. A seeder that dies stops
771
+ rewriting, the grace runs out as for any start, and the claim is revoked. If the claim is revoked while the seed still
772
+ runs, its next write or its barrier is refused, the instance is fenced (exit 75) with nothing published, and the next
773
+ start seeds again.
764
774
  - **On a local directory** (`localRun`) it is the same. As with `work/`, the files it writes are the operating system's
765
775
  to flush; a SQLite database it commits with `synchronous = FULL` is durable when the commit returns.
766
776
 
@@ -887,7 +897,35 @@ id and directory as the supervisor computed them.
887
897
  - A recovery is the takeover: the killed holder's delegations are revoked whatever its lease says, and the next
888
898
  instance starts. It reports `revoked` and `ms`. A newer incarnation than the one the place killed (another
889
899
  supervisor's) gets the usual lease and start-grace checks instead, so a live one is `HELD`, not revoked.
890
- - `status()` reads `run.json` over S3 and lists the run's delegations.
900
+ - `status()` reads `run.json` over S3 and lists the run's delegations, plus, when its host driver makes boxes
901
+ (`daytonaHost`), a receipt for each box the place started, released ones included (`boxes`, `HostBox`). A receipt
902
+ has the provider, the kind, `from` and `to` (null while it lives). It also has the published list price per second
903
+ (`usdPerSecond`), the page that price was read from (`source`) and the ISO date it was read (`priced`). A box with
904
+ no price has none of those three, never a price of 0. No field holds the provider's id of the box.
905
+ - `recordHome` (when the control can list, as `archilControl`'s can): an attempt's runner records into
906
+ `recordHome.dir(attempt)`, `<root>/record/<attempt>`, in the run's directory, so its delegation covers it and
907
+ `deleteRunTree` removes it. `bringHome(into, { final })` reads that directory through the control (S3 list and get:
908
+ nothing takes the run's claim) and brings the recording into the supervisor's `into`.
909
+ - Blobs come first, each checked against the hash its name carries and written whole (a temporary file, fsync,
910
+ rename). One that doesn't match is skipped; everything else still comes home, then the call throws
911
+ `BLOB_MISMATCH`.
912
+ - Then the new whole lines of each `.spool/*.jsonl` segment, with an offset per file name: what `into` already
913
+ holds of it. So a second call, or a call from another process, brings nothing twice. A line waits while a blob it
914
+ names isn't home yet. On `final` it comes home anyway, and the call throws `CARRY_FAILED` naming the blob.
915
+ - Carriers that overlap on one `into` never append at once. The place runs one call at a time per destination, and
916
+ each append happens under a lock file beside its spool file (`.<name>.lock`, naming its holder's pid), after the
917
+ destination's size is read again, so only the lines past it are appended. A lock whose holder is gone is taken
918
+ over; one held past 5 s is `CARRY_FAILED`. A second `into` gets every line: what the place fetched for one
919
+ destination says nothing of another.
920
+ - A list or get that fails is `CARRY_FAILED`.
921
+ - The recording is on the disk, not in the box, so a final call after a box kill still brings home everything the
922
+ box fsynced: this place never throws `BOX_GONE`.
923
+ - With `ensure.create` (the run user's uid and gid), the place makes the EMPTY directories the runner writes into
924
+ at each start: `signals/`, `record/<attempt>/.spool/` and `record/<attempt>/blobs/`. It makes them over the
925
+ control, owned by the run user, mode 0755, in ensureRunning's window where no delegation holds the run. A
926
+ directory a process made by path on an Archil mount was seen to refuse file creates afterwards (ENOENT), and one
927
+ made over the control didn't. Every file in them stays the runner's. A refused put fails the start
928
+ (`DIRS_REFUSED`, as the cause of `HOST_START_FAILED`), and without an owner nothing is made.
891
929
  - `requestFile(attempt)` is `<root>/signals/kill-<attempt>.json`, in the run's directory and outside `work/`, and
892
930
  `signals(attempt)` reads `runs/<id>/signals/kill-<attempt>.json` over S3 every 500 ms (`signalPollMs`). A runner on
893
931
  a box of its own can't reach the supervisor's files, so its request travels through the disk. The runner must write
@@ -912,6 +950,11 @@ when the run is `done` and that detail carries a record, `failed` for anything e
912
950
  run its instance left `running`, `paused` or `sleeping`). So an app that runs on a box and wants its supervisor to have
913
951
  its record puts the record there, in its final `setStatus("done", { record })`.
914
952
 
953
+ Either way the outcome may carry `models`, the role-to-model ids the run's live parts chose (`lead`, `eyes`, ...), on any
954
+ status, a failed run's too: from the outcome line's `models`, or from the final detail's (`setStatus("done", { record,
955
+ models })`). Both places keep it only when it is a plain object whose values are all strings; a missing or malformed one
956
+ is left out, never replaced by `{}`.
957
+
915
958
  ## Tests
916
959
 
917
960
  ```sh
@@ -7,12 +7,37 @@ export type SandboxInfo = {
7
7
  id: string;
8
8
  name: string;
9
9
  state?: string;
10
+ /** The box's size as Daytona allocates it: vCPUs, GPUs, memory and disk (the API's GB). */
11
+ cpu?: number;
12
+ gpu?: number;
13
+ memory?: number;
14
+ disk?: number;
15
+ createdAt?: string;
10
16
  snapshot?: string;
11
17
  labels?: Record<string, string>;
12
18
  target?: string;
13
19
  toolboxProxyUrl?: string;
14
20
  errorReason?: string;
15
21
  };
22
+ /**
23
+ * Daytona's published list prices, as its pricing page gave them on `readOn`: billed per second, quoted per hour, for
24
+ * each vCPU, each GiB of memory and each GiB of disk. The page's footnote "Price per GiB after first 5 free" is applied
25
+ * to storage (disk) only: its placement is ambiguous and could be read as covering memory too, so this is a choice.
26
+ * The page lists GPU rates per GPU model; this driver does not map a box's GPU to a model, so a GPU box is unpriced.
27
+ */
28
+ export declare const DAYTONA_LIST_PRICES: {
29
+ readonly page: "https://www.daytona.io/pricing";
30
+ readonly readOn: "2026-10-16";
31
+ readonly vcpuPerHour: 0.0504;
32
+ readonly memoryGiBPerHour: 0.0162;
33
+ readonly diskGiBPerHour: 0.000108;
34
+ readonly freeDiskGiB: 5;
35
+ };
36
+ /**
37
+ * A box's list price per second at DAYTONA_LIST_PRICES, from its size; null when its size is unknown or it has a GPU
38
+ * (priced per GPU model on the page, which this driver does not map).
39
+ */
40
+ export declare function daytonaUsdPerSecond(size: Pick<SandboxInfo, "cpu" | "gpu" | "memory" | "disk">): number | null;
16
41
  export type CreateSandboxBody = {
17
42
  name: string;
18
43
  snapshot: string;
@@ -21,6 +21,32 @@ import { TERMINAL_EXITS } from "./local-host.js";
21
21
  export const DAYTONA_API_URL = "https://app.daytona.io/api";
22
22
  export const LABEL_FLEET = "pda-fleet";
23
23
  export const LABEL_RUN = "pda-run";
24
+ /**
25
+ * Daytona's published list prices, as its pricing page gave them on `readOn`: billed per second, quoted per hour, for
26
+ * each vCPU, each GiB of memory and each GiB of disk. The page's footnote "Price per GiB after first 5 free" is applied
27
+ * to storage (disk) only: its placement is ambiguous and could be read as covering memory too, so this is a choice.
28
+ * The page lists GPU rates per GPU model; this driver does not map a box's GPU to a model, so a GPU box is unpriced.
29
+ */
30
+ export const DAYTONA_LIST_PRICES = {
31
+ page: "https://www.daytona.io/pricing",
32
+ readOn: "2026-10-16",
33
+ vcpuPerHour: 0.0504,
34
+ memoryGiBPerHour: 0.0162,
35
+ diskGiBPerHour: 0.000108,
36
+ freeDiskGiB: 5,
37
+ };
38
+ /**
39
+ * A box's list price per second at DAYTONA_LIST_PRICES, from its size; null when its size is unknown or it has a GPU
40
+ * (priced per GPU model on the page, which this driver does not map).
41
+ */
42
+ export function daytonaUsdPerSecond(size) {
43
+ const { cpu, memory, disk, gpu } = size;
44
+ const count = (n) => typeof n === "number" && Number.isFinite(n) && n >= 0;
45
+ if (!count(cpu) || !count(memory) || !count(disk) || (gpu !== undefined && gpu !== 0))
46
+ return null;
47
+ const p = DAYTONA_LIST_PRICES;
48
+ return (cpu * p.vcpuPerHour + memory * p.memoryGiBPerHour + Math.max(0, disk - p.freeDiskGiB) * p.diskGiBPerHour) / 3600;
49
+ }
24
50
  export class DaytonaApiError extends PdaError {
25
51
  /** The HTTP status, null when no response arrived (network failure or timeout). */
26
52
  status;
@@ -256,6 +282,13 @@ export function daytonaHost(opts) {
256
282
  throw new DaytonaHostError("START_FAILED", `${what} in ${box.name} exited ${r.exitCode}: ${r.result.trim().split("\n").at(-1)?.slice(0, 300) ?? ""}`);
257
283
  return r;
258
284
  }
285
+ /** Each box this driver made, for its receipt: what Daytona said of it, when it was made, and when it ended. */
286
+ const made = new Map();
287
+ const ended = (id) => {
288
+ const m = made.get(id);
289
+ if (m && m.to === null)
290
+ m.to = new Date(now()).toISOString();
291
+ };
259
292
  return {
260
293
  fleet,
261
294
  mountRoot,
@@ -279,8 +312,10 @@ export function daytonaHost(opts) {
279
312
  ...(opts.network?.allowList ? { networkAllowList: opts.network.allowList } : {}),
280
313
  };
281
314
  let box = await createOrAdopt(body, ref.id);
315
+ made.set(box.id, { info: box, from: box.createdAt ?? new Date(t0).toISOString(), to: null });
282
316
  try {
283
317
  box = await waitStarted(box, t0 + startTimeoutMs);
318
+ made.get(box.id).info = { ...made.get(box.id).info, ...box };
284
319
  const prepared = now();
285
320
  if (opts.prepare)
286
321
  await opts.prepare(box, client);
@@ -308,7 +343,9 @@ export function daytonaHost(opts) {
308
343
  return handle;
309
344
  }
310
345
  catch (err) {
346
+ // The box was made and is deleted now: its receipt ends here, deleted or not (a delete that failed is the sweep's).
311
347
  await client.remove(box.id).catch(() => { });
348
+ ended(box.id);
312
349
  if (err instanceof DaytonaHostError)
313
350
  throw err;
314
351
  throw new DaytonaHostError("START_FAILED", `starting an instance in ${name} failed: ${err.message}`, { cause: err });
@@ -324,8 +361,10 @@ export function daytonaHost(opts) {
324
361
  catch (err) {
325
362
  throw new DaytonaHostError("DAYTONA_API_FAILED", `reading sandbox ${h.name}: ${err.message}`, { cause: err });
326
363
  }
327
- if (!box)
364
+ if (!box) {
365
+ ended(h.sandboxId);
328
366
  return "gone";
367
+ }
329
368
  // The label, not the id in a handle, decides whose box this is: a handle can never reach another fleet's box.
330
369
  if (box.labels?.[LABEL_FLEET] !== fleet)
331
370
  return "unknown";
@@ -366,6 +405,7 @@ export function daytonaHost(opts) {
366
405
  throw new DaytonaHostError("STOP_FAILED", `deleting sandbox ${h.name} failed: ${err.message}`, { cause: first });
367
406
  }
368
407
  }
408
+ ended(id);
369
409
  },
370
410
  async kill(h) {
371
411
  if (!mine(h))
@@ -394,6 +434,24 @@ export function daytonaHost(opts) {
394
434
  throw new DaytonaHostError("KILL_FAILED", `deleting sandbox ${h.name} failed: ${err.message}`, { cause: first });
395
435
  }
396
436
  }
437
+ ended(id);
438
+ },
439
+ box(h) {
440
+ if (!mine(h))
441
+ return null;
442
+ const m = made.get(h.sandboxId);
443
+ if (!m)
444
+ return null;
445
+ const { cpu, gpu, memory, disk } = m.info;
446
+ const usdPerSecond = daytonaUsdPerSecond(m.info);
447
+ const size = [cpu, memory, disk].every((n) => typeof n === "number") ? ` ${cpu} vCPU, ${memory} GiB memory, ${disk} GiB disk${gpu ? `, ${gpu} GPU` : ""}` : "";
448
+ return {
449
+ provider: "daytona",
450
+ kind: `sandbox${size}`,
451
+ from: m.from,
452
+ to: m.to,
453
+ ...(usdPerSecond === null ? {} : { usdPerSecond, source: DAYTONA_LIST_PRICES.page, priced: DAYTONA_LIST_PRICES.readOn }),
454
+ };
397
455
  },
398
456
  };
399
457
  }
package/dist/index.d.ts CHANGED
@@ -18,17 +18,17 @@ export type { ArchilStore, Fence, FencedDatabase, Profile, StoreFencedError } fr
18
18
  export { AppError } from "./app.js";
19
19
  export type { AppContext, AppModule, AppOptions } from "./app.js";
20
20
  export { archilControl, deleteRunTree, ensureRunning, readRunStatus, SuperviseError } from "./supervise.js";
21
- export type { ArchilControl, CheckControl, CheckOptions, DiskClient, EnsureOptions, EnsureResult, HostDriver, HostHandle, HostStatus, Json, StartAttempt, StartReason, StonithOutcome, SuperviseErrorCode, SupervisorControl, TokenUser } from "./supervise.js";
21
+ export type { ArchilControl, CheckControl, CheckOptions, DiskClient, EnsureOptions, EnsureResult, HostBox, HostDriver, HostHandle, HostStatus, Json, StartAttempt, StartReason, StonithOutcome, SuperviseErrorCode, SupervisorControl, TokenUser } from "./supervise.js";
22
22
  export { localHost, LocalHostError } from "./hosts/local-host.js";
23
23
  export type { LocalHostOptions, Ran, Runner } from "./hosts/local-host.js";
24
24
  export { dockerHost, DockerHostError } from "./hosts/docker.js";
25
25
  export type { ContainerInfo, DockerHostOptions, DockerRan, DockerRunner } from "./hosts/docker.js";
26
- export { DaytonaApiError, daytonaHost, DaytonaHostError, daytonaRest } from "./hosts/daytona.js";
26
+ export { DAYTONA_LIST_PRICES, DaytonaApiError, daytonaHost, DaytonaHostError, daytonaRest, daytonaUsdPerSecond } from "./hosts/daytona.js";
27
27
  export type { CreateSandboxBody, DaytonaClient, DaytonaHostErrorCode, DaytonaHostOptions, DaytonaRestOptions, ExecResult, SandboxInfo } from "./hosts/daytona.js";
28
28
  export { requestRun, ServeConfigError, ServeError, serveRun } from "./serve.js";
29
29
  export type { RequestOptions, RunResponse, RunServer, ServeConfigErrorCode, ServeErrorCode, ServeOptions } from "./serve.js";
30
30
  export { archilPlace, localPlace, PlaceError } from "./place.js";
31
- export type { ArchilPlaceOptions, LocalPlaceOptions, Place, PlaceContext, PlaceErrorCode, PlaceExit, PlaceKill, PlaceOutcome, PlacePaths, PlaceRecovery, PlaceRunner, PlaceSignal, PlaceStatus, RunnerOrder } from "./place.js";
31
+ export type { ArchilPlaceOptions, LocalPlaceOptions, Place, PlaceContext, PlaceErrorCode, PlaceExit, PlaceKill, PlaceOutcome, PlacePaths, PlaceRecordHome, PlaceRecovery, PlaceRunner, PlaceSignal, PlaceStatus, RecordControl, RunnerOrder } from "./place.js";
32
32
  export { busyState, drain, leaseParkTarget, watchParking } from "./park.js";
33
33
  export type { BusyState, DrainOptions, Parking, ParkOptions, ParkResult, ParkTarget } from "./park.js";
34
34
  export { fork, ForkError, forkLocal } from "./fork.js";
package/dist/index.js CHANGED
@@ -17,7 +17,7 @@ export { AppError } from "./app.js";
17
17
  export { archilControl, deleteRunTree, ensureRunning, readRunStatus, SuperviseError } from "./supervise.js";
18
18
  export { localHost, LocalHostError } from "./hosts/local-host.js";
19
19
  export { dockerHost, DockerHostError } from "./hosts/docker.js";
20
- export { DaytonaApiError, daytonaHost, DaytonaHostError, daytonaRest } from "./hosts/daytona.js";
20
+ export { DAYTONA_LIST_PRICES, DaytonaApiError, daytonaHost, DaytonaHostError, daytonaRest, daytonaUsdPerSecond } from "./hosts/daytona.js";
21
21
  // Serve: one HTTP request runs one turn.
22
22
  export { requestRun, ServeConfigError, ServeError, serveRun } from "./serve.js";
23
23
  // Places: where a run lives for a supervisor that kills it and brings it back (agentrun's reference place seam).
package/dist/place.d.ts CHANGED
@@ -1,7 +1,7 @@
1
1
  import type { Delegation } from "disk";
2
2
  import { PdaError } from "./errors.js";
3
3
  import { type RunRecord } from "./status.js";
4
- import { type EnsureOptions, type HostDriver, type SupervisorControl } from "./supervise.js";
4
+ import { type EnsureOptions, type HostBox, type HostDriver, type SupervisorControl } from "./supervise.js";
5
5
  export type PlaceKill = "process" | "box";
6
6
  /** The run's paths in the place: its id, its directory, and the `work` and `store` under it. */
7
7
  export type PlacePaths = {
@@ -12,7 +12,8 @@ export type PlacePaths = {
12
12
  };
13
13
  /**
14
14
  * How the run stood when a runner ended: the instance's own report, or `run.json` as it left it. `status` is the run's
15
- * word: `complete` when it delivered, else `failed`.
15
+ * word: `complete` when it delivered, else `failed`. `models`, on any status: the role-to-model ids the run's live parts
16
+ * chose (`lead`, `eyes`, ...), when the app reported a plain object of strings; absent otherwise, never `{}` in its place.
16
17
  */
17
18
  export type PlaceOutcome = {
18
19
  status: string;
@@ -21,6 +22,7 @@ export type PlaceOutcome = {
21
22
  message?: string;
22
23
  record?: unknown;
23
24
  storeSha256?: string;
25
+ models?: Record<string, string>;
24
26
  };
25
27
  /** How a runner ended. `exit`: the exit code or signal (local), or the host's status (`stopped`, `failed`, `gone`) or `SIGKILL` when the place killed it (archil). */
26
28
  export type PlaceExit = {
@@ -42,9 +44,15 @@ export type RunnerOrder = {
42
44
  stdin?: string;
43
45
  };
44
46
  /** What the place reads of the run, the way it reads it: `run.json`, and on a disk the run's delegations. */
47
+ /**
48
+ * What the place reads of the run, the way it reads it: `run.json`; on a disk, the run's delegations; and the boxes the
49
+ * place made for its runners, each as a receipt (`HostBox`: provider, kind, from, to, list price per second and where
50
+ * that price was read), when its host driver makes boxes. A box the place released stays listed with its end.
51
+ */
45
52
  export type PlaceStatus = {
46
53
  record: RunRecord | null;
47
54
  delegations?: Pick<Delegation, "clientId" | "inodeId" | "path" | "isOrphaned" | "isPending">[];
55
+ boxes?: HostBox[];
48
56
  };
49
57
  /** What a recovery did. A takeover on a disk lists what it revoked, and `ms` is the whole of it (revoke and start). */
50
58
  export type PlaceRecovery = {
@@ -91,6 +99,23 @@ export type PlaceContext = {
91
99
  destroyBoxes?: () => void | Promise<void>;
92
100
  requestFile?: (attempt: number) => string;
93
101
  };
102
+ /**
103
+ * Where an attempt's runner records (its RECORD_DIR, in the place) and how the place brings that recording home. For a
104
+ * place whose runners cannot write the supervisor's recording directory.
105
+ */
106
+ export interface PlaceRecordHome {
107
+ /** The directory, in the place, attempt `attempt`'s runner records into. */
108
+ dir(attempt: number): string;
109
+ /**
110
+ * Brings home into `into` what the runners recorded: blobs first, then the new whole lines of each `.spool/*.jsonl`.
111
+ * Idempotent; `final` is the call after the runner ended, when nothing is still landing.
112
+ */
113
+ bringHome(into: string, options?: {
114
+ final?: boolean;
115
+ }): Promise<void>;
116
+ /** How far the place's clock is from the supervisor's, when it knows. */
117
+ readonly clockSkewMs?: number;
118
+ }
94
119
  export interface Place {
95
120
  /** `local` or `archil`, for the recording's lines. */
96
121
  readonly name: string;
@@ -122,8 +147,10 @@ export interface Place {
122
147
  move?(to: Place): Promise<Record<string, unknown>>;
123
148
  /** Lets the run go: stops what still runs (its commands too) and removes what the place made for it (tokens), never the run's directory. */
124
149
  release(): Promise<void>;
150
+ /** A place whose runners record where the supervisor cannot read says where, and brings the recording home. */
151
+ readonly recordHome?: PlaceRecordHome;
125
152
  }
126
- export type PlaceErrorCode = "INVALID_ARGUMENT" | "HELD" | "START_TIMED_OUT" | "STATUS_FAILED" | "KILL_UNSUPPORTED" | "RELEASE_FAILED";
153
+ export type PlaceErrorCode = "INVALID_ARGUMENT" | "HELD" | "START_TIMED_OUT" | "STATUS_FAILED" | "KILL_UNSUPPORTED" | "RELEASE_FAILED" | "BOX_GONE" | "CARRY_FAILED" | "BLOB_MISMATCH" | "DIRS_REFUSED";
127
154
  export declare class PlaceError extends PdaError {
128
155
  constructor(code: PlaceErrorCode, message: string, options?: {
129
156
  cause?: unknown;
@@ -150,8 +177,11 @@ export declare function localPlace(options: LocalPlaceOptions): (ctx: PlaceConte
150
177
  export interface ArchilPlaceOptions {
151
178
  disk: string;
152
179
  region: string;
153
- /** The disk's control API (`archilControl`). The place holds it; no instance ever does. */
154
- control: SupervisorControl;
180
+ /**
181
+ * The disk's control API (`archilControl`). The place holds it; no instance ever does. With `listObjects` (as
182
+ * `archilControl` has), the place has `recordHome`.
183
+ */
184
+ control: SupervisorControl & Partial<RecordControl>;
155
185
  /**
156
186
  * The driver that runs the instance (`localHost`, `dockerHost`, `daytonaHost`), set up with the app the instance runs.
157
187
  * Its `kill` and `kills` are the place's.
@@ -175,6 +205,22 @@ export interface ArchilPlaceOptions {
175
205
  */
176
206
  signalFailMs?: number;
177
207
  }
208
+ /** The listing `recordHome` reads the run's recordings with: S3 ListObjects, each object's size when the API gives it. */
209
+ export interface RecordControl {
210
+ listObjects(prefix: string, options?: {
211
+ recursive?: boolean;
212
+ }): Promise<{
213
+ objects: {
214
+ key: string;
215
+ size?: number;
216
+ }[];
217
+ }>;
218
+ }
219
+ /**
220
+ * Every byte of `bytes` to `fd`: a write may take fewer bytes than it was given, so it goes on from where the last one
221
+ * stopped. A write that takes none fails, as one that throws (a full disk) does.
222
+ */
223
+ export declare function writeAll(fd: number, bytes: Uint8Array, write?: (fd: number, bytes: Uint8Array, offset: number, length: number) => number): void;
178
224
  /**
179
225
  * The run on an Archil disk, its instance run by a host driver. `token(attempt)` mints the attempt's mount token through
180
226
  * the place's control, and the runner it is handed to mounts with it. A start is `ensureRunning` (it creates the run's
package/dist/place.js CHANGED
@@ -5,8 +5,9 @@
5
5
  // place seam (examples/reference/place.ts) structurally. The supervisor reads nothing under the run's directory: the
6
6
  // outcome, the run's record and its delegations come through the place.
7
7
  import { spawn } from "node:child_process";
8
- import { existsSync, readFileSync } from "node:fs";
9
- import { extname, join, posix, resolve } from "node:path";
8
+ import { createHash, randomUUID } from "node:crypto";
9
+ import { closeSync, existsSync, fsyncSync, mkdirSync, openSync, readdirSync, readFileSync, renameSync, rmSync, statSync, writeSync } from "node:fs";
10
+ import { dirname, extname, join, posix, resolve } from "node:path";
10
11
  import { fileURLToPath } from "node:url";
11
12
  import { DEFAULT_MOUNT_ROOT, findDelegations, mintMountToken, pathlessResolver, removeMountToken, runPath } from "./claim.js";
12
13
  import { reapCommands } from "./commands.js";
@@ -48,6 +49,7 @@ function outcomeOf(record) {
48
49
  return undefined;
49
50
  const detail = record.detail !== null && typeof record.detail === "object" && !Array.isArray(record.detail) ? record.detail : {};
50
51
  const delivered = record.status === "done" && "record" in detail && detail.record !== null && detail.record !== undefined;
52
+ const models = modelsOf(detail.models);
51
53
  return {
52
54
  status: delivered ? "complete" : "failed",
53
55
  generation: record.generation,
@@ -55,8 +57,22 @@ function outcomeOf(record) {
55
57
  ...(typeof detail.message === "string" ? { message: detail.message } : {}),
56
58
  ...("record" in detail ? { record: detail.record } : {}),
57
59
  ...(typeof detail.storeSha256 === "string" ? { storeSha256: detail.storeSha256 } : {}),
60
+ ...(models ? { models } : {}),
58
61
  };
59
62
  }
63
+ /**
64
+ * The role-to-model ids an outcome carries: a copy of `value` when it is a plain object whose values are all strings,
65
+ * else undefined, so a malformed report is left out and never stands in as `{}`.
66
+ */
67
+ function modelsOf(value) {
68
+ if (value === null || typeof value !== "object" || Array.isArray(value))
69
+ return undefined;
70
+ const proto = Object.getPrototypeOf(value);
71
+ if (proto !== Object.prototype && proto !== null)
72
+ return undefined;
73
+ const entries = Object.entries(value);
74
+ return entries.every(([, id]) => typeof id === "string") ? Object.fromEntries(entries) : undefined;
75
+ }
60
76
  const pathsUnder = (id, root) => ({ id, root, work: join(root, "work"), store: join(root, "store") });
61
77
  /**
62
78
  * The run on a directory of this machine: each runner is `pi-durable-disk run --dir <root> --id <id> --app <app>`, a
@@ -281,13 +297,20 @@ function requestIn(text) {
281
297
  return undefined;
282
298
  }
283
299
  }
284
- /** The `outcome` of a `{"outcome":{...}}` line with at least a status; undefined for any other line. */
300
+ /**
301
+ * The `outcome` of a `{"outcome":{...}}` line with at least a status; undefined for any other line. Its `models` stays
302
+ * only when it is a plain object of strings (`modelsOf`), as `run.json`'s does.
303
+ */
285
304
  function outcomeLine(line) {
286
305
  if (!line.startsWith("{"))
287
306
  return undefined;
288
307
  try {
289
308
  const parsed = JSON.parse(line);
290
- return parsed.outcome && typeof parsed.outcome === "object" && typeof parsed.outcome.status === "string" ? parsed.outcome : undefined;
309
+ if (!parsed.outcome || typeof parsed.outcome !== "object" || typeof parsed.outcome.status !== "string")
310
+ return undefined;
311
+ const { models: reported, ...outcome } = parsed.outcome;
312
+ const models = modelsOf(reported);
313
+ return models ? { ...outcome, models } : outcome;
291
314
  }
292
315
  catch {
293
316
  return undefined;
@@ -295,11 +318,263 @@ function outcomeLine(line) {
295
318
  }
296
319
  /** Where in the run's directory an Archil place's runners write their kill requests. */
297
320
  const REQUESTS = "signals";
298
- const requestName = (attempt) => {
321
+ /** Where in the run's directory an Archil place's runners record, one directory per attempt. */
322
+ const RECORDS = "record";
323
+ const BLOB_NAME = /^([0-9a-f]{64})\.[A-Za-z0-9]{1,16}$/;
324
+ const SPOOL_NAME = /^[A-Za-z0-9][A-Za-z0-9._-]{0,200}\.jsonl$/;
325
+ const NEWLINE = 0x0a;
326
+ /** A message for a recording's line: at most 240 characters, so it never carries more than the line can hold. */
327
+ const said = (text) => text.slice(0, MESSAGE_KEEP);
328
+ /**
329
+ * Every byte of `bytes` to `fd`: a write may take fewer bytes than it was given, so it goes on from where the last one
330
+ * stopped. A write that takes none fails, as one that throws (a full disk) does.
331
+ */
332
+ export function writeAll(fd, bytes, write = writeSync) {
333
+ for (let at = 0; at < bytes.length;) {
334
+ const n = write(fd, bytes, at, bytes.length - at);
335
+ if (!(n > 0))
336
+ throw new PlaceError("CARRY_FAILED", `a write took no bytes, ${bytes.length - at} of ${bytes.length} left`);
337
+ at += n;
338
+ }
339
+ }
340
+ /** `bytes` into `file` whole: a temporary file, fsync, rename, then the directory's fsync. */
341
+ function writeWhole(file, bytes) {
342
+ const dir = dirname(file);
343
+ mkdirSync(dir, { recursive: true });
344
+ const tmp = join(dir, `.${randomUUID()}.tmp`);
345
+ const fd = openSync(tmp, "wx", 0o644);
346
+ try {
347
+ writeAll(fd, bytes);
348
+ fsyncSync(fd);
349
+ }
350
+ catch (error) {
351
+ closeSync(fd);
352
+ rmSync(tmp, { force: true });
353
+ throw error;
354
+ }
355
+ closeSync(fd);
356
+ renameSync(tmp, file);
357
+ syncDir(dir);
358
+ }
359
+ /** `bytes` at the end of `file`, fsynced; a file it creates is made durable in its directory too. */
360
+ function appendDurably(file, bytes) {
361
+ const dir = dirname(file);
362
+ const created = !existsSync(file);
363
+ mkdirSync(dir, { recursive: true });
364
+ const fd = openSync(file, "a", 0o644);
365
+ try {
366
+ writeAll(fd, bytes);
367
+ fsyncSync(fd);
368
+ }
369
+ finally {
370
+ closeSync(fd);
371
+ }
372
+ if (created)
373
+ syncDir(dir);
374
+ }
375
+ /** How long a carrier waits for another one appending to the same destination file. */
376
+ const CARRY_LOCK_WAIT_MS = 5_000;
377
+ /** A lock file with no holder written yet (its taker is between the create and the write) counts as held this long. */
378
+ const CARRY_LOCK_UNNAMED_MS = 10_000;
379
+ const alive = (pid) => {
380
+ try {
381
+ process.kill(pid, 0);
382
+ return true;
383
+ }
384
+ catch (error) {
385
+ return error.code === "EPERM";
386
+ }
387
+ };
388
+ /**
389
+ * `fn` while holding `lock`, a file created exclusively that names its holder's pid, so two carriers on one destination
390
+ * (two places, two processes) never append at once. A lock whose holder is gone (its process ended) is taken over; one a
391
+ * live holder keeps past CARRY_LOCK_WAIT_MS is CARRY_FAILED. `fn` is synchronous, so the lock is held only for it.
392
+ */
393
+ async function withCarryLock(lock, fn) {
394
+ for (const t0 = Date.now();;) {
395
+ let fd;
396
+ try {
397
+ fd = openSync(lock, "wx", 0o644);
398
+ }
399
+ catch (error) {
400
+ if (error.code !== "EEXIST")
401
+ throw error;
402
+ }
403
+ if (fd !== undefined) {
404
+ // Once created, the lock goes on every way out: a failed write of the pid or close of the file included.
405
+ try {
406
+ try {
407
+ writeAll(fd, new TextEncoder().encode(String(process.pid)));
408
+ }
409
+ finally {
410
+ closeSync(fd);
411
+ }
412
+ fn();
413
+ }
414
+ finally {
415
+ rmSync(lock, { force: true });
416
+ }
417
+ return;
418
+ }
419
+ let holder = 0;
420
+ let ageMs = 0;
421
+ try {
422
+ holder = Number(readFileSync(lock, "utf8").trim());
423
+ ageMs = Date.now() - statSync(lock).mtimeMs;
424
+ }
425
+ catch (error) {
426
+ // Released meanwhile: take it on the next turn. Any other failure to read it (its mode, the disk) is the carry's.
427
+ if (error.code === "ENOENT" && Date.now() - t0 <= CARRY_LOCK_WAIT_MS)
428
+ continue;
429
+ throw new PlaceError("CARRY_FAILED", said(`reading the carry lock ${lock} failed: ${error?.message ?? error}`), { cause: error });
430
+ }
431
+ const gone = Number.isSafeInteger(holder) && holder > 0 ? !alive(holder) : ageMs > CARRY_LOCK_UNNAMED_MS;
432
+ if (gone) {
433
+ rmSync(lock, { force: true });
434
+ continue;
435
+ }
436
+ if (Date.now() - t0 > CARRY_LOCK_WAIT_MS)
437
+ throw new PlaceError("CARRY_FAILED", said(`another carrier (pid ${holder || "unnamed"}) holds ${lock} past ${CARRY_LOCK_WAIT_MS} ms`));
438
+ await new Promise((resolve) => setTimeout(resolve, 20));
439
+ }
440
+ }
441
+ function syncDir(dir) {
442
+ const fd = openSync(dir, "r");
443
+ try {
444
+ fsyncSync(fd);
445
+ }
446
+ finally {
447
+ closeSync(fd);
448
+ }
449
+ }
450
+ /** The blob hashes (SHA-256, hex) a recording's line names in its `blobs`; none for a line that is not JSON or names none. */
451
+ function blobsOf(line) {
452
+ try {
453
+ const v = JSON.parse(new TextDecoder().decode(line));
454
+ return Array.isArray(v?.blobs) ? v.blobs.filter((b) => typeof b === "string") : [];
455
+ }
456
+ catch {
457
+ return [];
458
+ }
459
+ }
460
+ /**
461
+ * Brings home into `into` what the runners recorded under `prefix` (`runs/<id>/record/`), read through the control (S3
462
+ * ListObjects and GetObject: nothing takes the run's claim). It keeps no state: a blob is home when `into/blobs/<name>`
463
+ * exists, and a spool file's carried bytes are what `into/.spool/<name>` holds, so a second call, or a call from another
464
+ * process, carries nothing twice. Carriers that overlap on one destination never append at once: the place runs one call
465
+ * at a time per destination, and every append happens under a lock file beside its spool file (`.<name>.lock`, naming
466
+ * its holder's pid), where the destination's size is read again so only the lines past it are appended. A lock whose
467
+ * holder is gone is taken over; one held past 5 s is CARRY_FAILED. Every write is carried to its last byte.
468
+ * - Blobs first: each one not yet home is fetched, checked against the hash its name carries, and written whole into
469
+ * `into/blobs`. One whose bytes do not match is skipped; everything else is brought home, then the call throws
470
+ * BLOB_MISMATCH, and the next call tries that blob again.
471
+ * - Then lines: the new whole lines of each `.spool/<name>.jsonl`, the recorder's segments (`<stream>.<n>.jsonl`), each
472
+ * with its own offset: what `into/.spool/<name>` already holds. A file is fetched only when it grew since this place
473
+ * last fetched it (the SDK has no ranged read). A line whose blobs are not home yet waits for the next call, so no line arrives before its blob; on
474
+ * `final` it is carried anyway, and its missing blobs are CARRY_FAILED once everything else is home.
475
+ * - A listing or fetch that fails is CARRY_FAILED. The recording is on the run's disk, not in the box, so a lost box
476
+ * costs only what it had not fsynced: this place never throws BOX_GONE.
477
+ */
478
+ async function carryHome(control, prefix, into, final, fetched) {
479
+ const fetch = async (key) => {
480
+ try {
481
+ return await control.getObject(key);
482
+ }
483
+ catch (error) {
484
+ throw new PlaceError("CARRY_FAILED", said(`reading ${key.slice(prefix.length)} of the run's recording failed: ${error?.message ?? error}`), { cause: error });
485
+ }
486
+ };
487
+ let objects;
488
+ try {
489
+ objects = (await control.listObjects(prefix, { recursive: true })).objects;
490
+ }
491
+ catch (error) {
492
+ throw new PlaceError("CARRY_FAILED", said(`listing the run's recording failed: ${error?.message ?? error}`), { cause: error });
493
+ }
494
+ // `<attempt>/blobs/<name>` and `<attempt>/.spool/<name>.jsonl`; anything else (`started`, a box's temporary file) is not carried.
495
+ const parts = (key) => key.slice(prefix.length).split("/");
496
+ const blobs = objects.filter((o) => {
497
+ const p = parts(o.key);
498
+ return p.length === 3 && p[1] === "blobs" && BLOB_NAME.test(p[2]);
499
+ });
500
+ const spools = objects.filter((o) => {
501
+ const p = parts(o.key);
502
+ return p.length === 3 && p[1] === ".spool" && SPOOL_NAME.test(p[2]);
503
+ });
504
+ const home = (name) => existsSync(join(into, "blobs", name));
505
+ const mismatched = [];
506
+ for (const blob of blobs) {
507
+ const name = parts(blob.key)[2];
508
+ if (home(name))
509
+ continue;
510
+ const bytes = await fetch(blob.key);
511
+ if (createHash("sha256").update(bytes).digest("hex") !== BLOB_NAME.exec(name)[1]) {
512
+ mismatched.push(name);
513
+ continue;
514
+ }
515
+ writeWhole(join(into, "blobs", name), bytes);
516
+ }
517
+ // A line names its blobs by hash; the file is `<hash>.<ext>`.
518
+ const homeHashes = new Set(existsSync(join(into, "blobs")) ? readdirSync(join(into, "blobs")).map((n) => BLOB_NAME.exec(n)?.[1]).filter((h) => h !== undefined) : []);
519
+ const waiting = new Set();
520
+ for (const spool of spools) {
521
+ const name = parts(spool.key)[2];
522
+ const dest = join(into, ".spool", name);
523
+ const carried = existsSync(dest) ? statSync(dest).size : 0;
524
+ // Not fetched again while its size is what this place last fetched and nothing in it waited for a blob (a partial
525
+ // last line keeps it bigger than what is home). Never skipped on `final`. Only a saving: what is home is still what
526
+ // the destination holds.
527
+ // Per destination: what this place fetched for one `into` says nothing of another.
528
+ const cacheKey = `${resolve(into)}\0${spool.key}`;
529
+ if (spool.size !== undefined && (spool.size <= carried || (!final && fetched.get(cacheKey) === spool.size)))
530
+ continue;
531
+ const bytes = await fetch(spool.key);
532
+ if (bytes.length <= carried)
533
+ continue;
534
+ let heldForBlob = false;
535
+ let end = carried;
536
+ for (let at = carried; at < bytes.length;) {
537
+ const nl = bytes.indexOf(NEWLINE, at);
538
+ if (nl < 0)
539
+ break; // a line still being written
540
+ const missing = blobsOf(bytes.subarray(at, nl)).filter((hash) => !homeHashes.has(hash));
541
+ if (missing.length > 0 && !final) {
542
+ heldForBlob = true;
543
+ break;
544
+ }
545
+ for (const b of missing)
546
+ waiting.add(b);
547
+ end = nl + 1;
548
+ at = nl + 1;
549
+ }
550
+ if (end > carried) {
551
+ // Another carrier on this destination (another place, another process) may have appended since `carried` was read.
552
+ // Under the file's lock, what the destination holds now decides: only the lines past it are appended.
553
+ mkdirSync(dirname(dest), { recursive: true });
554
+ await withCarryLock(join(dirname(dest), `.${name}.lock`), () => {
555
+ const now = existsSync(dest) ? statSync(dest).size : 0;
556
+ if (end > now)
557
+ appendDurably(dest, bytes.subarray(now, end));
558
+ });
559
+ }
560
+ if (heldForBlob)
561
+ fetched.delete(cacheKey);
562
+ else
563
+ fetched.set(cacheKey, bytes.length);
564
+ }
565
+ if (mismatched.length > 0) {
566
+ throw new PlaceError("BLOB_MISMATCH", said(`${mismatched.length} blob(s) of the run's recording do not hash to their names: ${mismatched.join(", ")}`));
567
+ }
568
+ if (final && waiting.size > 0) {
569
+ throw new PlaceError("CARRY_FAILED", said(`${waiting.size} blob(s) the run's recording names never reached its disk: ${[...waiting].join(", ")}`));
570
+ }
571
+ }
572
+ const countFrom1 = (attempt) => {
299
573
  if (!Number.isSafeInteger(attempt) || attempt < 1)
300
574
  throw new PlaceError("INVALID_ARGUMENT", `attempt ${attempt} is not a count from 1`);
301
- return `kill-${attempt}.json`;
575
+ return attempt;
302
576
  };
577
+ const requestName = (attempt) => `kill-${countFrom1(attempt)}.json`;
303
578
  /**
304
579
  * The run on an Archil disk, its instance run by a host driver. `token(attempt)` mints the attempt's mount token through
305
580
  * the place's control, and the runner it is handed to mounts with it. A start is `ensureRunning` (it creates the run's
@@ -330,8 +605,40 @@ export function archilPlace(options) {
330
605
  const paths = { id: ref.id, root, work: posix.join(root, "work"), store: posix.join(root, "store") };
331
606
  const minted = new Map();
332
607
  const handles = [];
608
+ /** Every handle the place started, released or not: the boxes it reports. */
609
+ const started = [];
333
610
  const killed = new Map();
334
611
  let resolver = pathlessResolver(control);
612
+ const records = `${runPath(ref.id)}/${RECORDS}/`;
613
+ const recording = typeof options.control.listObjects === "function";
614
+ const owner = options.ensure?.create;
615
+ /**
616
+ * Before attempt `gen`'s runner starts: the EMPTY directories it writes into, `signals/` and, with `recordHome`,
617
+ * `record/<gen>/.spool/` and `record/<gen>/blobs/`, made over the control (PutObject with the owner, as createRunDir
618
+ * does: an exec mkdir would make them root's) and owned by the run user (`ensure.create`), in ensureRunning's window
619
+ * where no delegation holds the run (a held run refuses them: DIRS_REFUSED). A directory a process
620
+ * made by path on an Archil mount was seen to refuse file creates afterwards (ENOENT), and one made over the control
621
+ * does not. Every file in them stays the runner's (its kill request, its started, its spool), and a runner that
622
+ * makes them itself finds them made. Without an owner nothing is made: a directory the run user does not own would
623
+ * refuse the runner's files.
624
+ */
625
+ const makeDirs = async (gen) => {
626
+ if (!owner)
627
+ return;
628
+ const run = `${runPath(ref.id)}/`;
629
+ const dirs = [`${run}${REQUESTS}/`, ...(recording ? [`${run}${RECORDS}/`, `${run}${RECORDS}/${gen}/`, `${run}${RECORDS}/${gen}/.spool/`, `${run}${RECORDS}/${gen}/blobs/`] : [])];
630
+ for (const dir of dirs) {
631
+ try {
632
+ if (!(await control.headObject(dir)))
633
+ await control.putObject(dir, "", { uid: owner.uid, gid: owner.gid, mode: owner.mode ?? 0o755 });
634
+ }
635
+ catch (error) {
636
+ throw new PlaceError("DIRS_REFUSED", said(`making ${dir.slice(run.length)} in the run over the control was refused: ${error?.message ?? error}`), { cause: error });
637
+ }
638
+ }
639
+ };
640
+ const fetchedSizes = new Map();
641
+ const carrying = new Map();
335
642
  const handleOf = new Map();
336
643
  const mint = async (attempt) => {
337
644
  const t = await mintMountToken(control, { run: { id: ref.id, attempt }, prefix: options.ensure?.tokenPrefix, ttl: options.ensure?.tokenTtl ?? TOKEN_TTL });
@@ -385,7 +692,11 @@ export function archilPlace(options) {
385
692
  const m = given && !given.removed ? given : await mint(order.gen);
386
693
  // The driver gets the order's environment with each start; everything else is the host's own.
387
694
  const driver = {
388
- start: (r, token, attempt) => host.start(r, token, { attempt: attempt?.attempt ?? order.gen, env: order.env }),
695
+ // Inside ensureRunning, after its revoke and start mark: no delegation holds the run, so S3 takes the writes.
696
+ start: async (r, token, attempt) => {
697
+ await makeDirs(order.gen);
698
+ return host.start(r, token, { attempt: attempt?.attempt ?? order.gen, env: order.env });
699
+ },
389
700
  status: (h) => host.status(h),
390
701
  stop: (h) => host.stop(h),
391
702
  };
@@ -414,6 +725,7 @@ export function archilPlace(options) {
414
725
  if (result.adopted)
415
726
  await drop(m).catch(() => { });
416
727
  handles.push(result.handle);
728
+ started.push(result.handle);
417
729
  const w = watch(result.handle);
418
730
  const mark = result.startMark ? await readStartMark(control, ref.id).catch(() => null) : null;
419
731
  const runner = { epoch: mark?.generation ?? order.gen, exited: w.exited };
@@ -460,6 +772,26 @@ export function archilPlace(options) {
460
772
  paths,
461
773
  kills: host.kill && host.kills ? [host.kills] : [],
462
774
  token: async (attempt) => (await mint(attempt)).token,
775
+ ...(recording
776
+ ? {
777
+ recordHome: {
778
+ // In the run's directory, so the runner's delegation covers it and deleteRunTree removes it.
779
+ dir: (attempt) => posix.join(root, RECORDS, String(countFrom1(attempt))),
780
+ // One carry at a time per destination in this place; another place or process is held off by the file lock.
781
+ bringHome: (into, carry = {}) => {
782
+ const key = resolve(into);
783
+ const run = (carrying.get(key) ?? Promise.resolve()).catch(() => { }).then(() => carryHome(control, records, into, carry.final === true, fetchedSizes));
784
+ const settled = run.then(() => { }, () => { });
785
+ carrying.set(key, settled);
786
+ void settled.then(() => {
787
+ if (carrying.get(key) === settled)
788
+ carrying.delete(key);
789
+ });
790
+ return run;
791
+ },
792
+ },
793
+ }
794
+ : {}),
463
795
  // The request goes in the run's directory, outside work/ where the agent's tools are confined, and the place reads
464
796
  // it over S3: a runner on a box of its own cannot reach the supervisor's files.
465
797
  requestFile: (attempt) => posix.join(root, REQUESTS, requestName(attempt)),
@@ -491,7 +823,8 @@ export function archilPlace(options) {
491
823
  throw error;
492
824
  }),
493
825
  ]);
494
- return { record, delegations: delegations.map(({ clientId, inodeId, path, isOrphaned, isPending }) => ({ clientId, inodeId, path, isOrphaned, isPending })) };
826
+ const boxes = host.box ? started.map((h) => host.box(h)).filter((b) => b !== null) : undefined;
827
+ return { record, delegations: delegations.map(({ clientId, inodeId, path, isOrphaned, isPending }) => ({ clientId, inodeId, path, isOrphaned, isPending })), ...(boxes ? { boxes } : {}) };
495
828
  },
496
829
  recover: async (last, kind, order) => {
497
830
  if (killed.has(last))
package/dist/run.js CHANGED
@@ -15,7 +15,7 @@ import { archilEnv } from "./env.js";
15
15
  import { EXIT_FENCED, FencedError, HeldError, ownWriteFenced, PdaError } from "./errors.js";
16
16
  import { openArchilStore } from "./store.js";
17
17
  import { LeaseWatchdog } from "./watchdog.js";
18
- import { currentHolder, DEFAULT_HEARTBEAT_MS, DEFAULT_LEASE_EXPIRY_MS, DEFAULT_LEASE_MARGIN_MS, openClaimDir, persistRecord, readRunRecord, RUN_JSON, RunRecordError, writeRunRecord, } from "./status.js";
18
+ import { currentHolder, DEFAULT_HEARTBEAT_MS, DEFAULT_LEASE_EXPIRY_MS, DEFAULT_LEASE_MARGIN_MS, keepStartMark, openClaimDir, persistRecord, readRunRecord, RUN_JSON, RunRecordError, writeRunRecord, } from "./status.js";
19
19
  /** The same-host owner lock under the run's root. */
20
20
  export const OWNER_LOCK = "owner.lock";
21
21
  /** pi's store under the claim's `store/` directory. */
@@ -358,7 +358,16 @@ class Lease {
358
358
  if (seed !== undefined && previous === undefined) {
359
359
  await step("seed", async () => {
360
360
  await this.#dir.assertMounted();
361
- await seed({ root: claim.root, storePath: join(claim.store, STORE_FILE), workDir: claim.work });
361
+ // While a new run seeds it has no `run.json` heartbeat, and a supervisor keeps a start in its grace only while its
362
+ // start mark is younger than the grace: the mark is kept fresh until the seed ends, through the held claim directory.
363
+ // A seeder that dies stops, and the grace runs out as for any start.
364
+ const stop = keepStartMark(this.#dir.at, this.#lease.heartbeatMs);
365
+ try {
366
+ await seed({ root: claim.root, storePath: join(claim.store, STORE_FILE), workDir: claim.work });
367
+ }
368
+ finally {
369
+ await stop();
370
+ }
362
371
  await this.#dir.assertMounted();
363
372
  await claim.barrier();
364
373
  });
package/dist/status.d.ts CHANGED
@@ -81,6 +81,27 @@ export declare const persistRecord: PersistRecord;
81
81
  * `PersistRecord`.
82
82
  */
83
83
  export declare function writeRunRecord(root: string | ClaimDir, record: RunRecord, persist?: PersistRecord, signal?: AbortSignal): Promise<void>;
84
+ /**
85
+ * The supervisor's record of its last start of a run, in the run's directory: the generation the instance will write,
86
+ * when, by whom, and the failed starts right before it (`failures`; `lastExit`, what the decision saw of the last one).
87
+ */
88
+ export declare const START_MARK = "start.json";
89
+ export type StartMark = {
90
+ generation: number;
91
+ at: string;
92
+ by: string | null;
93
+ failures: number;
94
+ lastExit: string | null;
95
+ };
96
+ /** A start mark's text as a mark, or null when it is not one (an unreadable mark gives no grace). */
97
+ export declare function parseStartMark(text: string): StartMark | null;
98
+ /**
99
+ * Keeps a supervisor's start mark for generation 1 in `dir` fresh: its `at` rewritten every `everyMs` (temp file, fsync,
100
+ * rename, fsync the directory), the rest of it as the supervisor wrote it; nothing is made when there is none. On a thread of
101
+ * its own, so a caller whose thread is blocked (a synchronous import) is kept fresh too. Best effort: a refused write leaves
102
+ * the mark as it was. Returns the stop, which resolves once the thread has exited, so no write outlives it.
103
+ */
104
+ export declare function keepStartMark(dir: string, everyMs: number): () => Promise<void>;
84
105
  /** The FencedError code of a claim root that is no longer the mount the instance opened (unmounted, lazily or not). */
85
106
  export declare const CLAIM_UNMOUNTED = "CLAIM_UNMOUNTED";
86
107
  /** The file system type of an Archil mount in the mount table. */
package/dist/status.js CHANGED
@@ -6,6 +6,7 @@ import { constants } from "node:fs";
6
6
  import { open, readFile, rename, rm, stat } from "node:fs/promises";
7
7
  import { hostname } from "node:os";
8
8
  import { join } from "node:path";
9
+ import { Worker } from "node:worker_threads";
9
10
  import { FencedError, ownWriteFenced, PdaError } from "./errors.js";
10
11
  import { thisBoot } from "./process-table.js";
11
12
  export const RUN_JSON = "run.json";
@@ -158,6 +159,84 @@ export async function writeRunRecord(root, record, persist = persistRecord, sign
158
159
  throw ownWriteFenced(join(typeof root === "string" ? root : root.root, RUN_JSON), error);
159
160
  }
160
161
  }
162
+ // ---- the supervisor's start mark --------------------------------------------------------------------------------------
163
+ /**
164
+ * The supervisor's record of its last start of a run, in the run's directory: the generation the instance will write,
165
+ * when, by whom, and the failed starts right before it (`failures`; `lastExit`, what the decision saw of the last one).
166
+ */
167
+ export const START_MARK = "start.json";
168
+ /** A start mark's text as a mark, or null when it is not one (an unreadable mark gives no grace). */
169
+ export function parseStartMark(text) {
170
+ try {
171
+ const v = JSON.parse(text);
172
+ if (!Number.isSafeInteger(v.generation) || v.generation < 1 || typeof v.at !== "string" || !Number.isFinite(Date.parse(v.at)))
173
+ return null;
174
+ const failures = Number.isSafeInteger(v.failures) && v.failures >= 0 ? v.failures : 0;
175
+ const lastExit = typeof v.lastExit === "string" ? v.lastExit : null;
176
+ return { generation: v.generation, at: v.at, by: typeof v.by === "string" ? v.by : null, failures, lastExit };
177
+ }
178
+ catch {
179
+ return null;
180
+ }
181
+ }
182
+ // The keeper's code is plain CommonJS in a string, started with `eval: true` and no inherited Node flags, as the lease
183
+ // watchdog's is (src/watchdog.ts says why). One refresh at a time, all synchronous: a stop (the message) is handled only
184
+ // between two refreshes, so none is still writing once the thread has exited.
185
+ const KEEPER_SOURCE = `"use strict";
186
+ const fs = require("node:fs");
187
+ const path = require("node:path");
188
+ const { parentPort, workerData } = require("node:worker_threads");
189
+ const { dir, name, everyMs } = workerData;
190
+ const file = path.join(dir, name);
191
+ function refresh() {
192
+ try {
193
+ const mark = JSON.parse(fs.readFileSync(file, "utf8"));
194
+ if (mark === null || typeof mark !== "object" || mark.generation !== 1 || typeof mark.at !== "string" || !Number.isFinite(Date.parse(mark.at))) return;
195
+ const temp = file + ".tmp";
196
+ const fd = fs.openSync(temp, "w", 0o644);
197
+ try {
198
+ fs.writeSync(fd, JSON.stringify({ ...mark, at: new Date().toISOString() }));
199
+ fs.fsyncSync(fd);
200
+ } finally {
201
+ fs.closeSync(fd);
202
+ }
203
+ fs.renameSync(temp, file);
204
+ const handle = fs.openSync(dir, "r");
205
+ try {
206
+ fs.fsyncSync(handle);
207
+ } finally {
208
+ fs.closeSync(handle);
209
+ }
210
+ } catch {
211
+ // absent, unreadable or refused
212
+ }
213
+ }
214
+ refresh();
215
+ const timer = setInterval(refresh, everyMs);
216
+ parentPort.once("message", () => {
217
+ clearInterval(timer);
218
+ parentPort.close();
219
+ });
220
+ `;
221
+ /**
222
+ * Keeps a supervisor's start mark for generation 1 in `dir` fresh: its `at` rewritten every `everyMs` (temp file, fsync,
223
+ * rename, fsync the directory), the rest of it as the supervisor wrote it; nothing is made when there is none. On a thread of
224
+ * its own, so a caller whose thread is blocked (a synchronous import) is kept fresh too. Best effort: a refused write leaves
225
+ * the mark as it was. Returns the stop, which resolves once the thread has exited, so no write outlives it.
226
+ */
227
+ export function keepStartMark(dir, everyMs) {
228
+ const worker = new Worker(KEEPER_SOURCE, { eval: true, workerData: { dir, name: START_MARK, everyMs }, execArgv: [] });
229
+ worker.unref();
230
+ worker.on("error", () => {
231
+ // the thread failed to start or threw: nothing more is refreshed, and the grace runs out as for any start
232
+ });
233
+ const exited = new Promise((resolve) => worker.once("exit", () => resolve()));
234
+ return () => {
235
+ worker.ref();
236
+ worker.postMessage("stop");
237
+ return exited;
238
+ };
239
+ }
161
240
  // ---- the claim directory ----------------------------------------------------------------------------------------------
162
241
  /** The FencedError code of a claim root that is no longer the mount the instance opened (unmounted, lazily or not). */
163
242
  export const CLAIM_UNMOUNTED = "CLAIM_UNMOUNTED";
@@ -1,7 +1,7 @@
1
1
  import type { Delegation } from "disk";
2
2
  import { type ArchilHost, type ControlApi, type PathlessResolver, type RunRef } from "./claim.js";
3
3
  import { PdaError } from "./errors.js";
4
- import { type RunRecord } from "./status.js";
4
+ import { type RunRecord, type StartMark } from "./status.js";
5
5
  export type Json = null | boolean | number | string | Json[] | {
6
6
  [key: string]: Json;
7
7
  };
@@ -21,6 +21,22 @@ export type StartAttempt = {
21
21
  attempt: number;
22
22
  env?: Record<string, string>;
23
23
  };
24
+ /**
25
+ * A box a driver made for an instance, as a receipt: who provided it, what kind, when it was made and when it ended
26
+ * (null while it lives), and its published list price per second (`usdPerSecond`), with the page that price was read
27
+ * from (`source`) and the ISO date it was read (`priced`). A box with no price (a kind the provider's page does not
28
+ * price, or of unknown size) has none of the three: never a price of 0. No field holds the provider's id of the box:
29
+ * the supervisor labels boxes itself and computes their seconds and cost.
30
+ */
31
+ export type HostBox = {
32
+ provider: string;
33
+ kind: string;
34
+ from: string;
35
+ to: string | null;
36
+ usdPerSecond?: number;
37
+ source?: string;
38
+ priced?: string;
39
+ };
24
40
  /** Compute is disposable and pluggable: a driver is these three calls. */
25
41
  export interface HostDriver {
26
42
  /**
@@ -41,6 +57,8 @@ export interface HostDriver {
41
57
  kill?(handle: HostHandle): Promise<void>;
42
58
  /** What `kill` takes: `process`, the instance's processes while its machine stays (a restart policy may bring them back), or `box`, its machine. */
43
59
  readonly kills?: "process" | "box";
60
+ /** Optional, for a driver whose instances get machines of their own: the box `handle`'s instance runs in, as a receipt. */
61
+ box?(handle: HostHandle): HostBox | null;
44
62
  }
45
63
  /** The control-plane calls the supervisor uses; an archil `Disk` satisfies it. */
46
64
  export interface SupervisorControl extends ControlApi {
@@ -227,18 +245,7 @@ export type EnsureResult =
227
245
  * supervisors are safe: the mount admits one claimant and the other instance exits 76.
228
246
  */
229
247
  export declare function ensureRunning(ref: RunRef, host: HostDriver, options: EnsureOptions): Promise<EnsureResult>;
230
- /**
231
- * The supervisor's record of its last start of a run: the generation the instance will write, when, by whom, and the
232
- * failed starts right before it (`failures`; `lastExit`, what the decision saw of the last one).
233
- */
234
- export declare const START_MARK = "start.json";
235
- export type StartMark = {
236
- generation: number;
237
- at: string;
238
- by: string | null;
239
- failures: number;
240
- lastExit: string | null;
241
- };
248
+ export { START_MARK, type StartMark } from "./status.js";
242
249
  /** The grace of a start that follows `failures` failed ones: the base doubled per failure, capped, never below the base. */
243
250
  export declare function startGrace(baseMs: number, failures: number, maxMs?: number): number;
244
251
  /**
package/dist/supervise.js CHANGED
@@ -8,7 +8,7 @@ import { hostname } from "node:os";
8
8
  import { join } from "node:path";
9
9
  import { acquire, createRunDir, findDelegations, matchDelegations, pathlessResolver, revokeBestEffort, revokeCompanions, mintMountToken, MOUNT_TOKEN_TTL, parseTokenNickname, removeMountToken, runPath, unmountClaim, } from "./claim.js";
10
10
  import { FencedError, HeldError, PdaError } from "./errors.js";
11
- import { parseRunRecord, RUN_JSON } from "./status.js";
11
+ import { parseRunRecord, parseStartMark, RUN_JSON, START_MARK } from "./status.js";
12
12
  export const CONTROL_TIMEOUT_MS = 10_000;
13
13
  /** The same control API with every call bounded: one that does not settle in `ms` fails as CONTROL_API_FAILED. */
14
14
  export function withTimeouts(control, ms) {
@@ -160,11 +160,8 @@ export async function ensureRunning(ref, host, options) {
160
160
  await revokeListed(opts.control, ref.id, held, all);
161
161
  return start("lease-expired", held, stonith);
162
162
  }
163
- /**
164
- * The supervisor's record of its last start of a run: the generation the instance will write, when, by whom, and the
165
- * failed starts right before it (`failures`; `lastExit`, what the decision saw of the last one).
166
- */
167
- export const START_MARK = "start.json";
163
+ // The start mark (`start.json`): its name, shape and parser live in status.ts, which the instance also reads it with.
164
+ export { START_MARK } from "./status.js";
168
165
  /** The grace of a start that follows `failures` failed ones: the base doubled per failure, capped, never below the base. */
169
166
  export function startGrace(baseMs, failures, maxMs = START_BACKOFF_MAX_MS) {
170
167
  if (baseMs <= 0)
@@ -193,17 +190,7 @@ export async function readStartMark(control, id) {
193
190
  return null;
194
191
  throw new SuperviseError("CONTROL_API_FAILED", `GetObject ${runPath(id)}/${START_MARK} failed`, { cause: err });
195
192
  }
196
- try {
197
- const v = JSON.parse(text);
198
- if (!Number.isSafeInteger(v.generation) || v.generation < 1 || typeof v.at !== "string" || !Number.isFinite(Date.parse(v.at)))
199
- return null;
200
- const failures = Number.isSafeInteger(v.failures) && v.failures >= 0 ? v.failures : 0;
201
- const lastExit = typeof v.lastExit === "string" ? v.lastExit : null;
202
- return { generation: v.generation, at: v.at, by: typeof v.by === "string" ? v.by : null, failures, lastExit };
203
- }
204
- catch {
205
- return null;
206
- }
193
+ return parseStartMark(text);
207
194
  }
208
195
  /** Only a run with no run.json and no holder gets here, so the directory is either absent or idle. */
209
196
  async function ensureRunDir(control, id, owner) {
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@parcha/pi-durable-disk",
3
- "version": "0.1.0-beta.15",
3
+ "version": "0.1.0-beta.16",
4
4
  "description": "A durable pi host on a network disk (Archil): one owner per run, a fence, durable commits, and a supervisor.",
5
5
  "keywords": [
6
6
  "pi",
@@ -56,7 +56,7 @@
56
56
  "test": "node --test --test-timeout=240000 'test/*.test.ts' 'test/examples/*.test.ts'",
57
57
  "test:acceptance": "node --test --test-concurrency=1 'test/acceptance/*.test.ts'",
58
58
  "test:live": "node --test --test-concurrency=1 'test/live/*.test.ts'",
59
- "test:live:claim": "node --test --test-concurrency=1 test/live/claim.live.test.ts",
59
+ "test:live:claim": "node --test --test-concurrency=1 test/live/claim.live.test.ts test/live/record-home.live.test.ts",
60
60
  "test:live:env": "node --test --test-concurrency=1 test/live/env.live.test.ts",
61
61
  "test:live:lifecycle": "node --test --test-concurrency=1 test/live/lifecycle.live.test.ts",
62
62
  "test:live:run": "node --test --test-concurrency=1 test/live/run.live.test.ts",