@parcha/pi-durable-disk 0.1.0-beta.23 → 0.1.0-beta.26
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +14 -0
- package/dist/claim.d.ts +1 -1
- package/dist/hosts/attempt-env.d.ts +9 -0
- package/dist/hosts/attempt-env.js +12 -1
- package/dist/hosts/daytona.d.ts +1 -1
- package/dist/hosts/daytona.js +2 -0
- package/dist/hosts/docker.d.ts +1 -1
- package/dist/hosts/docker.js +1 -0
- package/dist/hosts/local-host.d.ts +1 -1
- package/dist/hosts/local-host.js +6 -0
- package/dist/index.d.ts +3 -1
- package/dist/index.js +2 -0
- package/dist/place.d.ts +64 -1
- package/dist/place.js +133 -4
- package/dist/run-files.d.ts +72 -0
- package/dist/run-files.js +259 -0
- package/dist/supervise.d.ts +12 -0
- package/package.json +2 -2
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,20 @@
|
|
|
2
2
|
|
|
3
3
|
## Unreleased
|
|
4
4
|
|
|
5
|
+
## 0.1.0-beta.26 (2026-10-24)
|
|
6
|
+
|
|
7
|
+
- Pin the Docker image, planned-move example and live-test sandbox setup to Archil client 0.10.0, with SHA-256 checks for each downloaded package. The client verifies server certificates against the operating system trust store and rejects certificates from the compromised eMudhra intermediate authorities. Linux hosts need client 0.10.0 or later and the operating system CA bundle installed (`ca-certificates` on Debian/Ubuntu); rebuild Docker images and sandbox snapshots to use the new client.
|
|
8
|
+
- Archil's latest SDK: the package depends on the `disk` SDK 1.9.0 (was 1.7.0). The SDK calls the claim, lease and supervisor make are the same in 1.9.0, which adds disk branches and sandbox mounts. The client verbs the package runs take the same arguments in client 0.10.0, whose `mount` adds `--kernel-read-ahead-kb`. A host that installs `disk` itself installs 1.9.0, so npm keeps one copy.
|
|
9
|
+
- `archilPlace({ unheld })` takes a hook that runs in each start's window while nobody holds the run: after ensureRunning's revoke and start mark and after the run's directories are made, before the driver starts anything, once per start. It gets the start's `attempt` and, when the place hands labels over, exactly the `HOST_BOX` that start carries (`start.box`). It also gets the run's own files: `read`, `list`, and `put(path, text)`, which creates a file the run user owns (0644, parents 0755) and refuses a path already there with `EXISTS`. What it returns is added to that start's environment. It needs `ensure.create` and a control that lists. New types `UnheldHook`, `UnheldRun` and `UnheldStart`, and new `unheldTimeoutMs` (default 30 s).
|
|
10
|
+
- A put the hook does not await still settles inside the window: the driver starts only once every put has settled, within the hook's bound.
|
|
11
|
+
- `put` is a check, then a write, not a conditional create, so it is not exclusive between two supervisors in one window: put names unique to the start, and bind a shared name in the process that holds the run. It is for non-secret run files; a secret never goes through it.
|
|
12
|
+
- A start fails, and the driver starts no box, when a put was refused because the run is held (`WINDOW_CLOSED`, even if the hook caught it), when the hook throws a control failure (`CONTROL_API_FAILED`), when it throws anything else or overruns its bound (`UNHELD_FAILED`), or when its environment has a name the start already sets, a name that is not a variable name, a key-shaped name (a `KEY`, `TOKEN`, `SECRET`, `PASSWORD`, `PASSWD`, `CREDENTIAL` or `CREDENTIALS` segment), or a value that is not a string without NUL (`INVALID_ARGUMENT`, naming the variable, never its value). The caller sees `HOST_START_FAILED` with that `PlaceError` as its cause. `RunFileErrorCode` gains `EXISTS` and `WINDOW_CLOSED`; `PlaceErrorCode` gains `WINDOW_CLOSED`, `CONTROL_API_FAILED` and `UNHELD_FAILED`.
|
|
13
|
+
- Fix to merged behavior: a box the place already started and its driver adopts again (`adopted: true`, the same attempt started twice) keeps its key in `status().boxes`. New optional `HostDriver.instanceOf(handle)` names the instance a handle is for, the same for its fresh and its adopted handle; `dockerHost` (daemon and container name), `localHost` (host, boot, unit or tag) and `daytonaHost` (sandbox id) implement it, and without it no two handles are one box. It used to get a second entry: a key no runner claimed under `HOST_BOX`, the box counted twice in the receipts, and stopped twice at release.
|
|
14
|
+
- `readRunFile(control, ref, path)` and `listRunFiles(control, ref, dir)` read a run's own files over the disk's control (S3 GET, HEAD and LIST): they never mount, write, or take or wait on the claim, so they read a run another host holds. `readRunFile` returns the bytes, or `null` for a file not written yet; `listRunFiles` returns the names directly under a directory, sorted, names only. Refusals are `RunFileError`: `INVALID_PATH` (before any request: absolute, `.`/`..`, unsafe characters, `.archil/`), `RUN_MISSING`, `NOT_A_FILE` (a directory, or a symlink, decided on Archil's 501 for it, never from a listing, where a symlink looks like an empty file), and `CONTROL_API_FAILED`. Both take a `signal`.
|
|
15
|
+
- A place offers the same reads of its own run: `Place.readRunFile(path)` and `Place.listRunFiles(dir)`, optional members that `archilPlace` implements over its control and `localPlace` over its run directory on this machine, where a symlink anywhere on the way is `NOT_A_FILE`, never followed. A missing file is `null`; a directory not there yet lists `[]`; every other failure throws.
|
|
16
|
+
- `INVALID_PATH` names what is wrong with a path (empty, absolute, a `.` or `..` segment, an unsafe character at an index, under `.archil/`), never the path itself.
|
|
17
|
+
- `archilPlace` tells each host box's runner which label to claim: when its driver makes boxes and names their provider (new optional `HostDriver.provider`; `daytonaHost` names `daytona`) and the place can read the label back (its control lists), each start puts `HOST_BOX` (`{ provider, key, run }`: the box's index in `status().boxes`, the order's `RUN_LABEL`) in that order's environment. With it, `status()` fails (`STATUS_FAILED`) on a box its driver gives no receipt for, instead of shifting the keys of the boxes after it.
|
|
18
|
+
|
|
5
19
|
## 0.1.0-beta.23 (2026-10-22)
|
|
6
20
|
|
|
7
21
|
- The CLI's duration flags take a unit: `ms`, `s`, `m` or `h` (`500ms`, `30s`, `1.5m`, `1h`). A bare number used to mean milliseconds, so `--every 90` ran every 90 ms. It is now refused with exit 2 and a hint: `not a duration: "90": give a unit, e.g. 90s or 90ms`. Every duration flag a command is given is checked before the command acts. **Who breaks:** a script or service unit that passes a bare number to `--park-threshold`, `--park-idle`, `--drain-timeout`, `--stop-timeout`, `--control-timeout`, `--lease-expiry`, `--stonith-timeout`, `--start-grace`, `--start-backoff-max`, `--token-grace` or `--every`. **What to do:** add the unit you meant (`90ms` keeps today's behaviour).
|
package/dist/claim.d.ts
CHANGED
|
@@ -4,7 +4,7 @@ export type RunRef = {
|
|
|
4
4
|
region: string;
|
|
5
5
|
id: string;
|
|
6
6
|
};
|
|
7
|
-
/** The control-plane calls the claim uses; an archil `Disk` (SDK `disk` 1.
|
|
7
|
+
/** The control-plane calls the claim uses; an archil `Disk` (SDK `disk` 1.9.0) satisfies it. Supervisor side only. */
|
|
8
8
|
export interface ControlApi {
|
|
9
9
|
putObject(key: string, body: string, options: {
|
|
10
10
|
uid: number;
|
|
@@ -1,3 +1,12 @@
|
|
|
1
1
|
import type { StartAttempt } from "../supervise.js";
|
|
2
|
+
/** A name an environment variable can have. */
|
|
3
|
+
export declare const ENV_NAME: RegExp;
|
|
4
|
+
/**
|
|
5
|
+
* Whether `name` is a secret's name: one of its `_`-separated segments, in any case, is KEY, TOKEN, SECRET, PASSWORD,
|
|
6
|
+
* PASSWD, CREDENTIAL or CREDENTIALS. Segments, not suffixes, so MODAL_TOKEN_ID is one too; CACHE_KEY_PREFIX is refused
|
|
7
|
+
* as well, and renamed, which is the cheaper error. A start's environment is readable in the instance, so a secret never
|
|
8
|
+
* goes there.
|
|
9
|
+
*/
|
|
10
|
+
export declare function keyShaped(name: string): boolean;
|
|
2
11
|
/** The start's extra environment, checked: PATH, PDA_HOLDER, a name that is not a variable name, or a value with a NUL, is `refuse`'s error. */
|
|
3
12
|
export declare function attemptEnv(attempt: StartAttempt | undefined, refuse: (message: string) => Error): Record<string, string>;
|
|
@@ -1,5 +1,16 @@
|
|
|
1
|
-
|
|
1
|
+
/** A name an environment variable can have. */
|
|
2
|
+
export const ENV_NAME = /^[A-Za-z_][A-Za-z0-9_]*$/;
|
|
2
3
|
const DRIVER_KEYS = new Set(["PATH", "PDA_HOLDER"]);
|
|
4
|
+
const KEY_SEGMENTS = new Set(["KEY", "TOKEN", "SECRET", "PASSWORD", "PASSWD", "CREDENTIAL", "CREDENTIALS"]);
|
|
5
|
+
/**
|
|
6
|
+
* Whether `name` is a secret's name: one of its `_`-separated segments, in any case, is KEY, TOKEN, SECRET, PASSWORD,
|
|
7
|
+
* PASSWD, CREDENTIAL or CREDENTIALS. Segments, not suffixes, so MODAL_TOKEN_ID is one too; CACHE_KEY_PREFIX is refused
|
|
8
|
+
* as well, and renamed, which is the cheaper error. A start's environment is readable in the instance, so a secret never
|
|
9
|
+
* goes there.
|
|
10
|
+
*/
|
|
11
|
+
export function keyShaped(name) {
|
|
12
|
+
return name.split("_").some((segment) => KEY_SEGMENTS.has(segment.toUpperCase()));
|
|
13
|
+
}
|
|
3
14
|
/** The start's extra environment, checked: PATH, PDA_HOLDER, a name that is not a variable name, or a value with a NUL, is `refuse`'s error. */
|
|
4
15
|
export function attemptEnv(attempt, refuse) {
|
|
5
16
|
const env = attempt?.env ?? {};
|
package/dist/hosts/daytona.d.ts
CHANGED
|
@@ -158,7 +158,7 @@ export interface DaytonaHostOptions {
|
|
|
158
158
|
export declare function sandboxStatus(state: string | undefined): HostStatus | "check";
|
|
159
159
|
/** A Daytona-safe box name for a run: lowercase, at most 63 characters, unique per call. */
|
|
160
160
|
export declare function sandboxName(prefix: string, runId: string, now: number): string;
|
|
161
|
-
export declare function daytonaHost(opts: DaytonaHostOptions): HostDriver & Required<Pick<HostDriver, "kill" | "kills">> & {
|
|
161
|
+
export declare function daytonaHost(opts: DaytonaHostOptions): HostDriver & Required<Pick<HostDriver, "kill" | "kills" | "instanceOf">> & {
|
|
162
162
|
readonly fleet: string;
|
|
163
163
|
readonly mountRoot: string;
|
|
164
164
|
};
|
package/dist/hosts/daytona.js
CHANGED
|
@@ -293,6 +293,8 @@ export function daytonaHost(opts) {
|
|
|
293
293
|
fleet,
|
|
294
294
|
mountRoot,
|
|
295
295
|
kills: "box",
|
|
296
|
+
provider: "daytona",
|
|
297
|
+
instanceOf: (h) => (h.driver === "daytona" && typeof h.sandboxId === "string" ? `daytona\0${h.sandboxId}` : undefined),
|
|
296
298
|
async start(ref, mountToken, attempt) {
|
|
297
299
|
runPath(ref.id);
|
|
298
300
|
if (!mountToken)
|
package/dist/hosts/docker.d.ts
CHANGED
|
@@ -95,7 +95,7 @@ export type ContainerInfo = {
|
|
|
95
95
|
startedAt: string | null;
|
|
96
96
|
finishedAt: string | null;
|
|
97
97
|
};
|
|
98
|
-
export declare function dockerHost(opts?: DockerHostOptions): HostDriver & Required<Pick<HostDriver, "kill" | "kills">> & {
|
|
98
|
+
export declare function dockerHost(opts?: DockerHostOptions): HostDriver & Required<Pick<HostDriver, "kill" | "kills" | "instanceOf">> & {
|
|
99
99
|
readonly fleet: string;
|
|
100
100
|
readonly mountRoot: string;
|
|
101
101
|
describe(handle: HostHandle): Promise<ContainerInfo | null>;
|
package/dist/hosts/docker.js
CHANGED
|
@@ -272,6 +272,7 @@ export function dockerHost(opts = {}) {
|
|
|
272
272
|
fleet,
|
|
273
273
|
mountRoot,
|
|
274
274
|
kills: "box",
|
|
275
|
+
instanceOf: (h) => (h.driver === "docker" && typeof h.daemon === "string" && typeof h.name === "string" ? `docker\0${h.daemon}\0${h.name}` : undefined),
|
|
275
276
|
async start(ref, token, attemptInfo) {
|
|
276
277
|
runPath(ref.id);
|
|
277
278
|
if (!opts.image)
|
|
@@ -66,7 +66,7 @@ export declare function currentBootId(): string;
|
|
|
66
66
|
export declare function parseShow(text: string): Record<string, string>;
|
|
67
67
|
/** A unit's state as a host status: loaded and active (or restarting) is running; unloaded is gone. */
|
|
68
68
|
export declare function unitStatus(show: Record<string, string>): HostStatus;
|
|
69
|
-
export declare function localHost(opts?: LocalHostOptions): HostDriver & Required<Pick<HostDriver, "kill" | "kills">> & {
|
|
69
|
+
export declare function localHost(opts?: LocalHostOptions): HostDriver & Required<Pick<HostDriver, "kill" | "kills" | "instanceOf">> & {
|
|
70
70
|
readonly hostName: string;
|
|
71
71
|
readonly mountRoot: string;
|
|
72
72
|
};
|
package/dist/hosts/local-host.js
CHANGED
|
@@ -357,6 +357,12 @@ export function localHost(opts = {}) {
|
|
|
357
357
|
hostName,
|
|
358
358
|
mountRoot,
|
|
359
359
|
kills: "process",
|
|
360
|
+
instanceOf: (h) => {
|
|
361
|
+
if (h.driver !== "local" || typeof h.host !== "string")
|
|
362
|
+
return undefined;
|
|
363
|
+
const which = h.mode === "systemd" && typeof h.unit === "string" ? `unit\0${h.unit}` : h.mode === "child" && typeof h.tag === "string" ? `tag\0${h.tag}` : undefined;
|
|
364
|
+
return which === undefined ? undefined : `local\0${h.host}\0${String(h.bootId)}\0${which}`;
|
|
365
|
+
},
|
|
360
366
|
async start(ref, token, attempt) {
|
|
361
367
|
runPath(ref.id);
|
|
362
368
|
// Never as root: the instance and the agent's tools would share a uid with the archil daemon that holds the mount
|
package/dist/index.d.ts
CHANGED
|
@@ -10,6 +10,8 @@ export { acquire, ARCHIL_SCOPED, revoke, takeOver } from "./claim.js";
|
|
|
10
10
|
export type { AcquireOptions, ArchilHost, Claim, ControlApi, PathlessResolver, RunRef, Unmounted } from "./claim.js";
|
|
11
11
|
export { readReleased, withClaim } from "./with-claim.js";
|
|
12
12
|
export type { ClaimControl, HeldPath, WithClaimOptions } from "./with-claim.js";
|
|
13
|
+
export { listRunFiles, readRunFile, RunFileError } from "./run-files.js";
|
|
14
|
+
export type { RunFileControl, RunFileErrorCode } from "./run-files.js";
|
|
13
15
|
export { PathOutsideWorkError } from "./confine.js";
|
|
14
16
|
export type { RefusalReason } from "./confine.js";
|
|
15
17
|
export { ClaimError, FencedError, HeldError, PdaError } from "./errors.js";
|
|
@@ -28,7 +30,7 @@ export type { CreateSandboxBody, DaytonaClient, DaytonaHostErrorCode, DaytonaHos
|
|
|
28
30
|
export { requestRun, ServeConfigError, ServeError, serveRun } from "./serve.js";
|
|
29
31
|
export type { RequestOptions, RunResponse, RunServer, ServeConfigErrorCode, ServeErrorCode, ServeOptions } from "./serve.js";
|
|
30
32
|
export { archilPlace, localPlace, PlaceError } from "./place.js";
|
|
31
|
-
export type { ArchilPlaceOptions, LocalPlaceOptions, Place, PlaceContext, PlaceErrorCode, PlaceExit, PlaceKill, PlaceOutcome, PlacePaths, PlaceRecordHome, PlaceRecovery, PlaceRunner, PlaceSignal, PlaceStatus, RecordControl, RunnerOrder } from "./place.js";
|
|
33
|
+
export type { ArchilPlaceOptions, LocalPlaceOptions, Place, PlaceContext, PlaceErrorCode, PlaceExit, PlaceKill, PlaceOutcome, PlacePaths, PlaceRecordHome, PlaceRecovery, PlaceRunner, PlaceSignal, PlaceStatus, RecordControl, RunnerOrder, UnheldHook, UnheldRun, UnheldStart } from "./place.js";
|
|
32
34
|
export { busyState, drain, leaseParkTarget, watchParking } from "./park.js";
|
|
33
35
|
export type { BusyState, DrainOptions, Parking, ParkOptions, ParkResult, ParkTarget } from "./park.js";
|
|
34
36
|
export { fork, ForkError, forkLocal } from "./fork.js";
|
package/dist/index.js
CHANGED
|
@@ -9,6 +9,8 @@ export { ArchilCodingTools, archilEnv, EnvClosedError, EnvError, markReadOnlySaf
|
|
|
9
9
|
export { acquire, ARCHIL_SCOPED, revoke, takeOver } from "./claim.js";
|
|
10
10
|
// A short exclusive claim on any path of a disk, and a consistent S3 read of a released file (an agent home beside the runs).
|
|
11
11
|
export { readReleased, withClaim } from "./with-claim.js";
|
|
12
|
+
// A run's own files, read over the control: never a mount, a write, or a wait on the claim.
|
|
13
|
+
export { listRunFiles, readRunFile, RunFileError } from "./run-files.js";
|
|
12
14
|
export { PathOutsideWorkError } from "./confine.js";
|
|
13
15
|
export { ClaimError, FencedError, HeldError, PdaError } from "./errors.js";
|
|
14
16
|
// The app contract: what `--app` names.
|
package/dist/place.d.ts
CHANGED
|
@@ -148,8 +148,16 @@ export interface Place {
|
|
|
148
148
|
release(): Promise<void>;
|
|
149
149
|
/** A place whose runners record where the supervisor cannot read says where, and brings the recording home. */
|
|
150
150
|
readonly recordHome?: PlaceRecordHome;
|
|
151
|
+
/**
|
|
152
|
+
* The bytes of the run's own file `path` (`runs/<id>/<path>`), or null when there is no such file; every other failure
|
|
153
|
+
* throws a `RunFileError` (INVALID_PATH, RUN_MISSING, NOT_A_FILE, CONTROL_API_FAILED). Read only: never a write, never
|
|
154
|
+
* the claim. A place never parses what it reads.
|
|
155
|
+
*/
|
|
156
|
+
readRunFile?(path: string): Promise<Uint8Array | null>;
|
|
157
|
+
/** The names directly under `dir` in the run (`""` for its root), sorted; [] for a directory not there yet. A missing run, or a failed read, throws. */
|
|
158
|
+
listRunFiles?(dir: string): Promise<string[]>;
|
|
151
159
|
}
|
|
152
|
-
export type PlaceErrorCode = "INVALID_ARGUMENT" | "HELD" | "START_TIMED_OUT" | "STATUS_FAILED" | "KILL_UNSUPPORTED" | "RELEASE_FAILED" | "BOX_GONE" | "CARRY_FAILED" | "BLOB_MISMATCH" | "DIRS_REFUSED";
|
|
160
|
+
export type PlaceErrorCode = "INVALID_ARGUMENT" | "HELD" | "START_TIMED_OUT" | "STATUS_FAILED" | "KILL_UNSUPPORTED" | "RELEASE_FAILED" | "BOX_GONE" | "CARRY_FAILED" | "BLOB_MISMATCH" | "DIRS_REFUSED" | "WINDOW_CLOSED" | "CONTROL_API_FAILED" | "UNHELD_FAILED";
|
|
153
161
|
export declare class PlaceError extends PdaError {
|
|
154
162
|
constructor(code: PlaceErrorCode, message: string, options?: {
|
|
155
163
|
cause?: unknown;
|
|
@@ -203,7 +211,62 @@ export interface ArchilPlaceOptions {
|
|
|
203
211
|
* `signals-failed` and ends. Default 60 s.
|
|
204
212
|
*/
|
|
205
213
|
signalFailMs?: number;
|
|
214
|
+
/**
|
|
215
|
+
* A hook in each start's window while nobody holds the run: after ensureRunning's revoke and start mark and after the
|
|
216
|
+
* run's directories are made, before the driver starts anything. It runs once per start, a takeover's included, and
|
|
217
|
+
* what it returns is added to that start's environment. Needs `ensure.create` (what it puts is the run user's) and a
|
|
218
|
+
* control that lists. See `UnheldHook`.
|
|
219
|
+
*/
|
|
220
|
+
unheld?: UnheldHook;
|
|
221
|
+
/** How long `unheld` may take. Default 30 s; past it the hook's signal aborts and the start fails (UNHELD_FAILED). */
|
|
222
|
+
unheldTimeoutMs?: number;
|
|
206
223
|
}
|
|
224
|
+
/**
|
|
225
|
+
* The run's own files during a start's window (`runs/<id>/` only, K8's path rules): nobody holds the run, so the place
|
|
226
|
+
* is the only writer for this start. Two supervisors can still be in one window at once (the mount, not the window,
|
|
227
|
+
* admits one of their instances), so a put is not exclusive between them.
|
|
228
|
+
*/
|
|
229
|
+
export interface UnheldRun {
|
|
230
|
+
readonly id: string;
|
|
231
|
+
/** Aborted at the hook's bound (`unheldTimeoutMs`). */
|
|
232
|
+
readonly signal: AbortSignal;
|
|
233
|
+
/** The file's bytes, or null when there is no such file yet (`readRunFile`). */
|
|
234
|
+
read(path: string): Promise<Uint8Array | null>;
|
|
235
|
+
/** The names directly under `dir` ("" for the run's root), sorted; [] for a directory not made yet (`listRunFiles`). */
|
|
236
|
+
list(dir: string): Promise<string[]>;
|
|
237
|
+
/**
|
|
238
|
+
* Create `path` holding `text`, owned by the run user, mode 0644, making each missing parent first (the run user's,
|
|
239
|
+
* 0755). A path already there is EXISTS, and nothing is written. For non-secret run files (labels): a secret never
|
|
240
|
+
* goes through put, since a put file is readable in the run. A put the server refuses because a delegation holds the
|
|
241
|
+
* run (409 OperationAborted), and any put after the hook returned or overran, is WINDOW_CLOSED; any other failure is
|
|
242
|
+
* CONTROL_API_FAILED (`RunFileError`). A put the hook starts and does not await still settles inside the window: the
|
|
243
|
+
* driver starts only once every put has settled, within the hook's bound.
|
|
244
|
+
*/
|
|
245
|
+
put(path: string, text: string): Promise<void>;
|
|
246
|
+
}
|
|
247
|
+
/** Which start the window is for: the generation it opens, and the host box it will make when the place hands labels over. */
|
|
248
|
+
export type UnheldStart = {
|
|
249
|
+
readonly attempt: number;
|
|
250
|
+
/** Exactly the HOST_BOX this start carries; absent when the place hands no label over. */
|
|
251
|
+
readonly box?: {
|
|
252
|
+
readonly provider: string;
|
|
253
|
+
readonly key: string;
|
|
254
|
+
readonly run: string;
|
|
255
|
+
};
|
|
256
|
+
};
|
|
257
|
+
/**
|
|
258
|
+
* A hook in a start's window (`ArchilPlaceOptions.unheld`). What it returns is added to that start's environment, which
|
|
259
|
+
* the instance can read: plain values only. A name the start already sets (PATH, PDA_HOLDER, HOST_BOX or one in the
|
|
260
|
+
* order's environment), a name that is not a variable name, a key-shaped name (a KEY, TOKEN, SECRET, PASSWORD, PASSWD,
|
|
261
|
+
* CREDENTIAL or CREDENTIALS segment) or a value that is not a string without NUL fails the start (INVALID_ARGUMENT,
|
|
262
|
+
* naming the variable, never its value).
|
|
263
|
+
*
|
|
264
|
+
* A start fails, and the driver never starts a box, when any put in the hook was refused because the run is held
|
|
265
|
+
* (WINDOW_CLOSED, even if the hook caught it), when the hook throws a CONTROL_API_FAILED of the control
|
|
266
|
+
* (CONTROL_API_FAILED), or when it throws anything else or overruns its bound (UNHELD_FAILED). The start's caller sees
|
|
267
|
+
* the driver's start failure from ensureRunning (HOST_START_FAILED), with that PlaceError as its cause.
|
|
268
|
+
*/
|
|
269
|
+
export type UnheldHook = (run: UnheldRun, start: UnheldStart) => Promise<Readonly<Record<string, string>> | void>;
|
|
207
270
|
/** The listing `recordHome` reads the run's recordings with: S3 ListObjects, each object's size when the API gives it. */
|
|
208
271
|
export interface RecordControl {
|
|
209
272
|
listObjects(prefix: string, options?: {
|
package/dist/place.js
CHANGED
|
@@ -13,6 +13,8 @@ import { DEFAULT_MOUNT_ROOT, findDelegations, mintMountToken, pathlessResolver,
|
|
|
13
13
|
import { reapCommands } from "./commands.js";
|
|
14
14
|
import { HeldError, PdaError } from "./errors.js";
|
|
15
15
|
import { takeOwnerLock } from "./run.js";
|
|
16
|
+
import { listLocalRunFiles, listRunFiles, putRunFile, readLocalRunFile, readRunFile, RunFileError } from "./run-files.js";
|
|
17
|
+
import { ENV_NAME, keyShaped } from "./hosts/attempt-env.js";
|
|
16
18
|
import { readRunRecord } from "./status.js";
|
|
17
19
|
import { CONTROL_TIMEOUT_MS, ensureRunning, readRunStatus, readStartMark, TOKEN_TTL, withTimeouts, } from "./supervise.js";
|
|
18
20
|
export class PlaceError extends PdaError {
|
|
@@ -173,6 +175,8 @@ export function localPlace(options) {
|
|
|
173
175
|
return {
|
|
174
176
|
name: "local",
|
|
175
177
|
paths,
|
|
178
|
+
readRunFile: (path) => readLocalRunFile(root, paths.id, path),
|
|
179
|
+
listRunFiles: (dir) => listLocalRunFiles(root, paths.id, dir),
|
|
176
180
|
kills: ctx.destroyBoxes ? ["process", "box"] : ["process"],
|
|
177
181
|
start,
|
|
178
182
|
// The runner is a child of this machine, so its kill request is the supervisor's own file.
|
|
@@ -589,6 +593,17 @@ const requestName = (attempt) => `kill-${countFrom1(attempt)}.json`;
|
|
|
589
593
|
*/
|
|
590
594
|
export function archilPlace(options) {
|
|
591
595
|
const { host } = options;
|
|
596
|
+
const unheldMs = options.unheldTimeoutMs ?? 30_000;
|
|
597
|
+
if (options.unheld !== undefined) {
|
|
598
|
+
if (typeof options.unheld !== "function")
|
|
599
|
+
throw new PlaceError("INVALID_ARGUMENT", "unheld is not a function");
|
|
600
|
+
if (!options.ensure?.create)
|
|
601
|
+
throw new PlaceError("INVALID_ARGUMENT", "unheld needs ensure.create: what its hook puts is the run user's, or the run's process could not rely on it");
|
|
602
|
+
if (typeof options.control.listObjects !== "function")
|
|
603
|
+
throw new PlaceError("INVALID_ARGUMENT", "unheld needs a control that can list (listObjects): its hook's read and list use it");
|
|
604
|
+
if (!(Number.isFinite(unheldMs) && unheldMs > 0))
|
|
605
|
+
throw new PlaceError("INVALID_ARGUMENT", "unheldTimeoutMs is not a positive number of milliseconds");
|
|
606
|
+
}
|
|
592
607
|
// Every call the place makes on the disk is bounded as ensureRunning bounds its own: one that does not answer fails as
|
|
593
608
|
// CONTROL_API_FAILED, so a stalled request never leaves a recovery, a release or a runner's end waiting forever.
|
|
594
609
|
const control = withTimeouts(options.control, options.ensure?.controlTimeoutMs ?? CONTROL_TIMEOUT_MS);
|
|
@@ -611,6 +626,12 @@ export function archilPlace(options) {
|
|
|
611
626
|
let resolver = pathlessResolver(control);
|
|
612
627
|
const records = `${runPath(ref.id)}/${RECORDS}/`;
|
|
613
628
|
const recording = typeof options.control.listObjects === "function";
|
|
629
|
+
/**
|
|
630
|
+
* Whether each host box's runner is told its label (HOST_BOX): the driver makes boxes and names their provider, and this
|
|
631
|
+
* place can read the label back (its read members need a control that lists). A label it could never read is never
|
|
632
|
+
* handed over.
|
|
633
|
+
*/
|
|
634
|
+
const handingOver = recording && typeof host.box === "function" && typeof host.provider === "string" && host.provider !== "";
|
|
614
635
|
const owner = options.ensure?.create;
|
|
615
636
|
/**
|
|
616
637
|
* Before attempt `gen`'s runner starts: the EMPTY directories it writes into, `signals/` and, with `recordHome`,
|
|
@@ -637,6 +658,88 @@ export function archilPlace(options) {
|
|
|
637
658
|
}
|
|
638
659
|
}
|
|
639
660
|
};
|
|
661
|
+
/**
|
|
662
|
+
* The hook's turn in a start's window: its run members, its bound, then what it returned, checked. Fails closed: a put
|
|
663
|
+
* refused because the run is held fails the start even when the hook caught it, so no box starts without what the
|
|
664
|
+
* hook meant to write.
|
|
665
|
+
*/
|
|
666
|
+
const inWindow = async (hook, attempt, box, orderEnv) => {
|
|
667
|
+
const bound = new AbortController();
|
|
668
|
+
const timer = setTimeout(() => bound.abort(new PlaceError("UNHELD_FAILED", `the start window's hook overran its ${unheldMs} ms bound`)), unheldMs);
|
|
669
|
+
let open = true;
|
|
670
|
+
let closed;
|
|
671
|
+
/** Every put the hook started, settled or not: the driver starts only once all of them have settled. */
|
|
672
|
+
const pending = new Set();
|
|
673
|
+
const runControl = control;
|
|
674
|
+
const run = {
|
|
675
|
+
id: ref.id,
|
|
676
|
+
signal: bound.signal,
|
|
677
|
+
read: (path) => readRunFile(runControl, ref, path, { signal: bound.signal }),
|
|
678
|
+
list: (dir) => listRunFiles(runControl, ref, dir, { signal: bound.signal }),
|
|
679
|
+
put: (path, text) => {
|
|
680
|
+
if (!open)
|
|
681
|
+
return Promise.reject(new RunFileError("WINDOW_CLOSED", "the start window is closed (its hook returned or overran its bound): nothing more is put"));
|
|
682
|
+
const put = putRunFile(runControl, ref, path, text, owner, { signal: bound.signal }).catch((error) => {
|
|
683
|
+
if (error instanceof RunFileError && error.code === "WINDOW_CLOSED")
|
|
684
|
+
closed ??= error;
|
|
685
|
+
throw error;
|
|
686
|
+
});
|
|
687
|
+
const settled = put.then(() => undefined, () => undefined);
|
|
688
|
+
pending.add(settled);
|
|
689
|
+
void settled.then(() => pending.delete(settled));
|
|
690
|
+
return put;
|
|
691
|
+
},
|
|
692
|
+
};
|
|
693
|
+
// The bound, as a rejection to race against.
|
|
694
|
+
const overran = new Promise((_, reject) => bound.signal.addEventListener("abort", () => reject(bound.signal.reason), { once: true }));
|
|
695
|
+
overran.catch(() => undefined);
|
|
696
|
+
let returned;
|
|
697
|
+
let thrown;
|
|
698
|
+
try {
|
|
699
|
+
returned = await Promise.race([Promise.resolve().then(() => hook(run, box ? { attempt, box } : { attempt })), overran]);
|
|
700
|
+
}
|
|
701
|
+
catch (error) {
|
|
702
|
+
thrown = { error };
|
|
703
|
+
}
|
|
704
|
+
// No put starts once the hook is done, and every put it started (awaited or not) settles inside the bound: none is
|
|
705
|
+
// written, or refused, after the driver has started.
|
|
706
|
+
open = false;
|
|
707
|
+
if (pending.size > 0) {
|
|
708
|
+
try {
|
|
709
|
+
await Promise.race([Promise.allSettled([...pending]), overran]);
|
|
710
|
+
}
|
|
711
|
+
catch (error) {
|
|
712
|
+
thrown ??= { error };
|
|
713
|
+
}
|
|
714
|
+
}
|
|
715
|
+
clearTimeout(timer);
|
|
716
|
+
if (closed)
|
|
717
|
+
throw new PlaceError("WINDOW_CLOSED", "a put in the start window was refused because a delegation holds the run (409 OperationAborted): this start makes no box", { cause: closed });
|
|
718
|
+
if (thrown) {
|
|
719
|
+
const { error } = thrown;
|
|
720
|
+
if (error === bound.signal.reason && error instanceof PlaceError)
|
|
721
|
+
throw error;
|
|
722
|
+
if (error instanceof RunFileError && error.code === "CONTROL_API_FAILED")
|
|
723
|
+
throw new PlaceError("CONTROL_API_FAILED", `the start window's hook failed on the control: ${error.message}`, { cause: error });
|
|
724
|
+
throw new PlaceError("UNHELD_FAILED", `the start window's hook failed: ${String(error?.message ?? error).slice(0, 400)}`, { cause: error });
|
|
725
|
+
}
|
|
726
|
+
if (returned === undefined)
|
|
727
|
+
return {};
|
|
728
|
+
if (returned === null || typeof returned !== "object" || Array.isArray(returned))
|
|
729
|
+
throw new PlaceError("INVALID_ARGUMENT", "the start window's hook returned something other than an environment (names to strings)");
|
|
730
|
+
const taken = new Set(["PATH", "PDA_HOLDER", "HOST_BOX", ...Object.keys(orderEnv)]);
|
|
731
|
+
for (const [name, value] of Object.entries(returned)) {
|
|
732
|
+
if (!ENV_NAME.test(name))
|
|
733
|
+
throw new PlaceError("INVALID_ARGUMENT", `the start window's hook returned ${JSON.stringify(name)}, which is not an environment variable name`);
|
|
734
|
+
if (taken.has(name))
|
|
735
|
+
throw new PlaceError("INVALID_ARGUMENT", `the start window's hook returned ${name}, which this start already sets (the driver, HOST_BOX or the order)`);
|
|
736
|
+
if (keyShaped(name))
|
|
737
|
+
throw new PlaceError("INVALID_ARGUMENT", `the start window's hook returned ${name}, a secret's name (a KEY, TOKEN, SECRET, PASSWORD, PASSWD, CREDENTIAL or CREDENTIALS segment): the instance can read its environment, so a secret never goes there`);
|
|
738
|
+
if (typeof value !== "string" || value.includes("\0"))
|
|
739
|
+
throw new PlaceError("INVALID_ARGUMENT", `the start window's hook returned ${name} with a value that is not a string without NUL`);
|
|
740
|
+
}
|
|
741
|
+
return returned;
|
|
742
|
+
};
|
|
640
743
|
const fetchedSizes = new Map();
|
|
641
744
|
const carrying = new Map();
|
|
642
745
|
const handleOf = new Map();
|
|
@@ -695,7 +798,14 @@ export function archilPlace(options) {
|
|
|
695
798
|
// Inside ensureRunning, after its revoke and start mark: no delegation holds the run, so S3 takes the writes.
|
|
696
799
|
start: async (r, token, attempt) => {
|
|
697
800
|
await makeDirs(order.gen);
|
|
698
|
-
|
|
801
|
+
// The box this start makes is the next in `started`, so its index in status().boxes is started.length: its runner
|
|
802
|
+
// claims that label first (HOST_BOX), for the supervisor's life (RUN_LABEL).
|
|
803
|
+
const life = order.env.RUN_LABEL;
|
|
804
|
+
const box = handingOver && typeof life === "string" && life !== "" ? { provider: host.provider, key: String(started.length), run: life } : undefined;
|
|
805
|
+
const n = attempt?.attempt ?? order.gen;
|
|
806
|
+
const added = options.unheld ? await inWindow(options.unheld, n, box, order.env) : {};
|
|
807
|
+
const env = box ? { ...order.env, ...added, HOST_BOX: JSON.stringify(box) } : { ...order.env, ...added };
|
|
808
|
+
return host.start(r, token, { attempt: n, env });
|
|
699
809
|
},
|
|
700
810
|
status: (h) => host.status(h),
|
|
701
811
|
stop: (h) => host.stop(h),
|
|
@@ -724,8 +834,15 @@ export function archilPlace(options) {
|
|
|
724
834
|
// An adopted instance mounts with an earlier start's token: this one is unused, removed as above.
|
|
725
835
|
if (result.adopted)
|
|
726
836
|
await drop(m).catch(() => { });
|
|
727
|
-
|
|
728
|
-
|
|
837
|
+
// A box this place already started, adopted again (its attempt started twice), keeps the key it has: a second
|
|
838
|
+
// entry would be a key no runner claims (HOST_BOX), a box counted twice, and a box stopped twice. Only the driver
|
|
839
|
+
// can say two handles are one instance (`instanceOf`); a box it cannot name, and every fresh one, is a new entry.
|
|
840
|
+
const name = result.adopted === true ? host.instanceOf?.(result.handle) : undefined;
|
|
841
|
+
const known = (list) => name !== undefined && list.some((h) => host.instanceOf?.(h) === name);
|
|
842
|
+
if (!known(handles))
|
|
843
|
+
handles.push(result.handle);
|
|
844
|
+
if (!known(started))
|
|
845
|
+
started.push(result.handle);
|
|
729
846
|
const w = watch(result.handle);
|
|
730
847
|
const mark = result.startMark ? await readStartMark(control, ref.id).catch(() => null) : null;
|
|
731
848
|
const runner = { epoch: mark?.generation ?? order.gen, exited: w.exited };
|
|
@@ -770,6 +887,13 @@ export function archilPlace(options) {
|
|
|
770
887
|
return {
|
|
771
888
|
name: "archil",
|
|
772
889
|
paths,
|
|
890
|
+
// The run's own files over the control (GET, HEAD, LIST): never a mount, a write, or a wait on the claim.
|
|
891
|
+
...(recording
|
|
892
|
+
? {
|
|
893
|
+
readRunFile: (path) => readRunFile(control, ref, path),
|
|
894
|
+
listRunFiles: (dir) => listRunFiles(control, ref, dir),
|
|
895
|
+
}
|
|
896
|
+
: {}),
|
|
773
897
|
kills: host.kill && host.kills ? [host.kills] : [],
|
|
774
898
|
token: async (attempt) => (await mint(attempt)).token,
|
|
775
899
|
...(recording
|
|
@@ -823,7 +947,12 @@ export function archilPlace(options) {
|
|
|
823
947
|
throw error;
|
|
824
948
|
}),
|
|
825
949
|
]);
|
|
826
|
-
|
|
950
|
+
// With HOST_BOX in play a box's position is its key, so a box never drops out: no receipt fails the read instead.
|
|
951
|
+
const receipts = host.box ? started.map((h) => host.box(h)) : undefined;
|
|
952
|
+
const gap = receipts?.indexOf(null) ?? -1;
|
|
953
|
+
if (handingOver && gap >= 0)
|
|
954
|
+
throw new PlaceError("STATUS_FAILED", `the driver gave no receipt for host box ${gap}, which this place started: every later box's key would shift`);
|
|
955
|
+
const boxes = receipts?.filter((b) => b !== null);
|
|
827
956
|
return { record, delegations: delegations.map(({ clientId, inodeId, path, isOrphaned, isPending }) => ({ clientId, inodeId, path, isOrphaned, isPending })), ...(boxes ? { boxes } : {}) };
|
|
828
957
|
},
|
|
829
958
|
recover: async (last, kind, order) => {
|
|
@@ -0,0 +1,72 @@
|
|
|
1
|
+
import { type RunRef } from "./claim.js";
|
|
2
|
+
import { PdaError } from "./errors.js";
|
|
3
|
+
/** The calls of the disk's control these reads use; `archilControl` has all three. */
|
|
4
|
+
export interface RunFileControl {
|
|
5
|
+
getObject(key: string): Promise<Uint8Array>;
|
|
6
|
+
/** null when there is no such key. */
|
|
7
|
+
headObject(key: string): Promise<unknown | null>;
|
|
8
|
+
listObjects(prefix: string, options?: {
|
|
9
|
+
recursive?: boolean;
|
|
10
|
+
}): Promise<{
|
|
11
|
+
objects: {
|
|
12
|
+
key: string;
|
|
13
|
+
size?: number;
|
|
14
|
+
}[];
|
|
15
|
+
commonPrefixes?: string[];
|
|
16
|
+
}>;
|
|
17
|
+
}
|
|
18
|
+
export type RunFileErrorCode = "INVALID_PATH" | "RUN_MISSING" | "NOT_A_FILE" | "CONTROL_API_FAILED" | "EXISTS" | "WINDOW_CLOSED";
|
|
19
|
+
/** A read of a run's files that cannot be answered. Its message names the path, never what a file holds. */
|
|
20
|
+
export declare class RunFileError extends PdaError {
|
|
21
|
+
readonly code: RunFileErrorCode;
|
|
22
|
+
constructor(code: RunFileErrorCode, message: string, options?: {
|
|
23
|
+
cause?: unknown;
|
|
24
|
+
});
|
|
25
|
+
}
|
|
26
|
+
/** The segments of `path`, or INVALID_PATH naming what is wrong (never the path). `""` is the run's root where allowed. */
|
|
27
|
+
export declare function runFileSegments(path: string, allowRoot: boolean): string[];
|
|
28
|
+
/**
|
|
29
|
+
* The bytes of `path` in the run (`runs/<id>/<path>`), or null when there is no such file yet, so a poller waits. A
|
|
30
|
+
* directory or a symlink is NOT_A_FILE; a run that does not exist is RUN_MISSING. No polling here: the caller bounds its
|
|
31
|
+
* own wait, with `signal` and its own retries.
|
|
32
|
+
*/
|
|
33
|
+
export declare function readRunFile(control: RunFileControl, ref: RunRef, path: string, options?: {
|
|
34
|
+
signal?: AbortSignal;
|
|
35
|
+
}): Promise<Uint8Array | null>;
|
|
36
|
+
/**
|
|
37
|
+
* The names directly under `dir` in the run (`""` for the run's root), sorted: files and directories alike, names only,
|
|
38
|
+
* never a claim about what each one is. A directory not made yet lists nothing; a run that does not exist is RUN_MISSING.
|
|
39
|
+
*/
|
|
40
|
+
export declare function listRunFiles(control: RunFileControl, ref: RunRef, dir: string, options?: {
|
|
41
|
+
signal?: AbortSignal;
|
|
42
|
+
}): Promise<string[]>;
|
|
43
|
+
/** The calls of the disk's control a put uses: HEAD, and PutObject with an owner. `archilControl` has both. */
|
|
44
|
+
export interface RunFilePutControl {
|
|
45
|
+
headObject(key: string): Promise<unknown | null>;
|
|
46
|
+
putObject(key: string, body: string, options: {
|
|
47
|
+
uid: number;
|
|
48
|
+
gid: number;
|
|
49
|
+
mode: number;
|
|
50
|
+
}): Promise<unknown>;
|
|
51
|
+
}
|
|
52
|
+
/**
|
|
53
|
+
* Create `path` in the run (`runs/<id>/<path>`) holding `text`, owned by `owner` with mode 0644, after making each missing
|
|
54
|
+
* parent directory as a marker `owner` owns, mode 0755 (a directory made by path on a mount refuses later creates).
|
|
55
|
+
* Create-only: a file or a directory already at `path` is EXISTS and nothing is written. The check is a HEAD before the
|
|
56
|
+
* PutObject, not a conditional create (Archil's client has none), so it is not exclusive against another writer at the
|
|
57
|
+
* same moment: claim a shared name where a single writer is certain. Only a run nobody holds takes the write; a held one
|
|
58
|
+
* answers 409 OperationAborted, which is WINDOW_CLOSED. For non-secret run files: a put file is readable in the run.
|
|
59
|
+
*/
|
|
60
|
+
export declare function putRunFile(control: RunFilePutControl, ref: RunRef, path: string, text: string, owner: {
|
|
61
|
+
uid: number;
|
|
62
|
+
gid: number;
|
|
63
|
+
}, options?: {
|
|
64
|
+
signal?: AbortSignal;
|
|
65
|
+
}): Promise<void>;
|
|
66
|
+
/**
|
|
67
|
+
* `readRunFile` over the run directory `root` on this machine: the same path rules and answers. A directory, or a symlink
|
|
68
|
+
* anywhere on the way, is NOT_A_FILE, and a file swapped for a link after the check is refused, not followed.
|
|
69
|
+
*/
|
|
70
|
+
export declare function readLocalRunFile(root: string, id: string, path: string): Promise<Uint8Array | null>;
|
|
71
|
+
/** `listRunFiles` over the run directory `root` on this machine: the same rules; a symlink on the way is NOT_A_FILE. */
|
|
72
|
+
export declare function listLocalRunFiles(root: string, id: string, dir: string): Promise<string[]>;
|
|
@@ -0,0 +1,259 @@
|
|
|
1
|
+
// A run's own files, read over the disk's control (S3 GET, HEAD and LIST) and nothing else: these calls never mount, never
|
|
2
|
+
// write, and never take or wait on the run's claim, so anyone may read a run another host holds. A read sees what the
|
|
3
|
+
// holder has synced (its last barrier), and a file the holder writes whole (a temp file, then a link or a rename) is never
|
|
4
|
+
// seen half-written. What a file means is the reader's business: disk parses none of them. The one write here,
|
|
5
|
+
// `putRunFile`, creates a file in a run nobody holds (a held run refuses it), for a place's start window.
|
|
6
|
+
//
|
|
7
|
+
// How Archil's S3 API answers, measured on archil 0.8.42: a missing key is 404; a directory is a marker key ending in "/";
|
|
8
|
+
// a symlink is listed as an ordinary zero-byte key, but its HEAD and GET answer 501 NotImplemented, and S3 never follows a
|
|
9
|
+
// link. So a symlink is told on a read's 501, never from a listing.
|
|
10
|
+
import { constants } from "node:fs";
|
|
11
|
+
import { lstat, open, readdir, stat } from "node:fs/promises";
|
|
12
|
+
import { join } from "node:path";
|
|
13
|
+
import { runPath } from "./claim.js";
|
|
14
|
+
import { PdaError } from "./errors.js";
|
|
15
|
+
/** A read of a run's files that cannot be answered. Its message names the path, never what a file holds. */
|
|
16
|
+
export class RunFileError extends PdaError {
|
|
17
|
+
constructor(code, message, options = {}) {
|
|
18
|
+
super(code, message, options);
|
|
19
|
+
}
|
|
20
|
+
}
|
|
21
|
+
/**
|
|
22
|
+
* What is wrong with `path` as a relative path of safe segments inside the run, or null when nothing is. The answer names
|
|
23
|
+
* the fault only, never the path, which may hold anything a caller passed.
|
|
24
|
+
*/
|
|
25
|
+
function pathFault(path, allowRoot) {
|
|
26
|
+
if (path === "")
|
|
27
|
+
return allowRoot ? null : "it is empty";
|
|
28
|
+
if (path.startsWith("/"))
|
|
29
|
+
return "it is absolute";
|
|
30
|
+
const unsafe = path.search(/[^A-Za-z0-9._/-]/);
|
|
31
|
+
if (unsafe >= 0)
|
|
32
|
+
return `it has an unsafe character at index ${unsafe}`;
|
|
33
|
+
const segments = path.split("/");
|
|
34
|
+
if (segments.includes(""))
|
|
35
|
+
return "it has an empty segment";
|
|
36
|
+
if (segments.includes("."))
|
|
37
|
+
return 'it has a "." segment';
|
|
38
|
+
if (segments.includes(".."))
|
|
39
|
+
return 'it has a ".." segment';
|
|
40
|
+
if (segments[0] === ".archil")
|
|
41
|
+
return "it is under .archil/";
|
|
42
|
+
return null;
|
|
43
|
+
}
|
|
44
|
+
/** The segments of `path`, or INVALID_PATH naming what is wrong (never the path). `""` is the run's root where allowed. */
|
|
45
|
+
export function runFileSegments(path, allowRoot) {
|
|
46
|
+
const fault = pathFault(path, allowRoot);
|
|
47
|
+
if (fault !== null)
|
|
48
|
+
throw new RunFileError("INVALID_PATH", `not a relative path of safe segments (letters, digits, ".", "_", "-") inside the run: ${fault}`);
|
|
49
|
+
return path === "" ? [] : path.split("/");
|
|
50
|
+
}
|
|
51
|
+
/** The run's key prefix and `path` under it, or INVALID_PATH. */
|
|
52
|
+
function keyOf(ref, path, allowRoot) {
|
|
53
|
+
const run = `${runPath(ref.id)}/`;
|
|
54
|
+
const segments = runFileSegments(path, allowRoot);
|
|
55
|
+
return { run, key: segments.length === 0 ? run : `${run}${segments.join("/")}` };
|
|
56
|
+
}
|
|
57
|
+
const statusOf = (error) => error?.status ?? error?.statusCode;
|
|
58
|
+
const missing = (error) => statusOf(error) === 404 || error?.code === "NoSuchKey";
|
|
59
|
+
const notImplemented = (error) => statusOf(error) === 501 || error?.code === "NotImplemented";
|
|
60
|
+
/** `call`, ended at once when `signal` aborts (the request itself may still finish; nothing waits on it). */
|
|
61
|
+
function bounded(call, signal) {
|
|
62
|
+
signal?.throwIfAborted();
|
|
63
|
+
// A control that throws before it returns a promise rejects like any other, so the listener below is always removed.
|
|
64
|
+
const pending = Promise.resolve().then(call);
|
|
65
|
+
if (!signal)
|
|
66
|
+
return pending;
|
|
67
|
+
return new Promise((resolve, reject) => {
|
|
68
|
+
const onAbort = () => reject(signal.reason);
|
|
69
|
+
signal.addEventListener("abort", onAbort, { once: true });
|
|
70
|
+
pending.then((v) => (signal.removeEventListener("abort", onAbort), resolve(v)), (e) => (signal.removeEventListener("abort", onAbort), reject(e)));
|
|
71
|
+
});
|
|
72
|
+
}
|
|
73
|
+
/** A control call; any failure but the ones the caller handles is CONTROL_API_FAILED, an abort stays the caller's. */
|
|
74
|
+
async function ask(what, call, signal) {
|
|
75
|
+
try {
|
|
76
|
+
return await bounded(call, signal);
|
|
77
|
+
}
|
|
78
|
+
catch (error) {
|
|
79
|
+
if (signal?.aborted && error === signal.reason)
|
|
80
|
+
throw error;
|
|
81
|
+
throw error instanceof RunFileError ? error : new RunFileError("CONTROL_API_FAILED", `${what} over the control failed`, { cause: error });
|
|
82
|
+
}
|
|
83
|
+
}
|
|
84
|
+
async function runMissing(control, run, ref, signal) {
|
|
85
|
+
if ((await ask(`reading ${run}`, () => control.headObject(run), signal)) === null)
|
|
86
|
+
throw new RunFileError("RUN_MISSING", `there is no run ${ref.id} on this disk`);
|
|
87
|
+
}
|
|
88
|
+
/**
|
|
89
|
+
* The bytes of `path` in the run (`runs/<id>/<path>`), or null when there is no such file yet, so a poller waits. A
|
|
90
|
+
* directory or a symlink is NOT_A_FILE; a run that does not exist is RUN_MISSING. No polling here: the caller bounds its
|
|
91
|
+
* own wait, with `signal` and its own retries.
|
|
92
|
+
*/
|
|
93
|
+
export async function readRunFile(control, ref, path, options = {}) {
|
|
94
|
+
const { signal } = options;
|
|
95
|
+
const { run, key } = keyOf(ref, path, false);
|
|
96
|
+
signal?.throwIfAborted();
|
|
97
|
+
try {
|
|
98
|
+
return await bounded(() => control.getObject(key), signal);
|
|
99
|
+
}
|
|
100
|
+
catch (error) {
|
|
101
|
+
if (signal?.aborted && error === signal.reason)
|
|
102
|
+
throw error;
|
|
103
|
+
if (notImplemented(error))
|
|
104
|
+
throw new RunFileError("NOT_A_FILE", `${path} in run ${ref.id} is not a regular file (a symlink: the S3 API reads only files)`, { cause: error });
|
|
105
|
+
if (!missing(error))
|
|
106
|
+
throw new RunFileError("CONTROL_API_FAILED", `reading ${path} in run ${ref.id} over the control failed`, { cause: error });
|
|
107
|
+
}
|
|
108
|
+
// Not there: a directory of that name, no run at all, or a file not written yet.
|
|
109
|
+
if ((await ask(`reading ${path}/`, () => control.headObject(`${key}/`), signal)) !== null)
|
|
110
|
+
throw new RunFileError("NOT_A_FILE", `${path} in run ${ref.id} is a directory`);
|
|
111
|
+
await runMissing(control, run, ref, signal);
|
|
112
|
+
return null;
|
|
113
|
+
}
|
|
114
|
+
/**
|
|
115
|
+
* The names directly under `dir` in the run (`""` for the run's root), sorted: files and directories alike, names only,
|
|
116
|
+
* never a claim about what each one is. A directory not made yet lists nothing; a run that does not exist is RUN_MISSING.
|
|
117
|
+
*/
|
|
118
|
+
export async function listRunFiles(control, ref, dir, options = {}) {
|
|
119
|
+
const { signal } = options;
|
|
120
|
+
const { run, key } = keyOf(ref, dir, true);
|
|
121
|
+
const prefix = key.endsWith("/") ? key : `${key}/`;
|
|
122
|
+
const listed = await ask(`listing ${dir || "the run"} of run ${ref.id}`, () => control.listObjects(prefix, { recursive: false }), signal);
|
|
123
|
+
const names = new Set();
|
|
124
|
+
for (const k of [...listed.objects.map((o) => o.key), ...(listed.commonPrefixes ?? [])]) {
|
|
125
|
+
if (!k.startsWith(prefix))
|
|
126
|
+
continue;
|
|
127
|
+
const name = k.slice(prefix.length).replace(/\/$/, "");
|
|
128
|
+
if (name !== "" && !name.includes("/"))
|
|
129
|
+
names.add(name);
|
|
130
|
+
}
|
|
131
|
+
if (names.size === 0)
|
|
132
|
+
await runMissing(control, run, ref, signal);
|
|
133
|
+
return [...names].sort();
|
|
134
|
+
}
|
|
135
|
+
/** A put Archil refused because a delegation holds the run: exactly 409 OperationAborted, as D2 measured on archil 0.8.42. */
|
|
136
|
+
const windowClosed = (error) => statusOf(error) === 409 && error?.code === "OperationAborted";
|
|
137
|
+
/**
|
|
138
|
+
* Create `path` in the run (`runs/<id>/<path>`) holding `text`, owned by `owner` with mode 0644, after making each missing
|
|
139
|
+
* parent directory as a marker `owner` owns, mode 0755 (a directory made by path on a mount refuses later creates).
|
|
140
|
+
* Create-only: a file or a directory already at `path` is EXISTS and nothing is written. The check is a HEAD before the
|
|
141
|
+
* PutObject, not a conditional create (Archil's client has none), so it is not exclusive against another writer at the
|
|
142
|
+
* same moment: claim a shared name where a single writer is certain. Only a run nobody holds takes the write; a held one
|
|
143
|
+
* answers 409 OperationAborted, which is WINDOW_CLOSED. For non-secret run files: a put file is readable in the run.
|
|
144
|
+
*/
|
|
145
|
+
export async function putRunFile(control, ref, path, text, owner, options = {}) {
|
|
146
|
+
const { signal } = options;
|
|
147
|
+
const segments = runFileSegments(path, false);
|
|
148
|
+
const run = `${runPath(ref.id)}/`;
|
|
149
|
+
const key = `${run}${segments.join("/")}`;
|
|
150
|
+
const call = async (what, fn) => {
|
|
151
|
+
try {
|
|
152
|
+
return await bounded(fn, signal);
|
|
153
|
+
}
|
|
154
|
+
catch (error) {
|
|
155
|
+
if (signal?.aborted && error === signal.reason)
|
|
156
|
+
throw error;
|
|
157
|
+
if (windowClosed(error))
|
|
158
|
+
throw new RunFileError("WINDOW_CLOSED", `${what} was refused: a delegation holds run ${ref.id} (409 OperationAborted)`, { cause: error });
|
|
159
|
+
throw new RunFileError("CONTROL_API_FAILED", `${what} over the control failed`, { cause: error });
|
|
160
|
+
}
|
|
161
|
+
};
|
|
162
|
+
if ((await call(`reading ${path}`, () => control.headObject(key))) !== null || (await call(`reading ${path}/`, () => control.headObject(`${key}/`))) !== null) {
|
|
163
|
+
throw new RunFileError("EXISTS", `${path} is already in run ${ref.id}: a put creates, never replaces`);
|
|
164
|
+
}
|
|
165
|
+
for (let i = 1; i < segments.length; i++) {
|
|
166
|
+
const dir = segments.slice(0, i).join("/");
|
|
167
|
+
if ((await call(`reading ${dir}/`, () => control.headObject(`${run}${dir}/`))) === null) {
|
|
168
|
+
await call(`making ${dir}/`, () => control.putObject(`${run}${dir}/`, "", { uid: owner.uid, gid: owner.gid, mode: 0o755 }));
|
|
169
|
+
}
|
|
170
|
+
}
|
|
171
|
+
await call(`writing ${path}`, () => control.putObject(key, text, { uid: owner.uid, gid: owner.gid, mode: 0o644 }));
|
|
172
|
+
}
|
|
173
|
+
// ---- the same reads over a run directory on this machine (localPlace) -------------------------------------------------
|
|
174
|
+
const fsCode = (error) => error?.code;
|
|
175
|
+
async function localRunMissing(root, id) {
|
|
176
|
+
try {
|
|
177
|
+
if ((await stat(root)).isDirectory())
|
|
178
|
+
return;
|
|
179
|
+
}
|
|
180
|
+
catch (error) {
|
|
181
|
+
if (fsCode(error) !== "ENOENT" && fsCode(error) !== "ENOTDIR")
|
|
182
|
+
throw new RunFileError("CONTROL_API_FAILED", `reading run ${id} failed`, { cause: error });
|
|
183
|
+
}
|
|
184
|
+
throw new RunFileError("RUN_MISSING", `there is no run ${id} on this machine`);
|
|
185
|
+
}
|
|
186
|
+
/**
|
|
187
|
+
* Walk `segments` under `root` with lstat: a symlink anywhere on the way is NOT_A_FILE (never followed, so nothing outside
|
|
188
|
+
* the run is ever read). Resolves with the last path and what it is, or null when something on the way is not there.
|
|
189
|
+
*/
|
|
190
|
+
async function walk(root, id, path, segments) {
|
|
191
|
+
let at = root;
|
|
192
|
+
for (const [i, segment] of segments.entries()) {
|
|
193
|
+
at = join(at, segment);
|
|
194
|
+
let st;
|
|
195
|
+
try {
|
|
196
|
+
st = await lstat(at);
|
|
197
|
+
}
|
|
198
|
+
catch (error) {
|
|
199
|
+
if (fsCode(error) === "ENOENT" || fsCode(error) === "ENOTDIR")
|
|
200
|
+
return null;
|
|
201
|
+
throw new RunFileError("CONTROL_API_FAILED", `reading ${path} in run ${id} failed`, { cause: error });
|
|
202
|
+
}
|
|
203
|
+
if (st.isSymbolicLink())
|
|
204
|
+
throw new RunFileError("NOT_A_FILE", `${path} in run ${id} goes through a symlink, which is never followed`);
|
|
205
|
+
if (i < segments.length - 1 && !st.isDirectory())
|
|
206
|
+
return null;
|
|
207
|
+
if (i === segments.length - 1)
|
|
208
|
+
return { at, dir: st.isDirectory() };
|
|
209
|
+
}
|
|
210
|
+
return { at, dir: true };
|
|
211
|
+
}
|
|
212
|
+
/**
|
|
213
|
+
* `readRunFile` over the run directory `root` on this machine: the same path rules and answers. A directory, or a symlink
|
|
214
|
+
* anywhere on the way, is NOT_A_FILE, and a file swapped for a link after the check is refused, not followed.
|
|
215
|
+
*/
|
|
216
|
+
export async function readLocalRunFile(root, id, path) {
|
|
217
|
+
const segments = runFileSegments(path, false);
|
|
218
|
+
await localRunMissing(root, id);
|
|
219
|
+
const found = await walk(root, id, path, segments);
|
|
220
|
+
if (!found)
|
|
221
|
+
return null;
|
|
222
|
+
if (found.dir)
|
|
223
|
+
throw new RunFileError("NOT_A_FILE", `${path} in run ${id} is a directory`);
|
|
224
|
+
let handle;
|
|
225
|
+
try {
|
|
226
|
+
handle = await open(found.at, constants.O_RDONLY | constants.O_NOFOLLOW);
|
|
227
|
+
if (!(await handle.stat()).isFile())
|
|
228
|
+
throw new RunFileError("NOT_A_FILE", `${path} in run ${id} is not a regular file`);
|
|
229
|
+
return new Uint8Array(await handle.readFile());
|
|
230
|
+
}
|
|
231
|
+
catch (error) {
|
|
232
|
+
if (error instanceof RunFileError)
|
|
233
|
+
throw error;
|
|
234
|
+
if (fsCode(error) === "ELOOP")
|
|
235
|
+
throw new RunFileError("NOT_A_FILE", `${path} in run ${id} is a symlink, which is never followed`);
|
|
236
|
+
if (fsCode(error) === "ENOENT")
|
|
237
|
+
return null;
|
|
238
|
+
throw new RunFileError("CONTROL_API_FAILED", `reading ${path} in run ${id} failed`, { cause: error });
|
|
239
|
+
}
|
|
240
|
+
finally {
|
|
241
|
+
await handle?.close();
|
|
242
|
+
}
|
|
243
|
+
}
|
|
244
|
+
/** `listRunFiles` over the run directory `root` on this machine: the same rules; a symlink on the way is NOT_A_FILE. */
|
|
245
|
+
export async function listLocalRunFiles(root, id, dir) {
|
|
246
|
+
const segments = runFileSegments(dir, true);
|
|
247
|
+
await localRunMissing(root, id);
|
|
248
|
+
const found = await walk(root, id, dir, segments);
|
|
249
|
+
if (!found || !found.dir)
|
|
250
|
+
return [];
|
|
251
|
+
try {
|
|
252
|
+
return (await readdir(found.at)).sort();
|
|
253
|
+
}
|
|
254
|
+
catch (error) {
|
|
255
|
+
if (fsCode(error) === "ENOENT" || fsCode(error) === "ENOTDIR")
|
|
256
|
+
return [];
|
|
257
|
+
throw new RunFileError("CONTROL_API_FAILED", `listing ${dir || "the run"} of run ${id} failed`, { cause: error });
|
|
258
|
+
}
|
|
259
|
+
}
|
package/dist/supervise.d.ts
CHANGED
|
@@ -59,6 +59,18 @@ export interface HostDriver {
|
|
|
59
59
|
readonly kills?: "process" | "box";
|
|
60
60
|
/** Optional, for a driver whose instances get machines of their own: the box `handle`'s instance runs in, as a receipt. */
|
|
61
61
|
box?(handle: HostHandle): HostBox | null;
|
|
62
|
+
/**
|
|
63
|
+
* Optional, with `box`: the provider of the boxes this driver makes (its receipts' `provider`), known before a box
|
|
64
|
+
* exists. A place tells each host box's runner which label to claim by it (`HOST_BOX`).
|
|
65
|
+
*/
|
|
66
|
+
readonly provider?: string;
|
|
67
|
+
/**
|
|
68
|
+
* Optional: the name of the instance `handle` is for, the same for its fresh handle and an adopted one (dockerHost: the
|
|
69
|
+
* daemon and the container's name), or `undefined` for a handle that is not this driver's or names none. A place keys
|
|
70
|
+
* the boxes it started by it, so a box adopted again keeps its key. Two handles are one instance only when this names
|
|
71
|
+
* them both, equally: without it no two handles ever are.
|
|
72
|
+
*/
|
|
73
|
+
instanceOf?(handle: HostHandle): string | undefined;
|
|
62
74
|
}
|
|
63
75
|
/** The control-plane calls the supervisor uses; an archil `Disk` satisfies it. */
|
|
64
76
|
export interface SupervisorControl extends ControlApi {
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@parcha/pi-durable-disk",
|
|
3
|
-
"version": "0.1.0-beta.
|
|
3
|
+
"version": "0.1.0-beta.26",
|
|
4
4
|
"description": "The agent is the workspace: its conversation and its files live together in one directory on a disk. Mount it on any Linux host or container, release it, and mount it elsewhere: the agent carries on.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"pi",
|
|
@@ -72,7 +72,7 @@
|
|
|
72
72
|
"@earendil-works/pi-durable": ">=1.0.4 <1.4.0"
|
|
73
73
|
},
|
|
74
74
|
"dependencies": {
|
|
75
|
-
"disk": "1.
|
|
75
|
+
"disk": "1.9.0"
|
|
76
76
|
},
|
|
77
77
|
"devDependencies": {
|
|
78
78
|
"@earendil-works/chord": "1.1.0",
|