@parcha/pi-durable-disk 0.1.0-beta.14 → 0.1.0-beta.15
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +166 -0
- package/README.md +181 -14
- package/dist/app.d.ts +5 -1
- package/dist/claim.d.ts +45 -2
- package/dist/claim.js +145 -15
- package/dist/cli.js +16 -15
- package/dist/errors.d.ts +2 -1
- package/dist/hosts/attempt-env.d.ts +3 -0
- package/dist/hosts/attempt-env.js +15 -0
- package/dist/hosts/daytona-launch.js +8 -7
- package/dist/hosts/daytona.d.ts +2 -2
- package/dist/hosts/daytona.js +33 -2
- package/dist/hosts/docker.d.ts +2 -2
- package/dist/hosts/docker.js +22 -5
- package/dist/hosts/local-host.d.ts +2 -2
- package/dist/hosts/local-host.js +34 -6
- package/dist/index.d.ts +6 -2
- package/dist/index.js +5 -1
- package/dist/lease.d.ts +1 -1
- package/dist/place.d.ts +190 -0
- package/dist/place.js +526 -0
- package/dist/run.d.ts +51 -10
- package/dist/run.js +65 -32
- package/dist/status.d.ts +8 -0
- package/dist/status.js +10 -0
- package/dist/supervise.d.ts +24 -1
- package/dist/supervise.js +1 -1
- package/dist/with-claim.d.ts +63 -0
- package/dist/with-claim.js +325 -0
- package/package.json +2 -1
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,172 @@
|
|
|
2
2
|
|
|
3
3
|
## Unreleased
|
|
4
4
|
|
|
5
|
+
## 0.1.0-beta.15 (2026-10-16)
|
|
6
|
+
|
|
7
|
+
- **`HostDriver.kill`: take an instance away as a crash would, and `kills` says what that takes**
|
|
8
|
+
- `kill(handle)` is optional on the interface and implemented by the three drivers. It drains nothing, releases
|
|
9
|
+
nothing and cleans nothing: the run's claim is left as a crash leaves it, and the supervisor's next pass decides what
|
|
10
|
+
follows. A handle the driver cannot reach, or one from another fleet or boot, is a no-op.
|
|
11
|
+
- `kills` is `process` or `box`. `localHost` kills `process`: `systemctl kill --signal=SIGKILL --kill-whom=all` on the
|
|
12
|
+
unit, or SIGKILL to the child's process group (checked against its start ticks). `dockerHost` and `daytonaHost` kill
|
|
13
|
+
`box`: `docker kill` (the container is left exited and the next start removes it), and the Daytona sandbox's delete
|
|
14
|
+
(a refused delete stops it and deletes again). The launcher is not asked to stop.
|
|
15
|
+
- A kill that fails while the instance still runs throws `KILL_FAILED` (`LocalHostError`, `DockerHostError`,
|
|
16
|
+
`DaytonaHostError`).
|
|
17
|
+
- **`archilPlace.signals`: a runner's kill request travels through the run's disk, so a box can be killed at a named crossing**
|
|
18
|
+
- `Place.requestFile(attempt)` says where an attempt's runner writes its kill request; a place with `signals` has it,
|
|
19
|
+
and `signals(attempt)` reads what is written there. `localPlace`'s is the supervisor's own file (`ctx.requestFile`).
|
|
20
|
+
- `archilPlace`'s is `<root>/signals/kill-<attempt>.json`, in the run's directory and outside `work/`, read over S3
|
|
21
|
+
every 500 ms (`signalPollMs`). The runner writes it durably (a temporary file, fsync, rename, fsync of the directory)
|
|
22
|
+
or S3 may not see it. A missing object is a request not written yet; once reads have failed otherwise for
|
|
23
|
+
`signalFailMs` (60 s) without a break, the iterator yields one `signals-failed` signal (`code`, `message`) and ends.
|
|
24
|
+
It never rejects.
|
|
25
|
+
- A request is yielded once it names `point` and `n`, with every other field it carries in `crossing`, except the
|
|
26
|
+
per-attempt nonce from the supervisor's order, which is the signal's own `nonce` (`PlaceSignal.nonce`), where the
|
|
27
|
+
supervisor reads it for its check. The place's token (on a disk, the mount token) never goes into a request. The
|
|
28
|
+
iterator's `return()` ends a pending `next()` even while its read is in flight.
|
|
29
|
+
- **Places: `localPlace` and `archilPlace`, a run that a supervisor can kill and bring back**
|
|
30
|
+
- Each factory returns `(ctx) => Place`, matching agentrun's reference place seam structurally: `token`, `kills`,
|
|
31
|
+
`start`, `signals`, `kill`, `status`, `recover`, `release`. The package exports the types as `Place`, `PlaceRunner`, `PlaceExit`,
|
|
32
|
+
`RunnerOrder`, `PlaceSignal` and `PlaceContext`, with `PlaceError` for its failures.
|
|
33
|
+
- `localPlace({ app, root? })` runs `run --dir` as a child in its own process group, with no disk and no key. A process
|
|
34
|
+
kill is SIGKILL to the group, and the next runner resumes the run from the directory. `release()` also ends the
|
|
35
|
+
commands the run recorded, which run in process groups of their own, under the run's owner lock (while another
|
|
36
|
+
process holds the run, they are its own). Given `ctx.requestFile`,
|
|
37
|
+
`signals(attempt)` yields the runner's kill request (`PlaceSignal`), and the iterator's `return()` ends a pending
|
|
38
|
+
`next()`.
|
|
39
|
+
- `archilPlace({ disk, region, control, host })` mints each attempt's mount token itself (`token(attempt)`) and starts
|
|
40
|
+
through `ensureRunning`. Its kill is the driver's `kill`. Its recovery takes the run over: the killed holder's delegations
|
|
41
|
+
are revoked whatever its lease says, and the result lists them; a newer incarnation another supervisor started gets
|
|
42
|
+
the usual checks instead. A runner's outcome is `run.json`, read over S3.
|
|
43
|
+
`release()` stops what the place started and removes the tokens it minted; what it could not, it reports
|
|
44
|
+
(`RELEASE_FAILED`) and tries again on the next `release()`.
|
|
45
|
+
- `EnsureOptions.mountToken`: a start uses a token the caller minted instead of minting one, and removes it as it would
|
|
46
|
+
its own when the driver starts nothing or adopts.
|
|
47
|
+
- `StartAttempt.env`: what one start adds to its instance's environment. `localHost`, `dockerHost` and `daytonaHost` merge
|
|
48
|
+
it over their own `env`. An order that sets PATH or PDA_HOLDER (the driver's), or a name that is not a variable name,
|
|
49
|
+
is refused (`INVALID_ARGUMENT`).
|
|
50
|
+
- **`readLocalRunStatus(root)`: the status of a run on a local directory, from the root entry**
|
|
51
|
+
- New root export `readLocalRunStatus(root)` returns the `RunRecord` of a run on a local directory (`localRun`): the same strict
|
|
52
|
+
reader and the same type as `readRunStatus`, which reads a disk's `run.json` through the control API and so cannot serve a
|
|
53
|
+
directory. It answers `null` for a run with no `run.json` yet (or no directory) and throws `RunRecordError`
|
|
54
|
+
(`RUN_JSON_INVALID`) for a malformed record. Before, a caller had `parseRunRecord` and `RUN_JSON` from `./internals`, which
|
|
55
|
+
carries no compatibility promise. The `serve-and-park` example reads its run this way, so its Since is 0.1.0-beta.15.
|
|
56
|
+
Additive.
|
|
57
|
+
- **`seed`: what a new run starts with, written before anything sees the run**
|
|
58
|
+
- `openDurableRun` and `openRunLease` take `seed: (dir: { root, storePath, workDir }) => void | Promise<void>`. It runs
|
|
59
|
+
once, on the open that finds no `run.json`: inside the run's first claim, with the owner lock held, after `store/`,
|
|
60
|
+
`work/` and `tmp/` exist, and before `run.json` is written and the store opens (step `seed`). What it wrote is made
|
|
61
|
+
durable (the claim's barrier) before `run.json` publishes the run.
|
|
62
|
+
- It never runs for a run that has `run.json`. A seed that throws fails the open with its own error, and a process killed
|
|
63
|
+
while it seeds leaves the run unpublished; either way the next open seeds again. It works the same with `localRun`.
|
|
64
|
+
- New types `RunSeed` and `RunSeedDir`, exported from the package and from `./lease`.
|
|
65
|
+
- **`withClaim` and `readReleased`: a short exclusive claim on any path of a disk, and a consistent S3 read of what it
|
|
66
|
+
released**
|
|
67
|
+
- `withClaim({ control, disk, region, path }, fn, options)` holds `path` exclusively while `fn` runs (pi-durable-agent's
|
|
68
|
+
carry back of `agent/agent.sqlite` at a run's end). It mounts at a fresh `<mountRoot>/runs/<pid>-<uuid>`, writes `.claim`
|
|
69
|
+
with its deadline (`holdMs`, 60 s), runs `fn`, then barriers, unmounts and gives its mount token back, also when `fn`
|
|
70
|
+
throws.
|
|
71
|
+
- A lost holder is revoked before the mount: orphaned, or holding exactly the path with an expired `.claim` that stays
|
|
72
|
+
the same for 3 s. A live one within its deadline is waited for (`waitMs`, 30 s), never revoked. An abort of
|
|
73
|
+
`signal` before the mount runs nothing.
|
|
74
|
+
- It makes the path's directories the disk lacks (owned by the calling process, 0700, or `owner`): archil mounts no
|
|
75
|
+
path whose directory is missing.
|
|
76
|
+
- It refuses `INVALID_ARGUMENT` for an unsafe path or one under `runs/` or `.archil/`, `CLAIM_HELD` (76) for a live
|
|
77
|
+
holder after `waitMs`, and `CONTROL_API_FAILED`. It is `FENCED` (75) when revoked from under `fn`, whose writes are
|
|
78
|
+
then not durable.
|
|
79
|
+
- `readReleased({ control, path }, file, { sqlite })` reads a released file over S3, with no mount: no delegation before
|
|
80
|
+
or after, the object unchanged across the read, and for SQLite no `-wal` with content (`NOT_RELEASED`). It returns
|
|
81
|
+
`null` when the file does not exist.
|
|
82
|
+
- `CLAIM_HELD` from either is retryable: the path is not released yet. Retry within `waitMs`, and never fall back to an
|
|
83
|
+
older release.
|
|
84
|
+
- **On a host that refuses every unmount (Sysbox, as Daytona runs sandboxes), the archil client's own bookkeeping no longer fails the release**
|
|
85
|
+
- Deleting any file under an archil mount makes the client list two Active delegations of its own, `<mp>/.archil/client-<n>`
|
|
86
|
+
and `<mp>/.archil/client-<n>/unlinked`. They are outside the run's tree, a checkin of the run leaves them, and a
|
|
87
|
+
checkin of them fails `EROFS` once the mount is read-only.
|
|
88
|
+
- The release counted them as the run's. So any run whose app deleted a file failed every release with
|
|
89
|
+
`UNMOUNT_FAILED` ("still lists a delegation after archil checkin"), even after a completed run. Its instance then
|
|
90
|
+
exited failed, and the restarts failed `CLAIM_NOT_VERIFIED` until the box was deleted.
|
|
91
|
+
- They no longer count. Every other Active delegation under the mount still holds the run, and the mount is left where
|
|
92
|
+
it is, as before.
|
|
93
|
+
- **`run --app`: the app's factory runs once the claim is held, so `ctx.root` is the live run directory**
|
|
94
|
+
- The CLI called the app module's factory before the claim mounted, and a factory that wrote under `ctx.root` (a
|
|
95
|
+
recorder's spool, a `signals/` directory) wrote into the bare mountpoint. The mount hid those writes, and the run later
|
|
96
|
+
failed `ENOENT` on its own files. The claim is now taken first: the mount, the owner lock, the run's directories and
|
|
97
|
+
the end of a killed incarnation's commands. Then the factory runs, then this incarnation's `run.json`, and the Harness
|
|
98
|
+
opens with what the factory returned.
|
|
99
|
+
- `openDurableRun` gains `afterClaim(claim)`, awaited once the claim is held and before `run.json`, with the claim's
|
|
100
|
+
`ref`, `root`, `work` and `store`. Its result (`harness`, `beforeResume`, as `AfterClaim`) goes into the open, so
|
|
101
|
+
`harness` is now optional in the options when `afterClaim` gives it. With neither, the open is refused
|
|
102
|
+
(`INVALID_ARGUMENT`) before any claim; an `afterClaim` that returns no `harness` is refused the same way, after the
|
|
103
|
+
claim. A rejection fails the open and lets the claim go. The open step is `after-claim` (`OpenStep`).
|
|
104
|
+
- The open order changed, for `openRunLease` too: acquire, owner lock, layout (`store/`, `work/`, `tmp/`), the
|
|
105
|
+
`commands` reap, `after-claim`, `seed` (a new run's), `run.json`, heartbeat, then store, seal and Harness. `afterClaim`
|
|
106
|
+
may return a new run's `seed` (`AfterClaim.seed`), which runs over the options' one, right after it. The layout and the reap used to
|
|
107
|
+
come after `run.json` and the heartbeat, and the reap ran only in `openDurableRun`. Code that reads `onStep` sees the
|
|
108
|
+
new order.
|
|
109
|
+
- An open that fails before `run.json` publishes nothing of its incarnation: a new run keeps no `run.json`, and a
|
|
110
|
+
resumed run keeps the previous one as it was. So a module that does not load (`APP_FAILED`, exit 1, as before), or a
|
|
111
|
+
reap refused by a surviving command (exit 76), reads to `ensureRunning` as a failed start: it backs off (the grace
|
|
112
|
+
doubled per failure), instead of finding a fresh `running` record and starting the run again on every tick. The steps
|
|
113
|
+
before `run.json` run inside the supervisor's start grace, not under the heartbeat. Claim options, if one ever needs
|
|
114
|
+
them, are static (CLI flags or the environment), never returned by the factory.
|
|
115
|
+
- **The keyless examples run on macOS, from 0.1.0-beta.14, and say so**
|
|
116
|
+
- kill-9's own check that the cut command still runs read `/proc`, so on macOS the example crashed after the kill. It
|
|
117
|
+
reads `ps` where there is no `/proc`.
|
|
118
|
+
- The examples index says macOS needs 0.1.0-beta.14 or later (the first release that runs there), whatever an
|
|
119
|
+
example's Since says; it no longer says "Linux only".
|
|
120
|
+
- The stranger's walk of the examples installs, per platform, the later of an example's Since and the spec's
|
|
121
|
+
`examples.platformFloors` (macOS: 0.1.0-beta.14). Its Since alone installed 0.1.0-beta.13 on a Mac, which has no
|
|
122
|
+
macOS support, and failed kill-9, audit-across-crashes and no-double-charge there.
|
|
123
|
+
- The macOS CI job runs the keyless example tests (kill-9, audit-across-crashes, no-double-charge, fork-into-many,
|
|
124
|
+
serve-and-park) on a Mac, and its stranger step is blocking again.
|
|
125
|
+
- **A start whose own writes are in the bare mountpoint is refused, `MOUNTPOINT_WRITTEN`, instead of losing them**
|
|
126
|
+
- The CLI calls an app's factory before its claim mounts. A factory that wrote under `ctx.root` (a recorder, a `signals/`
|
|
127
|
+
directory) wrote into the bare mountpoint. Acquire then moved those entries aside as an earlier incarnation's
|
|
128
|
+
leftovers, the mount hid the rest, and the run later failed `ENOENT` on files it had written itself. Every pre-mount
|
|
129
|
+
write was lost without a word.
|
|
130
|
+
- Acquire now tells the two apart by time. Entries written since this process started are its own, so the start is
|
|
131
|
+
refused: `ClaimError` `MOUNTPOINT_WRITTEN`, exit 65 (terminal, since a restart would write them again). The message
|
|
132
|
+
names the first few as `<mp>/...` in at most 240 characters. Nothing moves aside and nothing mounts. Older leftovers
|
|
133
|
+
are moved aside as before, logged by count only.
|
|
134
|
+
- The check looks at every entry, since a directory keeps its time when a file in it is rewritten in place. A mountpoint
|
|
135
|
+
it cannot look at whole (more than 100,000 entries, or an entry this process cannot read) may hide such a write, so it
|
|
136
|
+
is refused too: `ClaimError` `MOUNTPOINT_UNCHECKED`, exit 65, naming none of the entries. Nothing moves aside and
|
|
137
|
+
nothing mounts; someone looks at the directory and moves it aside by hand.
|
|
138
|
+
- `ClaimError` takes an `exitCode`.
|
|
139
|
+
- **A place's outcome from `run.json` is in the run's word: `complete` or `failed`, not the record's status**
|
|
140
|
+
- `localPlace` (without the instance's outcome line) and `archilPlace` read a runner's outcome from `run.json`. Its
|
|
141
|
+
`status` was the record's (`done`, `failed`, `running`, ...), which a supervisor would have to know the package to
|
|
142
|
+
read. It is now `complete` when the run is `done` and the detail of the app's last status carries a `record`, and
|
|
143
|
+
`failed` for anything else: done with no record, failed, or a run its instance left `running`, `paused` or `sleeping`.
|
|
144
|
+
The generation, `code`, `message`, `record` and `storeSha256` are unchanged. An outcome line the instance prints is
|
|
145
|
+
passed through as it is.
|
|
146
|
+
- **A restart whose mount went with the failed incarnation mounts again on Daytona, and fails at once and clearly everywhere else**
|
|
147
|
+
- `daytonaHost`'s launcher handed the attempt's mount token to the first incarnation only, so a restart could only reuse a
|
|
148
|
+
live mount. When the failed incarnation's release had already let the mount go (always on Sysbox, where it is moved
|
|
149
|
+
aside), every restart had no token and no mount. Each one then failed with archil's misleading "Authentication failed",
|
|
150
|
+
until the launcher's restart limit, and the run never recovered. Every incarnation of an attempt now gets the token:
|
|
151
|
+
such a restart mounts again, and after a takeover that mount is `HeldError` (76, terminal), as before.
|
|
152
|
+
- Everywhere, an incarnation given no mount token (`--token-stdin` read nothing) that finds no live mount to reuse is now
|
|
153
|
+
fenced at once, before any mount: `FencedError` `CLAIM_GONE_NO_TOKEN`, exit 75, terminal. It no longer loops on a mount
|
|
154
|
+
that cannot succeed, and the supervisor's next start brings a new token. `localHost` keeps handing its token to the
|
|
155
|
+
first incarnation only.
|
|
156
|
+
- **The Daytona live helpers build a runtime snapshot, so a box is ready in about 2 s instead of 16 s**
|
|
157
|
+
- `test/live/daytona-snapshot.ts build <commit>` builds `pda-p9-runtime-<commit>-<stamp>` from `snapshotDockerfile`:
|
|
158
|
+
`daytona-medium`'s image with the pinned Node and archil client, the run user, the wrapper, the sudoers drop-in and the
|
|
159
|
+
package's dependencies. `prepareOnSnapshot` is the matching `prepare` for a `daytonaHost` start with `snapshot: <name>`:
|
|
160
|
+
it puts the checkout's package source over the snapshot's `node_modules` and installs dependencies only when they
|
|
161
|
+
changed. `check`, `baseline` and `delete` boot, compare and remove.
|
|
162
|
+
- `build <commit> --demo` adds the disk demo's dependencies: the third-party closure of the five workspace packages its box
|
|
163
|
+
installs (dsl, pi, pi-browser, pi-vision, pi-transport) and pi-codemode, pinned to the workspace lockfile at `commit`.
|
|
164
|
+
`installDemoPacks` then installs the checkout's packs offline, in about 1.5 s instead of 6 to 7 s over the network;
|
|
165
|
+
`check-demo` measures it and imports every entry the demo's child uses.
|
|
166
|
+
- The ledger (`P9-STATE.json`) records snapshots too. A snapshot is built once under a new name, never over an existing
|
|
167
|
+
one, and only one this lane built can be deleted. Every command needs the same `PDA_STATE_DIR`. A box whose delete
|
|
168
|
+
failed stays open in the ledger and fails the command, and a build still running when the wait ends stays open.
|
|
169
|
+
- No package code changes: `daytonaHost` already takes `snapshot` and `prepare`.
|
|
170
|
+
|
|
5
171
|
## 0.1.0-beta.14 (2026-10-16)
|
|
6
172
|
|
|
7
173
|
- **The root entry lists its exports, 134 instead of 0.1.0-beta.13's 228; the rest move to `./lease`, to a new `./internals`, or out of the package's entries.** Root values go from 136 to 46: 92 leave root (19 to `./lease`, 15 to `./internals`, 58 to no entry) and 2 are new. Who breaks: an import of one of the names below from `@parcha/pi-durable-disk`. What to do: import it from the entry named here; a name that no entry exports had no caller outside the package; open an issue if you need one.
|
package/README.md
CHANGED
|
@@ -267,7 +267,18 @@ Each example's README lists the exact commands the demo runs, so you can run the
|
|
|
267
267
|
|
|
268
268
|
An instance is `pi-durable-disk run --app <module> ...`: `openDurableRun` with the app's options. The module is an ES
|
|
269
269
|
module whose default export is a function of where the run lives and returns pi's Harness options without `env` (the run
|
|
270
|
-
builds `env` on its claim: the workspace is `<root>/work`), plus an optional `beforeResume` and an optional `onOpen
|
|
270
|
+
builds `env` on its claim: the workspace is `<root>/work`), plus an optional `beforeResume` and an optional `onOpen`.
|
|
271
|
+
|
|
272
|
+
The factory runs once the claim is held: the run's directory is mounted, its owner lock taken, `store/`, `work/` and
|
|
273
|
+
`tmp/` made, and the commands a killed incarnation left running ended (`openDurableRun`'s `afterClaim`). This
|
|
274
|
+
incarnation's `run.json` comes after it. So `where.root` is the live directory whenever app code runs, and a factory may
|
|
275
|
+
write under it (a recorder's spool, a `signals/` directory). Before, the factory ran before the mount, and its writes
|
|
276
|
+
landed in the bare mountpoint and were lost. The Harness opens after the factory, with what it returns. Claim options, if
|
|
277
|
+
one ever needs them, are static (CLI flags or the environment) and never returned by the factory, since the claim is
|
|
278
|
+
taken before the factory runs. A module that does not load, or a factory that throws, fails the open with `APP_FAILED`
|
|
279
|
+
(exit 1), and the claim is let go. Such a start publishes nothing: a new run still has no `run.json`, and a resumed run
|
|
280
|
+
keeps its previous one. A supervisor therefore counts it a failed start and backs off (`ensureRunning`), never starting
|
|
281
|
+
it again on every tick.
|
|
271
282
|
|
|
272
283
|
```ts
|
|
273
284
|
import { BACKGROUND_CONTEXT as ctx } from "@earendil-works/chord/context";
|
|
@@ -278,7 +289,7 @@ import { ArchilCodingTools } from "@parcha/pi-durable-disk";
|
|
|
278
289
|
import type { AppContext, AppOptions } from "@parcha/pi-durable-disk";
|
|
279
290
|
|
|
280
291
|
export default async function app(where: AppContext): Promise<AppOptions> {
|
|
281
|
-
// where = { ref: { disk, region, id }, root, work, store }:
|
|
292
|
+
// where = { ref: { disk, region, id }, root, work, store }: the claim is held, so root is the live run directory.
|
|
282
293
|
const models = createModels();
|
|
283
294
|
models.setProvider(openaiProvider()); // reads OPENAI_API_KEY from the instance's environment
|
|
284
295
|
const registry = createRegistry(); // every extension the run's tasks need
|
|
@@ -309,7 +320,8 @@ rerun. Any other tool is an effect: its intent is committed before it starts, an
|
|
|
309
320
|
was `interrupted`, never run again. `run.setStatus("done" | "failed" | "paused" | "sleeping", detail)` records the run's
|
|
310
321
|
status in `run.json`; `done` and `failed` run the durability barrier first. `run.release()` closes the Harness, kills the
|
|
311
322
|
run's commands, runs the barrier, seals `run.json` with the store's last sequence, and unmounts. A module that is missing
|
|
312
|
-
or throws, or an `onOpen` that rejects, is `AppError` (exit 1, which the unit retries
|
|
323
|
+
or throws, or an `onOpen` that rejects, is `AppError` (exit 1, which the unit retries; a module that fails lets the claim
|
|
324
|
+
go first). A `beforeResume` that rejects
|
|
313
325
|
fails the open (its step is `before-resume`) and releases the claim. Besides the Harness, the hook gets `storage`, the
|
|
314
326
|
store the Harness was opened over as four reads (`scanConversations`, `scanTasks`, `scanDocuments`, `document`) with no
|
|
315
327
|
way to a write, and `head`, the store's last committed sequence: a recorder snapshots every document at `head` there.
|
|
@@ -342,7 +354,7 @@ An instance's exit code tells the host's process supervisor what to do:
|
|
|
342
354
|
| 2 | Usage error | no |
|
|
343
355
|
| 65 | Data error: the store is behind its seal (an acknowledged commit is missing), or `run.json` is unreadable. `run.json` says `failed` | never |
|
|
344
356
|
| 70 | The store's head cannot be read (pi's schema moved). `run.json` says `failed` | never |
|
|
345
|
-
| 75 | Fenced: the claim was revoked, its mount failed or left the mount table (`CLAIM_UNMOUNTED`), or the lease lapsed | never in place; the supervisor decides |
|
|
357
|
+
| 75 | Fenced: the claim was revoked, its mount failed or left the mount table (`CLAIM_UNMOUNTED`), or the lease lapsed; or an incarnation given no mount token found no live mount to reuse (`CLAIM_GONE_NO_TOKEN`: a restart whose mount went with the failed incarnation) | never in place; the supervisor decides |
|
|
346
358
|
| 76 | Held: another client holds the claim (a delegation, live or orphaned; the supervisor revokes orphans), or another process on this host holds the run's owner lock | never in place |
|
|
347
359
|
|
|
348
360
|
## Host drivers and the supervisor
|
|
@@ -381,18 +393,29 @@ Revoking a holder that is alive is safe: its next fsync fails and it exits 75, a
|
|
|
381
393
|
reaches the new owner. No wait follows a revoke, because the fence is the server's, not a clock. Two supervisors racing
|
|
382
394
|
on one run are safe: the mount admits one claimant and the other instance exits 76.
|
|
383
395
|
|
|
384
|
-
A host driver is three calls, and nothing else in the package knows which compute runs an instance:
|
|
396
|
+
A host driver is three calls, plus an optional fourth, and nothing else in the package knows which compute runs an instance:
|
|
385
397
|
|
|
386
398
|
```ts
|
|
387
399
|
export interface HostDriver {
|
|
388
400
|
// The handle is JSON; the instance writes it into run.json. `attempt` (the generation the instance will open) lets a
|
|
389
|
-
// driver key a start, so a retry of the same attempt finds the instance it already started.
|
|
390
|
-
|
|
401
|
+
// driver key a start, so a retry of the same attempt finds the instance it already started. `env` is what this start
|
|
402
|
+
// adds to the instance's environment, over the driver's own `env`; PATH and PDA_HOLDER are the driver's (refused).
|
|
403
|
+
start(ref: RunRef, mountToken: string, attempt?: { attempt: number; env?: Record<string, string> }): Promise<HostHandle>;
|
|
391
404
|
status(handle: HostHandle): Promise<"running" | "stopped" | "failed" | "gone" | "unknown">;
|
|
392
405
|
stop(handle: HostHandle): Promise<void>;
|
|
406
|
+
// Optional: take the instance away as a crash would. No drain, no release, nothing cleaned.
|
|
407
|
+
kill?(handle: HostHandle): Promise<void>;
|
|
408
|
+
readonly kills?: "process" | "box";
|
|
393
409
|
}
|
|
394
410
|
```
|
|
395
411
|
|
|
412
|
+
`kill` leaves the run's claim the way a crash leaves it, and the supervisor's next pass decides what follows. `kills` says
|
|
413
|
+
what a kill takes. `localHost` kills `process`: `systemctl kill --signal=SIGKILL --kill-whom=all` on the unit, or SIGKILL
|
|
414
|
+
to the child's process group. Its machine stays, and with `restart` on, systemd starts the unit again. `dockerHost` kills
|
|
415
|
+
`box` (`docker kill`; the container stays exited until the next start removes it), and so does `daytonaHost` (the
|
|
416
|
+
sandbox's delete). A handle from another fleet or boot is a no-op, and a kill that fails while the instance still runs
|
|
417
|
+
throws `KILL_FAILED`.
|
|
418
|
+
|
|
396
419
|
`localHost()` runs instances on this machine as systemd transient units with `KillMode=control-group`, so stopping or
|
|
397
420
|
restarting the unit kills every command the instance started, pi's detached ones included. The unit restarts on failure,
|
|
398
421
|
except after exit 65, 70, 75 or 76. `localHost({ mode: "child" })` runs a detached child process instead, for development,
|
|
@@ -402,12 +425,16 @@ without that guarantee.
|
|
|
402
425
|
starts local units, so pass the driver to `ensureRunning`). The box comes from a snapshot that holds Node, the archil client
|
|
403
426
|
(`libfuse2`), this package, the run user and the sudoers drop-in, or from a default image that the caller's `prepare` installs
|
|
404
427
|
them into. A container sandbox has no systemd (PID 1 is the Daytona daemon), so an in-box launcher keeps the instance
|
|
405
|
-
alive in place and never restarts it after exit 0, 65, 70, 75 or 76.
|
|
428
|
+
alive in place and never restarts it after exit 0, 65, 70, 75 or 76. Every incarnation of an attempt gets the attempt's
|
|
429
|
+
token on stdin. A restart whose mount is still live reuses it, and one whose mount went with the failed incarnation mounts
|
|
430
|
+
again: after a takeover, that mount is `HeldError` (76, terminal). The token stays in the launcher's memory for the attempt. The mount token is uploaded to a root-only file and
|
|
406
431
|
handed to the instance on stdin, never put in the sandbox's environment, which Daytona stores and returns on every read.
|
|
407
432
|
Status comes from the sandbox state plus the launcher's own word, and `stop` drains the instance, then deletes the box. The
|
|
408
433
|
archil client needs the sandbox's egress to Archil's mount servers (Daytona's Tier 3 or 4, full egress; a domain allowlist
|
|
409
434
|
cannot mount). Where the kernel refuses every unmount (a Sysbox box), the claim's release checks the delegation in, moves the
|
|
410
|
-
mount aside (`archil-scoped retire`) and kills its daemon.
|
|
435
|
+
mount aside (`archil-scoped retire`) and kills its daemon. Before the move, the client must list no Active delegation
|
|
436
|
+
under the mount. Its own bookkeeping under `.archil/` doesn't count: the client makes `.archil/client-<n>` and its
|
|
437
|
+
`unlinked` whenever a file under the mount is deleted, and no checkin clears them once the mount is read-only. Measured live on Daytona's hosted `us` region, about 6 ms from
|
|
411
438
|
`aws-us-east-1`: a box takes about 15 s to start an instance, an exclusive mount 142 ms and a commit 8.55 ms p50; a
|
|
412
439
|
power-off, a freeze of the instance and its FUSE daemon, and a network partition were each taken over on another box (the
|
|
413
440
|
power-off through the orphaned delegation, the freeze and the partition through the lease, the partitioned box stopped by
|
|
@@ -549,6 +576,8 @@ mount token reaches the journal, and that nothing is left behind. It needs syste
|
|
|
549
576
|
| A zombie that never commits again | Its lease watchdog (a worker thread, so a main thread stuck in a FUSE request does not matter) kills its commands at 75 s without touching the mount | an in-flight command runs at most until the self-fence |
|
|
550
577
|
| The mount is taken away under a live instance (an operator's `umount -l`, a cleanup's `fusermount -u`) | The instance writes `run.json` and creates its directories through the directory it opened at acquire, and checks before each write that the run's path is still that mount; at the next heartbeat the path names the local directory under the mountpoint, so it exits 75 (`CLAIM_UNMOUNTED`) | nothing of its own is written to the local disk; as host lost |
|
|
551
578
|
| A run's mountpoint holds entries and nothing is mounted on it (files written by path after a mount vanished) | acquire moves the directory aside whole to `<mountRoot>/.stray/<run>-<tag>` (root-owned, 0700), puts an empty one in its place, logs one JSON line on stderr with the entry count (never a name or a byte of the files), and mounts; `MOUNTPOINT_NOT_EMPTY` when it cannot | nothing in the moved directory is read or brought into the run |
|
|
579
|
+
| The mountpoint holds entries written since the starting incarnation began, before its claim mounted (an app's factory writing under `ctx.root`, which the CLI runs before the mount) | acquire refuses the start: `MOUNTPOINT_WRITTEN`, exit 65 (terminal, since a restart would write them again), naming the first few as `<mp>/...` in at most 240 characters; nothing moves aside and nothing mounts | the writes stay where the app left them, never silently lost; write in `onOpen`. The two rules side by side: this incarnation's own fresh writes are named (they are what the app must fix); an earlier incarnation's leftovers are counted, never named (they may be another incarnation's files) |
|
|
580
|
+
| The mountpoint holds more entries than the stray check looks at (100,000), or one this process cannot read | acquire refuses the start: `MOUNTPOINT_UNCHECKED`, exit 65, naming none of the entries; nothing moves aside and nothing mounts. A directory keeps its time when a file in it is rewritten in place, so only a check that looked at every entry can rule out this incarnation's own writes | someone looks at the directory and moves it aside by hand |
|
|
552
581
|
| A stale archil daemon on the mountpoint with no mount (a client that failed at a token refresh drops its mount and keeps its process and control socket) | acquire kills it (`archil-scoped stale`) before mounting; archil's refusal ("an older Archil process is still running", "Failed to bind control socket") is `MOUNT_FAILED` (exit 1), not 76 | none |
|
|
553
582
|
| The FUSE daemon dies, the loop lives | I/O returns `ENOTCONN`, the claim is fenced, the instance exits 75 and the unit does not restart; the supervisor finds the claim orphaned, revokes it and starts an instance | as host lost |
|
|
554
583
|
| Archil is unavailable or its write buffer is full | The commit fails, pi poisons the session, the instance exits 75; the supervisor retries | nothing is published past the last durable commit |
|
|
@@ -566,7 +595,7 @@ about revocation, takeover and the lease's replacement do not apply. These do:
|
|
|
566
595
|
|
|
567
596
|
| Failure | What happens | Effects |
|
|
568
597
|
|---|---|---|
|
|
569
|
-
| The instance crashes or is SIGKILLed | Its owner lock goes with the process. The next open of the directory (by the app, or whatever restarts it) takes the run, ends the commands the killed incarnation left running (its `commands` step, before the
|
|
598
|
+
| The instance crashes or is SIGKILLed | Its owner lock goes with the process. The next open of the directory (by the app, or whatever restarts it) takes the run, ends the commands the killed incarnation left running (its `commands` step, before any code of the new incarnation runs), and `resume()`s from the last commit | running tasks go back to pending; effects in flight are `interrupted`, never rerun; no command of the killed incarnation still runs once the run resumes. pi makes each command the leader of its own process group, so the SIGKILL does not reach them and a local run has no unit cgroup: the instance records each command's group under `commands/` (by pid, start ticks and boot) and the next open kills what still matches. A command started within about 0.1 s before the kill can be missed |
|
|
570
599
|
| A process the killed incarnation's commands started does not die at the next open (an uninterruptible wait, or one the run's user may not signal) | The open is refused: `HeldError` `COMMANDS_ALIVE`, exit 76, with the message below; the records stay | none: the run does not resume beside it. End those processes and open the run again |
|
|
571
600
|
| Power loss or an OS crash | The store's commits are durable (SQLite at `synchronous = FULL`); `work/` is the operating system's to flush, since the claim's barrier syncs nothing | the store resumes from its last commit; files the agent wrote in `work/` can be older than what the store recorded |
|
|
572
601
|
| A second process opens the run on this machine | The owner lock is refused (`HeldError`, exit 76) | none |
|
|
@@ -664,8 +693,13 @@ then list the prefix and check it is empty.
|
|
|
664
693
|
|
|
665
694
|
## Library
|
|
666
695
|
|
|
667
|
-
`openDurableRun(ref, options)` is what `run` calls: mount, verify the claim, take the owner lock,
|
|
668
|
-
the store, check the seal, open and resume the
|
|
696
|
+
`openDurableRun(ref, options)` is what `run` calls: mount, verify the claim, take the owner lock, make `store/`, `work/`
|
|
697
|
+
and `tmp/`, end a killed incarnation's commands, write `run.json`, open the store, check the seal, open and resume the
|
|
698
|
+
Harness, keep the lease. Its `afterClaim(claim)` is awaited once the claim is held and before `run.json`, with the
|
|
699
|
+
claim's `ref`, `root`, `work` and `store`. What it returns (`harness`, `beforeResume`, `seed`) goes into the open, so
|
|
700
|
+
`harness` may come from there instead of the options, and a factory can give a new run its seed (over the options'
|
|
701
|
+
`seed`); the seed runs right after `afterClaim` and before `run.json`, on a new run only. A rejection fails the open with `run.json` as the open found it and lets the
|
|
702
|
+
claim go. `acquire`, `ensureRunning`, `localHost`,
|
|
669
703
|
`archilEnv` and the typed errors (`FencedError`, `HeldError`, `StoreBehindSealError`) are exported for apps that compose
|
|
670
704
|
their own lifecycle, and so are serve (`serveRun`, `requestRun`), parking (`watchParking`, `drain`, `busyState`, and
|
|
671
705
|
`leaseParkTarget` for a Harness a host opened over `openRunLease`) and `fork`. `src/index.ts` lists every export; the
|
|
@@ -693,6 +727,43 @@ pi-durable, chord and their dependencies).
|
|
|
693
727
|
`runPath`, `unmountClaim`, `createRunDir`, the mount-token and delegation helpers, `isLeaseFresh`, and the Daytona labels,
|
|
694
728
|
names and `sweepSandboxes`. It carries no compatibility promise: a name there can change or go in any release.
|
|
695
729
|
|
|
730
|
+
### Seeding a new run
|
|
731
|
+
|
|
732
|
+
`seed` (in `openDurableRun`'s options, and `openRunLease`'s) writes what a new run starts with before anything sees the
|
|
733
|
+
run: its store file, a SQLite database the Harness then opens (saved sign-in profiles, say), and files in its
|
|
734
|
+
workspace (`work/.agents/skills/`). `beforeResume` runs after the store is open, too late for that.
|
|
735
|
+
|
|
736
|
+
```ts
|
|
737
|
+
await openDurableRun(ref, {
|
|
738
|
+
...localRun("/var/runs/run-1"), // or a disk's mountToken
|
|
739
|
+
harness: { models, registry },
|
|
740
|
+
async seed({ storePath, workDir }) { // also root: the claim's root
|
|
741
|
+
const db = new DatabaseSync(storePath);
|
|
742
|
+
db.exec("CREATE TABLE IF NOT EXISTS x_agent_profiles (name TEXT PRIMARY KEY, profile BLOB)");
|
|
743
|
+
db.close(); // closed before it returns: the store opens it next
|
|
744
|
+
await cp(skillsDir, join(workDir, ".agents/skills"), { recursive: true });
|
|
745
|
+
},
|
|
746
|
+
});
|
|
747
|
+
```
|
|
748
|
+
|
|
749
|
+
- **When it runs.** Once, on the open that finds no `run.json`: inside the run's first claim, with the owner lock held,
|
|
750
|
+
after `store/`, `work/` and `tmp/` exist, a killed incarnation's commands are ended and `afterClaim` has run, before
|
|
751
|
+
`run.json` is written and before the store opens. Its step is `seed`. What it wrote is made durable (the claim's
|
|
752
|
+
barrier) before `run.json` publishes the run.
|
|
753
|
+
- **From the app.** `afterClaim` may return `seed` (as `AfterClaim`), over the options' one, so a factory that runs on the
|
|
754
|
+
claimed directory can give the run its starting state.
|
|
755
|
+
- **When it doesn't.** Never for a run that has `run.json`, whatever its generation, and never on a fork's first open
|
|
756
|
+
(`fork` writes `run.json`).
|
|
757
|
+
- **When it fails.** A seed that throws fails the open with its own error. A process killed while it seeds leaves the run
|
|
758
|
+
unpublished in the same way: no `run.json`, the claim let go. The next open seeds again, so a seed overwrites what an
|
|
759
|
+
earlier attempt wrote.
|
|
760
|
+
- **How long it may take.** Under a supervisor, a seed must end within the start grace (`--start-grace`,
|
|
761
|
+
`startGraceMs`; default the lease expiry, 90 s). Past it, `ensureRunning` takes the start as failed and revokes the
|
|
762
|
+
claim. The seed's next write or its barrier is then refused, the instance is fenced (exit 75) with nothing published,
|
|
763
|
+
and the next start seeds again. A deployment whose seeds can take longer raises the start grace.
|
|
764
|
+
- **On a local directory** (`localRun`) it is the same. As with `work/`, the files it writes are the operating system's
|
|
765
|
+
to flush; a SQLite database it commits with `synchronous = FULL` is durable when the commit returns.
|
|
766
|
+
|
|
696
767
|
### A run on a local directory
|
|
697
768
|
|
|
698
769
|
`localRun(root)` returns the `acquire` and `claimDir` that put a run on a plain directory of this machine, with no mount
|
|
@@ -707,6 +778,14 @@ const run = await openDurableRun({ disk: "local", region: "local", id: "run-1" }
|
|
|
707
778
|
});
|
|
708
779
|
```
|
|
709
780
|
|
|
781
|
+
The run's status is its `run.json`, and `readLocalRunStatus(root)` (the directory `localRun` takes) reads it, read-only and safe beside a live instance, the way `readRunStatus` reads a disk's: the
|
|
782
|
+
same `RunRecord` (`status`, `generation`, `wakeAt`, `holder`, `sealedSeq`, `detail`), `null` when the run has no `run.json`
|
|
783
|
+
yet, and `RunRecordError` (`RUN_JSON_INVALID`) when it is malformed.
|
|
784
|
+
|
|
785
|
+
```ts
|
|
786
|
+
const status = await readLocalRunStatus("/var/runs/run-1"); // { status: "sleeping", wakeAt: "...", ... } or null
|
|
787
|
+
```
|
|
788
|
+
|
|
710
789
|
It is for one machine (a laptop, a CI job, a test), and says so: the claim is the directory itself, so nothing but this
|
|
711
790
|
machine's owner lock keeps a second process off the run (a second open while one holds it is refused), and nothing at
|
|
712
791
|
all keeps out a process on another machine that can reach the directory. Its barrier refuses once the run is fenced,
|
|
@@ -742,8 +821,96 @@ A local run works on macOS for development, and is weaker there than on Linux in
|
|
|
742
821
|
|
|
743
822
|
`fsync` on macOS does not flush the drive's cache, so the store also sets SQLite's `fullfsync` and
|
|
744
823
|
`checkpoint_fullfsync` there (SQLite then syncs with `F_FULLFSYNC`), and refuses to open unless they read back. CI runs
|
|
745
|
-
the no-key quickstart
|
|
746
|
-
the file tools' confinement) on macOS.
|
|
824
|
+
the no-key quickstart, the local run's tests (its command records, `run --dir` and `status --dir`, the lease watchdog,
|
|
825
|
+
the file tools' confinement) and the keyless examples on macOS.
|
|
826
|
+
|
|
827
|
+
### A short claim on any path, and a read of what it released
|
|
828
|
+
|
|
829
|
+
`withClaim` holds one path of a disk exclusively for a moment, outside any run: pi-durable-agent carries an agent's home
|
|
830
|
+
(`agent/agent.sqlite`) back at a run's end this way. `readReleased` reads a file such a claim released over the S3 API,
|
|
831
|
+
with no mount, so a reader never blocks a writer. Both are supervisor side: they take the `control` of
|
|
832
|
+
`archilControl({ apiKey, disk, region })`.
|
|
833
|
+
|
|
834
|
+
```ts
|
|
835
|
+
await withClaim({ control, disk: "dsk-...", region: "aws-us-east-1", path: "agent" }, async (held) => {
|
|
836
|
+
const db = new DatabaseSync(join(held.root, "agent.sqlite"));
|
|
837
|
+
db.exec("INSERT INTO runs (id) VALUES ('job-1'); PRAGMA wal_checkpoint(TRUNCATE)");
|
|
838
|
+
db.close();
|
|
839
|
+
});
|
|
840
|
+
const store = await readReleased({ control, path: "agent" }, "agent.sqlite", { sqlite: true }); // null: none yet
|
|
841
|
+
```
|
|
842
|
+
|
|
843
|
+
A claim is one exclusive mount, about 0.4 to 1 s with its release, so it is for a run's end, never a turn.
|
|
844
|
+
- **Exclusive by the mount.** Archil refuses a second client's mount of the path, an ancestor or a descendant. The call
|
|
845
|
+
mounts at a fresh `<mountRoot>/runs/<pid>-<uuid>` (default `/mnt/archil/.claims`; the shape `archil-scoped`'s cleanup
|
|
846
|
+
takes on a host that cannot unmount) and never shares a mount. Archil mounts no path whose directory is missing, so it
|
|
847
|
+
first makes the path's directories the disk lacks, over S3, as the calling process's own (0700; `owner` sets them). It writes
|
|
848
|
+
`.claim` with its deadline (`holdMs`, default 60 s), runs `fn`, then barriers, unmounts and gives its mount token back,
|
|
849
|
+
whether `fn` resolved or threw.
|
|
850
|
+
- **Lost holders are revoked first.** A holder whose client is gone (orphaned) is revoked before the mount, and so is a
|
|
851
|
+
holder of exactly the path whose `.claim` names a deadline that has passed and stays the same for 3 s (a holder that
|
|
852
|
+
just mounted rewrites it at once; a holder of a path above or below is never revoked by this path's `.claim`). A live holder within its deadline is waited for (`waitMs`, default 30 s), never
|
|
853
|
+
revoked. A holder revoked while still alive fails its next fsync, so its commit rolls back and its own call is FENCED:
|
|
854
|
+
the deadline never decides what is durable, the fence does.
|
|
855
|
+
- **What it refuses.** A path that is not relative and of safe segments, or that is under `runs/` (a run's own) or
|
|
856
|
+
`.archil/`, is `ClaimError` `INVALID_ARGUMENT`, before any call. A live holder after `waitMs` is `HeldError`
|
|
857
|
+
`CLAIM_HELD` (exit 76). The control API failing, or a delegation with no path it cannot attribute, is
|
|
858
|
+
`CONTROL_API_FAILED`. A claim revoked from under `fn` is `FencedError` (exit 75), and `fn`'s writes are then not
|
|
859
|
+
durable.
|
|
860
|
+
- **`CLAIM_HELD` is retryable.** The path is not released yet: retry within `waitMs`, and never fall back to an older
|
|
861
|
+
release.
|
|
862
|
+
|
|
863
|
+
`readReleased` returns the bytes only when the read is of a released file. Nothing holds the path (or an ancestor or a
|
|
864
|
+
descendant) before or after the read, and the object is unchanged across it (its etag, else its size and time). For
|
|
865
|
+
SQLite (`sqlite: true`), its `-wal` must also be absent or empty, so its writer checkpointed with `TRUNCATE` and closed. A
|
|
866
|
+
held path is `CLAIM_HELD`, retryable as above, and a WAL with content is `NOT_RELEASED`. A file that changed during every
|
|
867
|
+
one of `attempts` reads (default 3) is `CLAIM_HELD`. Open the bytes as a local file, read-only (`?immutable=1`).
|
|
868
|
+
|
|
869
|
+
### Places: a run a supervisor can kill and bring back
|
|
870
|
+
|
|
871
|
+
A place is where one run lives and how its instance starts, is killed and comes back, for a supervisor that decides when
|
|
872
|
+
to kill (agentrun's reference demo). Each factory takes its options and returns `(ctx) => Place`; `ctx.paths` is the run's
|
|
873
|
+
id and directory as the supervisor computed them.
|
|
874
|
+
|
|
875
|
+
- `localPlace({ app, root? })` runs `pi-durable-disk run --dir <root> --id <id> --app <app>` as a child in its own
|
|
876
|
+
process group, with `{ PATH, ...order.env }` and nothing else of the caller's environment. It needs no disk, no key and
|
|
877
|
+
no token. A process kill is SIGKILL to the group, and the next runner resumes the run from the directory. A box kill
|
|
878
|
+
also calls `ctx.destroyBoxes` (a deletion that failed is tried again by the next kill or recover). `release()` kills
|
|
879
|
+
what still runs, then, under the run's owner lock, ends the commands the run recorded (pi starts each in a process
|
|
880
|
+
group of its own, which the instance's kill does not reach). While another process holds the run, they are its own,
|
|
881
|
+
and release leaves them. With `ctx.requestFile`, `signals(attempt)` yields the kill request the runner writes
|
|
882
|
+
there when it parks at its planned crossing, and `requestFile` is that same file. `return()` ends a pending `next()`.
|
|
883
|
+
- `archilPlace({ disk, region, control, host })` runs the instance on the disk through a host driver set up with the app.
|
|
884
|
+
- `token(attempt)` mints that attempt's mount token through `control`, and the runner handed that token mounts with it.
|
|
885
|
+
- A start is `ensureRunning` with the token (`mountToken`) and the order's environment (`StartAttempt.env`).
|
|
886
|
+
- A kill is the driver's `kill`, and `kills` is the driver's.
|
|
887
|
+
- A recovery is the takeover: the killed holder's delegations are revoked whatever its lease says, and the next
|
|
888
|
+
instance starts. It reports `revoked` and `ms`. A newer incarnation than the one the place killed (another
|
|
889
|
+
supervisor's) gets the usual lease and start-grace checks instead, so a live one is `HELD`, not revoked.
|
|
890
|
+
- `status()` reads `run.json` over S3 and lists the run's delegations.
|
|
891
|
+
- `requestFile(attempt)` is `<root>/signals/kill-<attempt>.json`, in the run's directory and outside `work/`, and
|
|
892
|
+
`signals(attempt)` reads `runs/<id>/signals/kill-<attempt>.json` over S3 every 500 ms (`signalPollMs`). A runner on
|
|
893
|
+
a box of its own can't reach the supervisor's files, so its request travels through the disk. The runner must write
|
|
894
|
+
it durably for S3 to see it: a temporary file, fsync, rename, fsync of the directory. A missing object is a request
|
|
895
|
+
not written yet; any other read failure is retried. Once reads have failed for `signalFailMs` (60 s) without a
|
|
896
|
+
break, the iterator yields one `{ kind: "signals-failed", code, message }` and ends, so a parked runner never waits
|
|
897
|
+
with no word of why. `code` is the failing read's own error code when it has one (`CONTROL_API_FAILED` for a read
|
|
898
|
+
that timed out), else `SIGNALS_FAILED`. The iterator never rejects.
|
|
899
|
+
- A request names `point` and `n`, and every other field it carries stays in the signal's `crossing`, except the
|
|
900
|
+
per-attempt nonce the supervisor put in the order's env. That one is lifted to the signal's own `nonce` (when it is a
|
|
901
|
+
string), where the supervisor reads it for its check, as from the local place. The place's token never goes into a
|
|
902
|
+
request: on a disk it is the mount token, and the app never holds it.
|
|
903
|
+
- `release()` stops every instance the place started and removes every token it minted. What it could not stop or
|
|
904
|
+
remove, it reports (`RELEASE_FAILED`) and tries again on the next `release()`. The run's directory stays
|
|
905
|
+
(`deleteRunTree`).
|
|
906
|
+
|
|
907
|
+
A runner's `exited` says how it ended. Locally that is the exit code or signal, and the outcome is the instance's
|
|
908
|
+
`{"outcome":{...}}` line on stdout. Without that line, or on a disk (a box's stdout is out of reach), the outcome is
|
|
909
|
+
`run.json`'s, with its generation, and `code`, `message`, `record` and `storeSha256` from the detail of the last status
|
|
910
|
+
the app set. Its `status` is the run's word, not the record's, so a supervisor needs no package vocabulary: `complete`
|
|
911
|
+
when the run is `done` and that detail carries a record, `failed` for anything else (done with no record, failed, or a
|
|
912
|
+
run its instance left `running`, `paused` or `sleeping`). So an app that runs on a box and wants its supervisor to have
|
|
913
|
+
its record puts the record there, in its final `setStatus("done", { record })`.
|
|
747
914
|
|
|
748
915
|
## Tests
|
|
749
916
|
|
package/dist/app.d.ts
CHANGED
|
@@ -26,7 +26,11 @@ export type AppOptions = OpenDurableRunOptions["harness"] & {
|
|
|
26
26
|
/** Record a parking run's wake (park.ts `ParkOptions.wake`); default run.json `sleeping` with `wakeAt`. */
|
|
27
27
|
wake?(run: DurableRun, at: number | null): Promise<void>;
|
|
28
28
|
};
|
|
29
|
-
/**
|
|
29
|
+
/**
|
|
30
|
+
* What `--app` names: `export default async (ctx) => ({ registry, models, onOpen })`. The CLI calls it once the claim is
|
|
31
|
+
* held, so `ctx.root` is the live run directory and the factory may write under it; the Harness opens with what it
|
|
32
|
+
* returns.
|
|
33
|
+
*/
|
|
30
34
|
export type AppModule = (ctx: AppContext) => AppOptions | Promise<AppOptions>;
|
|
31
35
|
/**
|
|
32
36
|
* The app module is missing, does not load, has no default export function, throws, returns no options, or its
|
package/dist/claim.d.ts
CHANGED
|
@@ -143,7 +143,7 @@ export declare function parseTokenNickname(nickname: string, prefix?: string): {
|
|
|
143
143
|
* whose TTL passes fences its mount the same way at the next refresh, so an instance never outlives its token by more
|
|
144
144
|
* than one refresh. The nickname is given, or built from `run` so the user can be traced back to its run.
|
|
145
145
|
*/
|
|
146
|
-
export declare function mintMountToken(control: ControlApi, opts: {
|
|
146
|
+
export declare function mintMountToken(control: Pick<ControlApi, "addUser">, opts: {
|
|
147
147
|
nickname?: string;
|
|
148
148
|
run?: {
|
|
149
149
|
id: string;
|
|
@@ -158,7 +158,7 @@ export declare function mintMountToken(control: ControlApi, opts: {
|
|
|
158
158
|
identifier: string;
|
|
159
159
|
nickname: string;
|
|
160
160
|
}>;
|
|
161
|
-
export declare function removeMountToken(control: ControlApi, identifier: string): Promise<void>;
|
|
161
|
+
export declare function removeMountToken(control: Pick<ControlApi, "removeUser">, identifier: string): Promise<void>;
|
|
162
162
|
/**
|
|
163
163
|
* Delegations on `runs/<id>` and anything under it; never a sibling or a parent. The control API resolves a delegation's
|
|
164
164
|
* path best-effort: it can list one with no path, live or orphaned, on a never-reused run. So when nothing matches by
|
|
@@ -205,6 +205,12 @@ export declare function revoke(control: ControlApi, id: string): Promise<Delegat
|
|
|
205
205
|
export declare function revokeBestEffort(control: Pick<ControlApi, "listDelegations" | "revokeDelegation" | "exec">, id: string): Promise<Delegation[]>;
|
|
206
206
|
/** Revoke through the control API, then mount plainly; if the API path fails, mount with `--force`. */
|
|
207
207
|
export declare function takeOver(control: ControlApi, opts: AcquireOptions): Promise<Claim>;
|
|
208
|
+
/** A path of a disk, as one exclusive mount names it: `runs/<id>` for a run, any safe path for `withClaim`. */
|
|
209
|
+
export type MountTarget = {
|
|
210
|
+
readonly disk: string;
|
|
211
|
+
readonly region: string;
|
|
212
|
+
readonly path: string;
|
|
213
|
+
};
|
|
208
214
|
/**
|
|
209
215
|
* The errno codes of a stat on a FUSE mount whose daemon is gone: ENOTCONN once its connection is torn down, ECONNABORTED
|
|
210
216
|
* for a stat already in flight while it is torn down. Codes, never messages: a message is translated by the locale.
|
|
@@ -216,6 +222,18 @@ export declare const DEAD_CONNECTION: ReadonlySet<string>;
|
|
|
216
222
|
* in and moved aside. Any other live mount that refuses is never forced.
|
|
217
223
|
*/
|
|
218
224
|
export declare function unmountClaim(mountpoint: string, host?: ArchilHost): Promise<Unmounted>;
|
|
225
|
+
/** The most entries of a bare mountpoint the stray check looks at; past it, nothing is moved aside. */
|
|
226
|
+
export declare const STRAY_CHECK_LIMIT = 100000;
|
|
227
|
+
/**
|
|
228
|
+
* The entries under `dir` written since `since` (ms since the epoch), as paths relative to it, walked without following
|
|
229
|
+
* links. `unchecked` says why the walk could not look at every entry: more than `limit` of them, or one it could not
|
|
230
|
+
* read (an entry gone meanwhile is not one). A directory keeps its time when a file in it is rewritten in place, so only
|
|
231
|
+
* a walk that looked at every entry shows there is none.
|
|
232
|
+
*/
|
|
233
|
+
export declare function writtenSince(dir: string, since: number, limit?: number): Promise<{
|
|
234
|
+
fresh: string[];
|
|
235
|
+
unchecked: string | null;
|
|
236
|
+
}>;
|
|
219
237
|
/**
|
|
220
238
|
* Take the run's claim: mount `<disk>:/runs/<id>` exclusively at `<mountRoot>/runs/<id>` and verify it. A live mount of
|
|
221
239
|
* the same run already there is kept (in-place restart); a dead one is cleaned first. With no mount there, a stale archil
|
|
@@ -224,4 +242,29 @@ export declare function unmountClaim(mountpoint: string, host?: ArchilHost): Pro
|
|
|
224
242
|
* verifying write is refused, MOUNTPOINT_NOT_EMPTY when the mountpoint cannot be emptied or archil still finds it not
|
|
225
243
|
* empty, MOUNT_FAILED (exit 1) for a daemon of this host that is still running there.
|
|
226
244
|
*/
|
|
245
|
+
/**
|
|
246
|
+
* The token an instance holds when it was given none (`--token-stdin` read nothing): enough to reuse its live mount, never
|
|
247
|
+
* to mount. A restart in place gets it from a driver that hands the token to the first incarnation only (localHost).
|
|
248
|
+
*/
|
|
249
|
+
export declare const NO_TOKEN = "none";
|
|
227
250
|
export declare function acquire(opts: AcquireOptions): Promise<Claim>;
|
|
251
|
+
/** One exclusive mount of a path, as `withClaim` holds it: its own mountpoint, never reused. */
|
|
252
|
+
export interface ExclusiveMount {
|
|
253
|
+
readonly root: string;
|
|
254
|
+
/** `archil sync`; a refusal (the mount was revoked) is a FencedError. */
|
|
255
|
+
barrier(): Promise<{
|
|
256
|
+
ms: number;
|
|
257
|
+
}>;
|
|
258
|
+
/** `archil unmount` (a dead mount is cleaned), and the mountpoint removed. */
|
|
259
|
+
unmount(): Promise<Unmounted>;
|
|
260
|
+
}
|
|
261
|
+
/**
|
|
262
|
+
* Mount `<disk>:/<path>` exclusively at `mountpoint`, a fresh directory of the caller's own, and verify it, with `probe` as
|
|
263
|
+
* the claim probe's content. Another client's delegation is HeldError CLAIM_HELD (`held` is added to its message); the
|
|
264
|
+
* rest fails as `acquire` does. Unlike `acquire`, a mount already there is never reused, and nothing is forced.
|
|
265
|
+
*/
|
|
266
|
+
export declare function exclusiveMount(target: MountTarget, mountpoint: string, token: string, options: {
|
|
267
|
+
readonly host?: ArchilHost;
|
|
268
|
+
readonly probe: string;
|
|
269
|
+
readonly held?: string;
|
|
270
|
+
}): Promise<ExclusiveMount>;
|