@parcha/pi-durable-disk 0.1.0-beta.14 → 0.1.0-beta.16
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +211 -0
- package/README.md +225 -15
- package/dist/app.d.ts +5 -1
- package/dist/claim.d.ts +45 -2
- package/dist/claim.js +145 -15
- package/dist/cli.js +16 -15
- package/dist/errors.d.ts +2 -1
- package/dist/hosts/attempt-env.d.ts +3 -0
- package/dist/hosts/attempt-env.js +15 -0
- package/dist/hosts/daytona-launch.js +8 -7
- package/dist/hosts/daytona.d.ts +27 -2
- package/dist/hosts/daytona.js +92 -3
- package/dist/hosts/docker.d.ts +2 -2
- package/dist/hosts/docker.js +22 -5
- package/dist/hosts/local-host.d.ts +2 -2
- package/dist/hosts/local-host.js +34 -6
- package/dist/index.d.ts +8 -4
- package/dist/index.js +6 -2
- package/dist/lease.d.ts +1 -1
- package/dist/place.d.ts +236 -0
- package/dist/place.js +859 -0
- package/dist/run.d.ts +51 -10
- package/dist/run.js +75 -33
- package/dist/status.d.ts +29 -0
- package/dist/status.js +89 -0
- package/dist/supervise.d.ts +44 -14
- package/dist/supervise.js +5 -18
- package/dist/with-claim.d.ts +63 -0
- package/dist/with-claim.js +325 -0
- package/package.json +3 -2
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,217 @@
|
|
|
2
2
|
|
|
3
3
|
## Unreleased
|
|
4
4
|
|
|
5
|
+
## 0.1.0-beta.16 (2026-10-18)
|
|
6
|
+
|
|
7
|
+
- **A place's outcome carries the models the run's live parts chose (`PlaceOutcome.models`)**
|
|
8
|
+
- `PlaceOutcome.models?: Record<string, string>`: the role-to-model ids (`lead`, `eyes`, ...) the app reported, on any
|
|
9
|
+
outcome status, so a supervisor's manifest names the models of a remote run as of a local one.
|
|
10
|
+
- `archilPlace` reads it from `run.json`'s final status detail (`detail.models`); `localPlace` from the instance's
|
|
11
|
+
`{"outcome":...}` line, or from `run.json` without one.
|
|
12
|
+
- Both keep it only when it is a plain object whose values are all strings. A missing or malformed one is left out, never
|
|
13
|
+
replaced by `{}`. Before, `localPlace` passed the outcome line's `models` through unchecked, a malformed one included,
|
|
14
|
+
and `archilPlace` never carried it.
|
|
15
|
+
- **`archilPlace.recordHome`: a runner on a box records into the run's disk, and the place brings the recording home**
|
|
16
|
+
- `Place.recordHome` (`PlaceRecordHome`): `dir(attempt)` is where that attempt's runner records, and
|
|
17
|
+
`bringHome(into, { final })` brings what it recorded into the supervisor's recording directory. `archilPlace` has
|
|
18
|
+
it when its control can list (`RecordControl`, as `archilControl` does); `localPlace` has none, since its runner
|
|
19
|
+
writes the supervisor's directory itself.
|
|
20
|
+
- `dir(attempt)` is `<root>/record/<attempt>`, inside the run's directory. `bringHome` reads it over S3 (list and get,
|
|
21
|
+
never the run's claim). Blobs come first, hash-checked and written whole; then the new whole lines of each
|
|
22
|
+
`.spool/*.jsonl` segment, with an offset per file name kept as what `into` already holds, so it is idempotent across
|
|
23
|
+
calls and processes. Carriers that overlap on one destination append each line once: one call at a time per
|
|
24
|
+
destination in a place, and a lock file per spool file across places and processes (a gone holder's is taken over;
|
|
25
|
+
one held past 5 s is `CARRY_FAILED`). Every write is carried to its last byte.
|
|
26
|
+
- With `ensure.create`, each start first makes the EMPTY `signals/`, `record/<attempt>/.spool/` and
|
|
27
|
+
`record/<attempt>/blobs/` over the control, owned by the run user (mode 0755), while no delegation holds the run. A
|
|
28
|
+
directory made by path on an Archil mount was seen to refuse file creates. Every file in them stays the runner's; a
|
|
29
|
+
refused put fails the start with `DIRS_REFUSED`.
|
|
30
|
+
- Box receipts: `HostDriver.box(handle)` (`HostBox`) and `PlaceStatus.boxes`. `archilPlace.status()` lists a receipt
|
|
31
|
+
for each box it started, released ones included: provider, kind, `from`, `to` and, when it has a price, the published
|
|
32
|
+
list price per second (`usdPerSecond`), the page it was read from (`source`) and the date read (`priced`). An unpriced
|
|
33
|
+
box has none of the three, never 0, and no field holds the provider's box id. `daytonaHost` reports its sandboxes, priced at `DAYTONA_LIST_PRICES`
|
|
34
|
+
(https://www.daytona.io/pricing, read 2026-10-16), with the "first 5 GiB free" applied to disk (the footnote's
|
|
35
|
+
placement is ambiguous). A box of unknown size has no price, nor does a box with a GPU, which the page prices per
|
|
36
|
+
model (`daytonaUsdPerSecond`). `localPlace` reports none.
|
|
37
|
+
- Failures carry codes (`PlaceError`). `BLOB_MISMATCH` is thrown after everything else came home. `CARRY_FAILED` is
|
|
38
|
+
thrown for a list or get that fails, and on `final` for a line whose blob never landed. `BOX_GONE` is never thrown
|
|
39
|
+
by this place: the recording lives on the disk, which outlives the box.
|
|
40
|
+
- **A seed of any length stays in the supervisor's start grace**
|
|
41
|
+
- While a new run seeds it has no `run.json` heartbeat, and `ensureRunning` keeps a start in its grace only while the
|
|
42
|
+
start mark (`runs/<id>/start.json`) is younger than the grace (90 s by default). A longer seed was taken as a failed
|
|
43
|
+
start: revoked, then seeded again.
|
|
44
|
+
- Now, while the seed runs, the instance rewrites the mark's `at` every heartbeat through its claim (fsync and rename).
|
|
45
|
+
It does so only when a supervisor's mark for generation 1 is there, keeping its generation, `by`, `failures` and
|
|
46
|
+
`lastExit`. The rewrite runs on a thread of its own, one at a time, so a seed that blocks its thread (a synchronous
|
|
47
|
+
import) stays covered. It stops when the seed ends, before the open goes on, and `run.json`'s heartbeat takes over.
|
|
48
|
+
- A seeder that dies stops rewriting, and the grace runs out as for any start. The mark's format does not change.
|
|
49
|
+
|
|
50
|
+
## 0.1.0-beta.15 (2026-10-16)
|
|
51
|
+
|
|
52
|
+
- **`HostDriver.kill`: take an instance away as a crash would, and `kills` says what that takes**
|
|
53
|
+
- `kill(handle)` is optional on the interface and implemented by the three drivers. It drains nothing, releases
|
|
54
|
+
nothing and cleans nothing: the run's claim is left as a crash leaves it, and the supervisor's next pass decides what
|
|
55
|
+
follows. A handle the driver cannot reach, or one from another fleet or boot, is a no-op.
|
|
56
|
+
- `kills` is `process` or `box`. `localHost` kills `process`: `systemctl kill --signal=SIGKILL --kill-whom=all` on the
|
|
57
|
+
unit, or SIGKILL to the child's process group (checked against its start ticks). `dockerHost` and `daytonaHost` kill
|
|
58
|
+
`box`: `docker kill` (the container is left exited and the next start removes it), and the Daytona sandbox's delete
|
|
59
|
+
(a refused delete stops it and deletes again). The launcher is not asked to stop.
|
|
60
|
+
- A kill that fails while the instance still runs throws `KILL_FAILED` (`LocalHostError`, `DockerHostError`,
|
|
61
|
+
`DaytonaHostError`).
|
|
62
|
+
- **`archilPlace.signals`: a runner's kill request travels through the run's disk, so a box can be killed at a named crossing**
|
|
63
|
+
- `Place.requestFile(attempt)` says where an attempt's runner writes its kill request; a place with `signals` has it,
|
|
64
|
+
and `signals(attempt)` reads what is written there. `localPlace`'s is the supervisor's own file (`ctx.requestFile`).
|
|
65
|
+
- `archilPlace`'s is `<root>/signals/kill-<attempt>.json`, in the run's directory and outside `work/`, read over S3
|
|
66
|
+
every 500 ms (`signalPollMs`). The runner writes it durably (a temporary file, fsync, rename, fsync of the directory)
|
|
67
|
+
or S3 may not see it. A missing object is a request not written yet; once reads have failed otherwise for
|
|
68
|
+
`signalFailMs` (60 s) without a break, the iterator yields one `signals-failed` signal (`code`, `message`) and ends.
|
|
69
|
+
It never rejects.
|
|
70
|
+
- A request is yielded once it names `point` and `n`, with every other field it carries in `crossing`, except the
|
|
71
|
+
per-attempt nonce from the supervisor's order, which is the signal's own `nonce` (`PlaceSignal.nonce`), where the
|
|
72
|
+
supervisor reads it for its check. The place's token (on a disk, the mount token) never goes into a request. The
|
|
73
|
+
iterator's `return()` ends a pending `next()` even while its read is in flight.
|
|
74
|
+
- **Places: `localPlace` and `archilPlace`, a run that a supervisor can kill and bring back**
|
|
75
|
+
- Each factory returns `(ctx) => Place`, matching agentrun's reference place seam structurally: `token`, `kills`,
|
|
76
|
+
`start`, `signals`, `kill`, `status`, `recover`, `release`. The package exports the types as `Place`, `PlaceRunner`, `PlaceExit`,
|
|
77
|
+
`RunnerOrder`, `PlaceSignal` and `PlaceContext`, with `PlaceError` for its failures.
|
|
78
|
+
- `localPlace({ app, root? })` runs `run --dir` as a child in its own process group, with no disk and no key. A process
|
|
79
|
+
kill is SIGKILL to the group, and the next runner resumes the run from the directory. `release()` also ends the
|
|
80
|
+
commands the run recorded, which run in process groups of their own, under the run's owner lock (while another
|
|
81
|
+
process holds the run, they are its own). Given `ctx.requestFile`,
|
|
82
|
+
`signals(attempt)` yields the runner's kill request (`PlaceSignal`), and the iterator's `return()` ends a pending
|
|
83
|
+
`next()`.
|
|
84
|
+
- `archilPlace({ disk, region, control, host })` mints each attempt's mount token itself (`token(attempt)`) and starts
|
|
85
|
+
through `ensureRunning`. Its kill is the driver's `kill`. Its recovery takes the run over: the killed holder's delegations
|
|
86
|
+
are revoked whatever its lease says, and the result lists them; a newer incarnation another supervisor started gets
|
|
87
|
+
the usual checks instead. A runner's outcome is `run.json`, read over S3.
|
|
88
|
+
`release()` stops what the place started and removes the tokens it minted; what it could not, it reports
|
|
89
|
+
(`RELEASE_FAILED`) and tries again on the next `release()`.
|
|
90
|
+
- `EnsureOptions.mountToken`: a start uses a token the caller minted instead of minting one, and removes it as it would
|
|
91
|
+
its own when the driver starts nothing or adopts.
|
|
92
|
+
- `StartAttempt.env`: what one start adds to its instance's environment. `localHost`, `dockerHost` and `daytonaHost` merge
|
|
93
|
+
it over their own `env`. An order that sets PATH or PDA_HOLDER (the driver's), or a name that is not a variable name,
|
|
94
|
+
is refused (`INVALID_ARGUMENT`).
|
|
95
|
+
- **`readLocalRunStatus(root)`: the status of a run on a local directory, from the root entry**
|
|
96
|
+
- New root export `readLocalRunStatus(root)` returns the `RunRecord` of a run on a local directory (`localRun`): the same strict
|
|
97
|
+
reader and the same type as `readRunStatus`, which reads a disk's `run.json` through the control API and so cannot serve a
|
|
98
|
+
directory. It answers `null` for a run with no `run.json` yet (or no directory) and throws `RunRecordError`
|
|
99
|
+
(`RUN_JSON_INVALID`) for a malformed record. Before, a caller had `parseRunRecord` and `RUN_JSON` from `./internals`, which
|
|
100
|
+
carries no compatibility promise. The `serve-and-park` example reads its run this way, so its Since is 0.1.0-beta.15.
|
|
101
|
+
Additive.
|
|
102
|
+
- **`seed`: what a new run starts with, written before anything sees the run**
|
|
103
|
+
- `openDurableRun` and `openRunLease` take `seed: (dir: { root, storePath, workDir }) => void | Promise<void>`. It runs
|
|
104
|
+
once, on the open that finds no `run.json`: inside the run's first claim, with the owner lock held, after `store/`,
|
|
105
|
+
`work/` and `tmp/` exist, and before `run.json` is written and the store opens (step `seed`). What it wrote is made
|
|
106
|
+
durable (the claim's barrier) before `run.json` publishes the run.
|
|
107
|
+
- It never runs for a run that has `run.json`. A seed that throws fails the open with its own error, and a process killed
|
|
108
|
+
while it seeds leaves the run unpublished; either way the next open seeds again. It works the same with `localRun`.
|
|
109
|
+
- New types `RunSeed` and `RunSeedDir`, exported from the package and from `./lease`.
|
|
110
|
+
- **`withClaim` and `readReleased`: a short exclusive claim on any path of a disk, and a consistent S3 read of what it
|
|
111
|
+
released**
|
|
112
|
+
- `withClaim({ control, disk, region, path }, fn, options)` holds `path` exclusively while `fn` runs (pi-durable-agent's
|
|
113
|
+
carry back of `agent/agent.sqlite` at a run's end). It mounts at a fresh `<mountRoot>/runs/<pid>-<uuid>`, writes `.claim`
|
|
114
|
+
with its deadline (`holdMs`, 60 s), runs `fn`, then barriers, unmounts and gives its mount token back, also when `fn`
|
|
115
|
+
throws.
|
|
116
|
+
- A lost holder is revoked before the mount: orphaned, or holding exactly the path with an expired `.claim` that stays
|
|
117
|
+
the same for 3 s. A live one within its deadline is waited for (`waitMs`, 30 s), never revoked. An abort of
|
|
118
|
+
`signal` before the mount runs nothing.
|
|
119
|
+
- It makes the path's directories the disk lacks (owned by the calling process, 0700, or `owner`): archil mounts no
|
|
120
|
+
path whose directory is missing.
|
|
121
|
+
- It refuses `INVALID_ARGUMENT` for an unsafe path or one under `runs/` or `.archil/`, `CLAIM_HELD` (76) for a live
|
|
122
|
+
holder after `waitMs`, and `CONTROL_API_FAILED`. It is `FENCED` (75) when revoked from under `fn`, whose writes are
|
|
123
|
+
then not durable.
|
|
124
|
+
- `readReleased({ control, path }, file, { sqlite })` reads a released file over S3, with no mount: no delegation before
|
|
125
|
+
or after, the object unchanged across the read, and for SQLite no `-wal` with content (`NOT_RELEASED`). It returns
|
|
126
|
+
`null` when the file does not exist.
|
|
127
|
+
- `CLAIM_HELD` from either is retryable: the path is not released yet. Retry within `waitMs`, and never fall back to an
|
|
128
|
+
older release.
|
|
129
|
+
- **On a host that refuses every unmount (Sysbox, as Daytona runs sandboxes), the archil client's own bookkeeping no longer fails the release**
|
|
130
|
+
- Deleting any file under an archil mount makes the client list two Active delegations of its own, `<mp>/.archil/client-<n>`
|
|
131
|
+
and `<mp>/.archil/client-<n>/unlinked`. They are outside the run's tree, a checkin of the run leaves them, and a
|
|
132
|
+
checkin of them fails `EROFS` once the mount is read-only.
|
|
133
|
+
- The release counted them as the run's. So any run whose app deleted a file failed every release with
|
|
134
|
+
`UNMOUNT_FAILED` ("still lists a delegation after archil checkin"), even after a completed run. Its instance then
|
|
135
|
+
exited failed, and the restarts failed `CLAIM_NOT_VERIFIED` until the box was deleted.
|
|
136
|
+
- They no longer count. Every other Active delegation under the mount still holds the run, and the mount is left where
|
|
137
|
+
it is, as before.
|
|
138
|
+
- **`run --app`: the app's factory runs once the claim is held, so `ctx.root` is the live run directory**
|
|
139
|
+
- The CLI called the app module's factory before the claim mounted, and a factory that wrote under `ctx.root` (a
|
|
140
|
+
recorder's spool, a `signals/` directory) wrote into the bare mountpoint. The mount hid those writes, and the run later
|
|
141
|
+
failed `ENOENT` on its own files. The claim is now taken first: the mount, the owner lock, the run's directories and
|
|
142
|
+
the end of a killed incarnation's commands. Then the factory runs, then this incarnation's `run.json`, and the Harness
|
|
143
|
+
opens with what the factory returned.
|
|
144
|
+
- `openDurableRun` gains `afterClaim(claim)`, awaited once the claim is held and before `run.json`, with the claim's
|
|
145
|
+
`ref`, `root`, `work` and `store`. Its result (`harness`, `beforeResume`, as `AfterClaim`) goes into the open, so
|
|
146
|
+
`harness` is now optional in the options when `afterClaim` gives it. With neither, the open is refused
|
|
147
|
+
(`INVALID_ARGUMENT`) before any claim; an `afterClaim` that returns no `harness` is refused the same way, after the
|
|
148
|
+
claim. A rejection fails the open and lets the claim go. The open step is `after-claim` (`OpenStep`).
|
|
149
|
+
- The open order changed, for `openRunLease` too: acquire, owner lock, layout (`store/`, `work/`, `tmp/`), the
|
|
150
|
+
`commands` reap, `after-claim`, `seed` (a new run's), `run.json`, heartbeat, then store, seal and Harness. `afterClaim`
|
|
151
|
+
may return a new run's `seed` (`AfterClaim.seed`), which runs over the options' one, right after it. The layout and the reap used to
|
|
152
|
+
come after `run.json` and the heartbeat, and the reap ran only in `openDurableRun`. Code that reads `onStep` sees the
|
|
153
|
+
new order.
|
|
154
|
+
- An open that fails before `run.json` publishes nothing of its incarnation: a new run keeps no `run.json`, and a
|
|
155
|
+
resumed run keeps the previous one as it was. So a module that does not load (`APP_FAILED`, exit 1, as before), or a
|
|
156
|
+
reap refused by a surviving command (exit 76), reads to `ensureRunning` as a failed start: it backs off (the grace
|
|
157
|
+
doubled per failure), instead of finding a fresh `running` record and starting the run again on every tick. The steps
|
|
158
|
+
before `run.json` run inside the supervisor's start grace, not under the heartbeat. Claim options, if one ever needs
|
|
159
|
+
them, are static (CLI flags or the environment), never returned by the factory.
|
|
160
|
+
- **The keyless examples run on macOS, from 0.1.0-beta.14, and say so**
|
|
161
|
+
- kill-9's own check that the cut command still runs read `/proc`, so on macOS the example crashed after the kill. It
|
|
162
|
+
reads `ps` where there is no `/proc`.
|
|
163
|
+
- The examples index says macOS needs 0.1.0-beta.14 or later (the first release that runs there), whatever an
|
|
164
|
+
example's Since says; it no longer says "Linux only".
|
|
165
|
+
- The stranger's walk of the examples installs, per platform, the later of an example's Since and the spec's
|
|
166
|
+
`examples.platformFloors` (macOS: 0.1.0-beta.14). Its Since alone installed 0.1.0-beta.13 on a Mac, which has no
|
|
167
|
+
macOS support, and failed kill-9, audit-across-crashes and no-double-charge there.
|
|
168
|
+
- The macOS CI job runs the keyless example tests (kill-9, audit-across-crashes, no-double-charge, fork-into-many,
|
|
169
|
+
serve-and-park) on a Mac, and its stranger step is blocking again.
|
|
170
|
+
- **A start whose own writes are in the bare mountpoint is refused, `MOUNTPOINT_WRITTEN`, instead of losing them**
|
|
171
|
+
- The CLI calls an app's factory before its claim mounts. A factory that wrote under `ctx.root` (a recorder, a `signals/`
|
|
172
|
+
directory) wrote into the bare mountpoint. Acquire then moved those entries aside as an earlier incarnation's
|
|
173
|
+
leftovers, the mount hid the rest, and the run later failed `ENOENT` on files it had written itself. Every pre-mount
|
|
174
|
+
write was lost without a word.
|
|
175
|
+
- Acquire now tells the two apart by time. Entries written since this process started are its own, so the start is
|
|
176
|
+
refused: `ClaimError` `MOUNTPOINT_WRITTEN`, exit 65 (terminal, since a restart would write them again). The message
|
|
177
|
+
names the first few as `<mp>/...` in at most 240 characters. Nothing moves aside and nothing mounts. Older leftovers
|
|
178
|
+
are moved aside as before, logged by count only.
|
|
179
|
+
- The check looks at every entry, since a directory keeps its time when a file in it is rewritten in place. A mountpoint
|
|
180
|
+
it cannot look at whole (more than 100,000 entries, or an entry this process cannot read) may hide such a write, so it
|
|
181
|
+
is refused too: `ClaimError` `MOUNTPOINT_UNCHECKED`, exit 65, naming none of the entries. Nothing moves aside and
|
|
182
|
+
nothing mounts; someone looks at the directory and moves it aside by hand.
|
|
183
|
+
- `ClaimError` takes an `exitCode`.
|
|
184
|
+
- **A place's outcome from `run.json` is in the run's word: `complete` or `failed`, not the record's status**
|
|
185
|
+
- `localPlace` (without the instance's outcome line) and `archilPlace` read a runner's outcome from `run.json`. Its
|
|
186
|
+
`status` was the record's (`done`, `failed`, `running`, ...), which a supervisor would have to know the package to
|
|
187
|
+
read. It is now `complete` when the run is `done` and the detail of the app's last status carries a `record`, and
|
|
188
|
+
`failed` for anything else: done with no record, failed, or a run its instance left `running`, `paused` or `sleeping`.
|
|
189
|
+
The generation, `code`, `message`, `record` and `storeSha256` are unchanged. An outcome line the instance prints is
|
|
190
|
+
passed through as it is.
|
|
191
|
+
- **A restart whose mount went with the failed incarnation mounts again on Daytona, and fails at once and clearly everywhere else**
|
|
192
|
+
- `daytonaHost`'s launcher handed the attempt's mount token to the first incarnation only, so a restart could only reuse a
|
|
193
|
+
live mount. When the failed incarnation's release had already let the mount go (always on Sysbox, where it is moved
|
|
194
|
+
aside), every restart had no token and no mount. Each one then failed with archil's misleading "Authentication failed",
|
|
195
|
+
until the launcher's restart limit, and the run never recovered. Every incarnation of an attempt now gets the token:
|
|
196
|
+
such a restart mounts again, and after a takeover that mount is `HeldError` (76, terminal), as before.
|
|
197
|
+
- Everywhere, an incarnation given no mount token (`--token-stdin` read nothing) that finds no live mount to reuse is now
|
|
198
|
+
fenced at once, before any mount: `FencedError` `CLAIM_GONE_NO_TOKEN`, exit 75, terminal. It no longer loops on a mount
|
|
199
|
+
that cannot succeed, and the supervisor's next start brings a new token. `localHost` keeps handing its token to the
|
|
200
|
+
first incarnation only.
|
|
201
|
+
- **The Daytona live helpers build a runtime snapshot, so a box is ready in about 2 s instead of 16 s**
|
|
202
|
+
- `test/live/daytona-snapshot.ts build <commit>` builds `pda-p9-runtime-<commit>-<stamp>` from `snapshotDockerfile`:
|
|
203
|
+
`daytona-medium`'s image with the pinned Node and archil client, the run user, the wrapper, the sudoers drop-in and the
|
|
204
|
+
package's dependencies. `prepareOnSnapshot` is the matching `prepare` for a `daytonaHost` start with `snapshot: <name>`:
|
|
205
|
+
it puts the checkout's package source over the snapshot's `node_modules` and installs dependencies only when they
|
|
206
|
+
changed. `check`, `baseline` and `delete` boot, compare and remove.
|
|
207
|
+
- `build <commit> --demo` adds the disk demo's dependencies: the third-party closure of the five workspace packages its box
|
|
208
|
+
installs (dsl, pi, pi-browser, pi-vision, pi-transport) and pi-codemode, pinned to the workspace lockfile at `commit`.
|
|
209
|
+
`installDemoPacks` then installs the checkout's packs offline, in about 1.5 s instead of 6 to 7 s over the network;
|
|
210
|
+
`check-demo` measures it and imports every entry the demo's child uses.
|
|
211
|
+
- The ledger (`P9-STATE.json`) records snapshots too. A snapshot is built once under a new name, never over an existing
|
|
212
|
+
one, and only one this lane built can be deleted. Every command needs the same `PDA_STATE_DIR`. A box whose delete
|
|
213
|
+
failed stays open in the ledger and fails the command, and a build still running when the wait ends stays open.
|
|
214
|
+
- No package code changes: `daytonaHost` already takes `snapshot` and `prepare`.
|
|
215
|
+
|
|
5
216
|
## 0.1.0-beta.14 (2026-10-16)
|
|
6
217
|
|
|
7
218
|
- **The root entry lists its exports, 134 instead of 0.1.0-beta.13's 228; the rest move to `./lease`, to a new `./internals`, or out of the package's entries.** Root values go from 136 to 46: 92 leave root (19 to `./lease`, 15 to `./internals`, 58 to no entry) and 2 are new. Who breaks: an import of one of the names below from `@parcha/pi-durable-disk`. What to do: import it from the entry named here; a name that no entry exports had no caller outside the package; open an issue if you need one.
|
package/README.md
CHANGED
|
@@ -267,7 +267,18 @@ Each example's README lists the exact commands the demo runs, so you can run the
|
|
|
267
267
|
|
|
268
268
|
An instance is `pi-durable-disk run --app <module> ...`: `openDurableRun` with the app's options. The module is an ES
|
|
269
269
|
module whose default export is a function of where the run lives and returns pi's Harness options without `env` (the run
|
|
270
|
-
builds `env` on its claim: the workspace is `<root>/work`), plus an optional `beforeResume` and an optional `onOpen
|
|
270
|
+
builds `env` on its claim: the workspace is `<root>/work`), plus an optional `beforeResume` and an optional `onOpen`.
|
|
271
|
+
|
|
272
|
+
The factory runs once the claim is held: the run's directory is mounted, its owner lock taken, `store/`, `work/` and
|
|
273
|
+
`tmp/` made, and the commands a killed incarnation left running ended (`openDurableRun`'s `afterClaim`). This
|
|
274
|
+
incarnation's `run.json` comes after it. So `where.root` is the live directory whenever app code runs, and a factory may
|
|
275
|
+
write under it (a recorder's spool, a `signals/` directory). Before, the factory ran before the mount, and its writes
|
|
276
|
+
landed in the bare mountpoint and were lost. The Harness opens after the factory, with what it returns. Claim options, if
|
|
277
|
+
one ever needs them, are static (CLI flags or the environment) and never returned by the factory, since the claim is
|
|
278
|
+
taken before the factory runs. A module that does not load, or a factory that throws, fails the open with `APP_FAILED`
|
|
279
|
+
(exit 1), and the claim is let go. Such a start publishes nothing: a new run still has no `run.json`, and a resumed run
|
|
280
|
+
keeps its previous one. A supervisor therefore counts it a failed start and backs off (`ensureRunning`), never starting
|
|
281
|
+
it again on every tick.
|
|
271
282
|
|
|
272
283
|
```ts
|
|
273
284
|
import { BACKGROUND_CONTEXT as ctx } from "@earendil-works/chord/context";
|
|
@@ -278,7 +289,7 @@ import { ArchilCodingTools } from "@parcha/pi-durable-disk";
|
|
|
278
289
|
import type { AppContext, AppOptions } from "@parcha/pi-durable-disk";
|
|
279
290
|
|
|
280
291
|
export default async function app(where: AppContext): Promise<AppOptions> {
|
|
281
|
-
// where = { ref: { disk, region, id }, root, work, store }:
|
|
292
|
+
// where = { ref: { disk, region, id }, root, work, store }: the claim is held, so root is the live run directory.
|
|
282
293
|
const models = createModels();
|
|
283
294
|
models.setProvider(openaiProvider()); // reads OPENAI_API_KEY from the instance's environment
|
|
284
295
|
const registry = createRegistry(); // every extension the run's tasks need
|
|
@@ -309,7 +320,8 @@ rerun. Any other tool is an effect: its intent is committed before it starts, an
|
|
|
309
320
|
was `interrupted`, never run again. `run.setStatus("done" | "failed" | "paused" | "sleeping", detail)` records the run's
|
|
310
321
|
status in `run.json`; `done` and `failed` run the durability barrier first. `run.release()` closes the Harness, kills the
|
|
311
322
|
run's commands, runs the barrier, seals `run.json` with the store's last sequence, and unmounts. A module that is missing
|
|
312
|
-
or throws, or an `onOpen` that rejects, is `AppError` (exit 1, which the unit retries
|
|
323
|
+
or throws, or an `onOpen` that rejects, is `AppError` (exit 1, which the unit retries; a module that fails lets the claim
|
|
324
|
+
go first). A `beforeResume` that rejects
|
|
313
325
|
fails the open (its step is `before-resume`) and releases the claim. Besides the Harness, the hook gets `storage`, the
|
|
314
326
|
store the Harness was opened over as four reads (`scanConversations`, `scanTasks`, `scanDocuments`, `document`) with no
|
|
315
327
|
way to a write, and `head`, the store's last committed sequence: a recorder snapshots every document at `head` there.
|
|
@@ -342,7 +354,7 @@ An instance's exit code tells the host's process supervisor what to do:
|
|
|
342
354
|
| 2 | Usage error | no |
|
|
343
355
|
| 65 | Data error: the store is behind its seal (an acknowledged commit is missing), or `run.json` is unreadable. `run.json` says `failed` | never |
|
|
344
356
|
| 70 | The store's head cannot be read (pi's schema moved). `run.json` says `failed` | never |
|
|
345
|
-
| 75 | Fenced: the claim was revoked, its mount failed or left the mount table (`CLAIM_UNMOUNTED`), or the lease lapsed | never in place; the supervisor decides |
|
|
357
|
+
| 75 | Fenced: the claim was revoked, its mount failed or left the mount table (`CLAIM_UNMOUNTED`), or the lease lapsed; or an incarnation given no mount token found no live mount to reuse (`CLAIM_GONE_NO_TOKEN`: a restart whose mount went with the failed incarnation) | never in place; the supervisor decides |
|
|
346
358
|
| 76 | Held: another client holds the claim (a delegation, live or orphaned; the supervisor revokes orphans), or another process on this host holds the run's owner lock | never in place |
|
|
347
359
|
|
|
348
360
|
## Host drivers and the supervisor
|
|
@@ -381,33 +393,54 @@ Revoking a holder that is alive is safe: its next fsync fails and it exits 75, a
|
|
|
381
393
|
reaches the new owner. No wait follows a revoke, because the fence is the server's, not a clock. Two supervisors racing
|
|
382
394
|
on one run are safe: the mount admits one claimant and the other instance exits 76.
|
|
383
395
|
|
|
384
|
-
A host driver is three calls, and nothing else in the package knows which compute runs an instance:
|
|
396
|
+
A host driver is three calls, plus an optional fourth, and nothing else in the package knows which compute runs an instance:
|
|
385
397
|
|
|
386
398
|
```ts
|
|
387
399
|
export interface HostDriver {
|
|
388
400
|
// The handle is JSON; the instance writes it into run.json. `attempt` (the generation the instance will open) lets a
|
|
389
|
-
// driver key a start, so a retry of the same attempt finds the instance it already started.
|
|
390
|
-
|
|
401
|
+
// driver key a start, so a retry of the same attempt finds the instance it already started. `env` is what this start
|
|
402
|
+
// adds to the instance's environment, over the driver's own `env`; PATH and PDA_HOLDER are the driver's (refused).
|
|
403
|
+
start(ref: RunRef, mountToken: string, attempt?: { attempt: number; env?: Record<string, string> }): Promise<HostHandle>;
|
|
391
404
|
status(handle: HostHandle): Promise<"running" | "stopped" | "failed" | "gone" | "unknown">;
|
|
392
405
|
stop(handle: HostHandle): Promise<void>;
|
|
406
|
+
// Optional: take the instance away as a crash would. No drain, no release, nothing cleaned.
|
|
407
|
+
kill?(handle: HostHandle): Promise<void>;
|
|
408
|
+
readonly kills?: "process" | "box";
|
|
393
409
|
}
|
|
394
410
|
```
|
|
395
411
|
|
|
412
|
+
`kill` leaves the run's claim the way a crash leaves it, and the supervisor's next pass decides what follows. `kills` says
|
|
413
|
+
what a kill takes. `localHost` kills `process`: `systemctl kill --signal=SIGKILL --kill-whom=all` on the unit, or SIGKILL
|
|
414
|
+
to the child's process group. Its machine stays, and with `restart` on, systemd starts the unit again. `dockerHost` kills
|
|
415
|
+
`box` (`docker kill`; the container stays exited until the next start removes it), and so does `daytonaHost` (the
|
|
416
|
+
sandbox's delete). A handle from another fleet or boot is a no-op, and a kill that fails while the instance still runs
|
|
417
|
+
throws `KILL_FAILED`.
|
|
418
|
+
|
|
396
419
|
`localHost()` runs instances on this machine as systemd transient units with `KillMode=control-group`, so stopping or
|
|
397
420
|
restarting the unit kills every command the instance started, pi's detached ones included. The unit restarts on failure,
|
|
398
421
|
except after exit 65, 70, 75 or 76. `localHost({ mode: "child" })` runs a detached child process instead, for development,
|
|
399
422
|
without that guarantee.
|
|
400
423
|
|
|
401
424
|
`daytonaHost({ client, snapshot, ... })` runs each instance in a Daytona sandbox of its own (library only: `supervise`
|
|
402
|
-
starts local units, so pass the driver to `ensureRunning`).
|
|
425
|
+
starts local units, so pass the driver to `ensureRunning`). Its `box(handle)` is that sandbox's receipt: its size
|
|
426
|
+
(vCPUs, memory, disk), when it was made and when the driver deleted it, and its list price per second at
|
|
427
|
+
`DAYTONA_LIST_PRICES`. Those are Daytona's published rates, with the page and the date they were read
|
|
428
|
+
(https://www.daytona.io/pricing, 2026-10-16): $0.0504 per vCPU-hour, $0.0162 per GiB-hour of memory and $0.000108 per
|
|
429
|
+
GiB-hour of disk past the first 5 GiB, billed per second. The page's "first 5 free" footnote is applied to disk only (its
|
|
430
|
+
placement is ambiguous). A box of unknown size gets no price, and neither does a box with a GPU: the page prices GPUs per
|
|
431
|
+
model, which the driver doesn't map (`daytonaUsdPerSecond`). The box comes from a snapshot that holds Node, the archil client
|
|
403
432
|
(`libfuse2`), this package, the run user and the sudoers drop-in, or from a default image that the caller's `prepare` installs
|
|
404
433
|
them into. A container sandbox has no systemd (PID 1 is the Daytona daemon), so an in-box launcher keeps the instance
|
|
405
|
-
alive in place and never restarts it after exit 0, 65, 70, 75 or 76.
|
|
434
|
+
alive in place and never restarts it after exit 0, 65, 70, 75 or 76. Every incarnation of an attempt gets the attempt's
|
|
435
|
+
token on stdin. A restart whose mount is still live reuses it, and one whose mount went with the failed incarnation mounts
|
|
436
|
+
again: after a takeover, that mount is `HeldError` (76, terminal). The token stays in the launcher's memory for the attempt. The mount token is uploaded to a root-only file and
|
|
406
437
|
handed to the instance on stdin, never put in the sandbox's environment, which Daytona stores and returns on every read.
|
|
407
438
|
Status comes from the sandbox state plus the launcher's own word, and `stop` drains the instance, then deletes the box. The
|
|
408
439
|
archil client needs the sandbox's egress to Archil's mount servers (Daytona's Tier 3 or 4, full egress; a domain allowlist
|
|
409
440
|
cannot mount). Where the kernel refuses every unmount (a Sysbox box), the claim's release checks the delegation in, moves the
|
|
410
|
-
mount aside (`archil-scoped retire`) and kills its daemon.
|
|
441
|
+
mount aside (`archil-scoped retire`) and kills its daemon. Before the move, the client must list no Active delegation
|
|
442
|
+
under the mount. Its own bookkeeping under `.archil/` doesn't count: the client makes `.archil/client-<n>` and its
|
|
443
|
+
`unlinked` whenever a file under the mount is deleted, and no checkin clears them once the mount is read-only. Measured live on Daytona's hosted `us` region, about 6 ms from
|
|
411
444
|
`aws-us-east-1`: a box takes about 15 s to start an instance, an exclusive mount 142 ms and a commit 8.55 ms p50; a
|
|
412
445
|
power-off, a freeze of the instance and its FUSE daemon, and a network partition were each taken over on another box (the
|
|
413
446
|
power-off through the orphaned delegation, the freeze and the partition through the lease, the partitioned box stopped by
|
|
@@ -549,6 +582,8 @@ mount token reaches the journal, and that nothing is left behind. It needs syste
|
|
|
549
582
|
| A zombie that never commits again | Its lease watchdog (a worker thread, so a main thread stuck in a FUSE request does not matter) kills its commands at 75 s without touching the mount | an in-flight command runs at most until the self-fence |
|
|
550
583
|
| The mount is taken away under a live instance (an operator's `umount -l`, a cleanup's `fusermount -u`) | The instance writes `run.json` and creates its directories through the directory it opened at acquire, and checks before each write that the run's path is still that mount; at the next heartbeat the path names the local directory under the mountpoint, so it exits 75 (`CLAIM_UNMOUNTED`) | nothing of its own is written to the local disk; as host lost |
|
|
551
584
|
| A run's mountpoint holds entries and nothing is mounted on it (files written by path after a mount vanished) | acquire moves the directory aside whole to `<mountRoot>/.stray/<run>-<tag>` (root-owned, 0700), puts an empty one in its place, logs one JSON line on stderr with the entry count (never a name or a byte of the files), and mounts; `MOUNTPOINT_NOT_EMPTY` when it cannot | nothing in the moved directory is read or brought into the run |
|
|
585
|
+
| The mountpoint holds entries written since the starting incarnation began, before its claim mounted (an app's factory writing under `ctx.root`, which the CLI runs before the mount) | acquire refuses the start: `MOUNTPOINT_WRITTEN`, exit 65 (terminal, since a restart would write them again), naming the first few as `<mp>/...` in at most 240 characters; nothing moves aside and nothing mounts | the writes stay where the app left them, never silently lost; write in `onOpen`. The two rules side by side: this incarnation's own fresh writes are named (they are what the app must fix); an earlier incarnation's leftovers are counted, never named (they may be another incarnation's files) |
|
|
586
|
+
| The mountpoint holds more entries than the stray check looks at (100,000), or one this process cannot read | acquire refuses the start: `MOUNTPOINT_UNCHECKED`, exit 65, naming none of the entries; nothing moves aside and nothing mounts. A directory keeps its time when a file in it is rewritten in place, so only a check that looked at every entry can rule out this incarnation's own writes | someone looks at the directory and moves it aside by hand |
|
|
552
587
|
| A stale archil daemon on the mountpoint with no mount (a client that failed at a token refresh drops its mount and keeps its process and control socket) | acquire kills it (`archil-scoped stale`) before mounting; archil's refusal ("an older Archil process is still running", "Failed to bind control socket") is `MOUNT_FAILED` (exit 1), not 76 | none |
|
|
553
588
|
| The FUSE daemon dies, the loop lives | I/O returns `ENOTCONN`, the claim is fenced, the instance exits 75 and the unit does not restart; the supervisor finds the claim orphaned, revokes it and starts an instance | as host lost |
|
|
554
589
|
| Archil is unavailable or its write buffer is full | The commit fails, pi poisons the session, the instance exits 75; the supervisor retries | nothing is published past the last durable commit |
|
|
@@ -566,7 +601,7 @@ about revocation, takeover and the lease's replacement do not apply. These do:
|
|
|
566
601
|
|
|
567
602
|
| Failure | What happens | Effects |
|
|
568
603
|
|---|---|---|
|
|
569
|
-
| The instance crashes or is SIGKILLed | Its owner lock goes with the process. The next open of the directory (by the app, or whatever restarts it) takes the run, ends the commands the killed incarnation left running (its `commands` step, before the
|
|
604
|
+
| The instance crashes or is SIGKILLed | Its owner lock goes with the process. The next open of the directory (by the app, or whatever restarts it) takes the run, ends the commands the killed incarnation left running (its `commands` step, before any code of the new incarnation runs), and `resume()`s from the last commit | running tasks go back to pending; effects in flight are `interrupted`, never rerun; no command of the killed incarnation still runs once the run resumes. pi makes each command the leader of its own process group, so the SIGKILL does not reach them and a local run has no unit cgroup: the instance records each command's group under `commands/` (by pid, start ticks and boot) and the next open kills what still matches. A command started within about 0.1 s before the kill can be missed |
|
|
570
605
|
| A process the killed incarnation's commands started does not die at the next open (an uninterruptible wait, or one the run's user may not signal) | The open is refused: `HeldError` `COMMANDS_ALIVE`, exit 76, with the message below; the records stay | none: the run does not resume beside it. End those processes and open the run again |
|
|
571
606
|
| Power loss or an OS crash | The store's commits are durable (SQLite at `synchronous = FULL`); `work/` is the operating system's to flush, since the claim's barrier syncs nothing | the store resumes from its last commit; files the agent wrote in `work/` can be older than what the store recorded |
|
|
572
607
|
| A second process opens the run on this machine | The owner lock is refused (`HeldError`, exit 76) | none |
|
|
@@ -664,8 +699,13 @@ then list the prefix and check it is empty.
|
|
|
664
699
|
|
|
665
700
|
## Library
|
|
666
701
|
|
|
667
|
-
`openDurableRun(ref, options)` is what `run` calls: mount, verify the claim, take the owner lock,
|
|
668
|
-
the store, check the seal, open and resume the
|
|
702
|
+
`openDurableRun(ref, options)` is what `run` calls: mount, verify the claim, take the owner lock, make `store/`, `work/`
|
|
703
|
+
and `tmp/`, end a killed incarnation's commands, write `run.json`, open the store, check the seal, open and resume the
|
|
704
|
+
Harness, keep the lease. Its `afterClaim(claim)` is awaited once the claim is held and before `run.json`, with the
|
|
705
|
+
claim's `ref`, `root`, `work` and `store`. What it returns (`harness`, `beforeResume`, `seed`) goes into the open, so
|
|
706
|
+
`harness` may come from there instead of the options, and a factory can give a new run its seed (over the options'
|
|
707
|
+
`seed`); the seed runs right after `afterClaim` and before `run.json`, on a new run only. A rejection fails the open with `run.json` as the open found it and lets the
|
|
708
|
+
claim go. `acquire`, `ensureRunning`, `localHost`,
|
|
669
709
|
`archilEnv` and the typed errors (`FencedError`, `HeldError`, `StoreBehindSealError`) are exported for apps that compose
|
|
670
710
|
their own lifecycle, and so are serve (`serveRun`, `requestRun`), parking (`watchParking`, `drain`, `busyState`, and
|
|
671
711
|
`leaseParkTarget` for a Harness a host opened over `openRunLease`) and `fork`. `src/index.ts` lists every export; the
|
|
@@ -693,6 +733,47 @@ pi-durable, chord and their dependencies).
|
|
|
693
733
|
`runPath`, `unmountClaim`, `createRunDir`, the mount-token and delegation helpers, `isLeaseFresh`, and the Daytona labels,
|
|
694
734
|
names and `sweepSandboxes`. It carries no compatibility promise: a name there can change or go in any release.
|
|
695
735
|
|
|
736
|
+
### Seeding a new run
|
|
737
|
+
|
|
738
|
+
`seed` (in `openDurableRun`'s options, and `openRunLease`'s) writes what a new run starts with before anything sees the
|
|
739
|
+
run: its store file, a SQLite database the Harness then opens (saved sign-in profiles, say), and files in its
|
|
740
|
+
workspace (`work/.agents/skills/`). `beforeResume` runs after the store is open, too late for that.
|
|
741
|
+
|
|
742
|
+
```ts
|
|
743
|
+
await openDurableRun(ref, {
|
|
744
|
+
...localRun("/var/runs/run-1"), // or a disk's mountToken
|
|
745
|
+
harness: { models, registry },
|
|
746
|
+
async seed({ storePath, workDir }) { // also root: the claim's root
|
|
747
|
+
const db = new DatabaseSync(storePath);
|
|
748
|
+
db.exec("CREATE TABLE IF NOT EXISTS x_agent_profiles (name TEXT PRIMARY KEY, profile BLOB)");
|
|
749
|
+
db.close(); // closed before it returns: the store opens it next
|
|
750
|
+
await cp(skillsDir, join(workDir, ".agents/skills"), { recursive: true });
|
|
751
|
+
},
|
|
752
|
+
});
|
|
753
|
+
```
|
|
754
|
+
|
|
755
|
+
- **When it runs.** Once, on the open that finds no `run.json`: inside the run's first claim, with the owner lock held,
|
|
756
|
+
after `store/`, `work/` and `tmp/` exist, a killed incarnation's commands are ended and `afterClaim` has run, before
|
|
757
|
+
`run.json` is written and before the store opens. Its step is `seed`. What it wrote is made durable (the claim's
|
|
758
|
+
barrier) before `run.json` publishes the run.
|
|
759
|
+
- **From the app.** `afterClaim` may return `seed` (as `AfterClaim`), over the options' one, so a factory that runs on the
|
|
760
|
+
claimed directory can give the run its starting state.
|
|
761
|
+
- **When it doesn't.** Never for a run that has `run.json`, whatever its generation, and never on a fork's first open
|
|
762
|
+
(`fork` writes `run.json`).
|
|
763
|
+
- **When it fails.** A seed that throws fails the open with its own error. A process killed while it seeds leaves the run
|
|
764
|
+
unpublished in the same way: no `run.json`, the claim let go. The next open seeds again, so a seed overwrites what an
|
|
765
|
+
earlier attempt wrote.
|
|
766
|
+
- **How long it may take.** A new run has no `run.json` heartbeat while it seeds. A supervisor keeps a start in its grace
|
|
767
|
+
(`--start-grace`, `startGraceMs`; default the lease expiry, 90 s) only while its start mark (`start.json`) is younger
|
|
768
|
+
than the grace. So while the seed runs, the instance rewrites the mark's time every heartbeat, through its claim, when a
|
|
769
|
+
supervisor's mark for generation 1 is there. It does so on a thread of its own, so a seed that holds its thread (a
|
|
770
|
+
synchronous import through `DatabaseSync`) is covered too. A live seed of any length stays `starting`. A seeder that dies stops
|
|
771
|
+
rewriting, the grace runs out as for any start, and the claim is revoked. If the claim is revoked while the seed still
|
|
772
|
+
runs, its next write or its barrier is refused, the instance is fenced (exit 75) with nothing published, and the next
|
|
773
|
+
start seeds again.
|
|
774
|
+
- **On a local directory** (`localRun`) it is the same. As with `work/`, the files it writes are the operating system's
|
|
775
|
+
to flush; a SQLite database it commits with `synchronous = FULL` is durable when the commit returns.
|
|
776
|
+
|
|
696
777
|
### A run on a local directory
|
|
697
778
|
|
|
698
779
|
`localRun(root)` returns the `acquire` and `claimDir` that put a run on a plain directory of this machine, with no mount
|
|
@@ -707,6 +788,14 @@ const run = await openDurableRun({ disk: "local", region: "local", id: "run-1" }
|
|
|
707
788
|
});
|
|
708
789
|
```
|
|
709
790
|
|
|
791
|
+
The run's status is its `run.json`, and `readLocalRunStatus(root)` (the directory `localRun` takes) reads it, read-only and safe beside a live instance, the way `readRunStatus` reads a disk's: the
|
|
792
|
+
same `RunRecord` (`status`, `generation`, `wakeAt`, `holder`, `sealedSeq`, `detail`), `null` when the run has no `run.json`
|
|
793
|
+
yet, and `RunRecordError` (`RUN_JSON_INVALID`) when it is malformed.
|
|
794
|
+
|
|
795
|
+
```ts
|
|
796
|
+
const status = await readLocalRunStatus("/var/runs/run-1"); // { status: "sleeping", wakeAt: "...", ... } or null
|
|
797
|
+
```
|
|
798
|
+
|
|
710
799
|
It is for one machine (a laptop, a CI job, a test), and says so: the claim is the directory itself, so nothing but this
|
|
711
800
|
machine's owner lock keeps a second process off the run (a second open while one holds it is refused), and nothing at
|
|
712
801
|
all keeps out a process on another machine that can reach the directory. Its barrier refuses once the run is fenced,
|
|
@@ -742,8 +831,129 @@ A local run works on macOS for development, and is weaker there than on Linux in
|
|
|
742
831
|
|
|
743
832
|
`fsync` on macOS does not flush the drive's cache, so the store also sets SQLite's `fullfsync` and
|
|
744
833
|
`checkpoint_fullfsync` there (SQLite then syncs with `F_FULLFSYNC`), and refuses to open unless they read back. CI runs
|
|
745
|
-
the no-key quickstart
|
|
746
|
-
the file tools' confinement) on macOS.
|
|
834
|
+
the no-key quickstart, the local run's tests (its command records, `run --dir` and `status --dir`, the lease watchdog,
|
|
835
|
+
the file tools' confinement) and the keyless examples on macOS.
|
|
836
|
+
|
|
837
|
+
### A short claim on any path, and a read of what it released
|
|
838
|
+
|
|
839
|
+
`withClaim` holds one path of a disk exclusively for a moment, outside any run: pi-durable-agent carries an agent's home
|
|
840
|
+
(`agent/agent.sqlite`) back at a run's end this way. `readReleased` reads a file such a claim released over the S3 API,
|
|
841
|
+
with no mount, so a reader never blocks a writer. Both are supervisor side: they take the `control` of
|
|
842
|
+
`archilControl({ apiKey, disk, region })`.
|
|
843
|
+
|
|
844
|
+
```ts
|
|
845
|
+
await withClaim({ control, disk: "dsk-...", region: "aws-us-east-1", path: "agent" }, async (held) => {
|
|
846
|
+
const db = new DatabaseSync(join(held.root, "agent.sqlite"));
|
|
847
|
+
db.exec("INSERT INTO runs (id) VALUES ('job-1'); PRAGMA wal_checkpoint(TRUNCATE)");
|
|
848
|
+
db.close();
|
|
849
|
+
});
|
|
850
|
+
const store = await readReleased({ control, path: "agent" }, "agent.sqlite", { sqlite: true }); // null: none yet
|
|
851
|
+
```
|
|
852
|
+
|
|
853
|
+
A claim is one exclusive mount, about 0.4 to 1 s with its release, so it is for a run's end, never a turn.
|
|
854
|
+
- **Exclusive by the mount.** Archil refuses a second client's mount of the path, an ancestor or a descendant. The call
|
|
855
|
+
mounts at a fresh `<mountRoot>/runs/<pid>-<uuid>` (default `/mnt/archil/.claims`; the shape `archil-scoped`'s cleanup
|
|
856
|
+
takes on a host that cannot unmount) and never shares a mount. Archil mounts no path whose directory is missing, so it
|
|
857
|
+
first makes the path's directories the disk lacks, over S3, as the calling process's own (0700; `owner` sets them). It writes
|
|
858
|
+
`.claim` with its deadline (`holdMs`, default 60 s), runs `fn`, then barriers, unmounts and gives its mount token back,
|
|
859
|
+
whether `fn` resolved or threw.
|
|
860
|
+
- **Lost holders are revoked first.** A holder whose client is gone (orphaned) is revoked before the mount, and so is a
|
|
861
|
+
holder of exactly the path whose `.claim` names a deadline that has passed and stays the same for 3 s (a holder that
|
|
862
|
+
just mounted rewrites it at once; a holder of a path above or below is never revoked by this path's `.claim`). A live holder within its deadline is waited for (`waitMs`, default 30 s), never
|
|
863
|
+
revoked. A holder revoked while still alive fails its next fsync, so its commit rolls back and its own call is FENCED:
|
|
864
|
+
the deadline never decides what is durable, the fence does.
|
|
865
|
+
- **What it refuses.** A path that is not relative and of safe segments, or that is under `runs/` (a run's own) or
|
|
866
|
+
`.archil/`, is `ClaimError` `INVALID_ARGUMENT`, before any call. A live holder after `waitMs` is `HeldError`
|
|
867
|
+
`CLAIM_HELD` (exit 76). The control API failing, or a delegation with no path it cannot attribute, is
|
|
868
|
+
`CONTROL_API_FAILED`. A claim revoked from under `fn` is `FencedError` (exit 75), and `fn`'s writes are then not
|
|
869
|
+
durable.
|
|
870
|
+
- **`CLAIM_HELD` is retryable.** The path is not released yet: retry within `waitMs`, and never fall back to an older
|
|
871
|
+
release.
|
|
872
|
+
|
|
873
|
+
`readReleased` returns the bytes only when the read is of a released file. Nothing holds the path (or an ancestor or a
|
|
874
|
+
descendant) before or after the read, and the object is unchanged across it (its etag, else its size and time). For
|
|
875
|
+
SQLite (`sqlite: true`), its `-wal` must also be absent or empty, so its writer checkpointed with `TRUNCATE` and closed. A
|
|
876
|
+
held path is `CLAIM_HELD`, retryable as above, and a WAL with content is `NOT_RELEASED`. A file that changed during every
|
|
877
|
+
one of `attempts` reads (default 3) is `CLAIM_HELD`. Open the bytes as a local file, read-only (`?immutable=1`).
|
|
878
|
+
|
|
879
|
+
### Places: a run a supervisor can kill and bring back
|
|
880
|
+
|
|
881
|
+
A place is where one run lives and how its instance starts, is killed and comes back, for a supervisor that decides when
|
|
882
|
+
to kill (agentrun's reference demo). Each factory takes its options and returns `(ctx) => Place`; `ctx.paths` is the run's
|
|
883
|
+
id and directory as the supervisor computed them.
|
|
884
|
+
|
|
885
|
+
- `localPlace({ app, root? })` runs `pi-durable-disk run --dir <root> --id <id> --app <app>` as a child in its own
|
|
886
|
+
process group, with `{ PATH, ...order.env }` and nothing else of the caller's environment. It needs no disk, no key and
|
|
887
|
+
no token. A process kill is SIGKILL to the group, and the next runner resumes the run from the directory. A box kill
|
|
888
|
+
also calls `ctx.destroyBoxes` (a deletion that failed is tried again by the next kill or recover). `release()` kills
|
|
889
|
+
what still runs, then, under the run's owner lock, ends the commands the run recorded (pi starts each in a process
|
|
890
|
+
group of its own, which the instance's kill does not reach). While another process holds the run, they are its own,
|
|
891
|
+
and release leaves them. With `ctx.requestFile`, `signals(attempt)` yields the kill request the runner writes
|
|
892
|
+
there when it parks at its planned crossing, and `requestFile` is that same file. `return()` ends a pending `next()`.
|
|
893
|
+
- `archilPlace({ disk, region, control, host })` runs the instance on the disk through a host driver set up with the app.
|
|
894
|
+
- `token(attempt)` mints that attempt's mount token through `control`, and the runner handed that token mounts with it.
|
|
895
|
+
- A start is `ensureRunning` with the token (`mountToken`) and the order's environment (`StartAttempt.env`).
|
|
896
|
+
- A kill is the driver's `kill`, and `kills` is the driver's.
|
|
897
|
+
- A recovery is the takeover: the killed holder's delegations are revoked whatever its lease says, and the next
|
|
898
|
+
instance starts. It reports `revoked` and `ms`. A newer incarnation than the one the place killed (another
|
|
899
|
+
supervisor's) gets the usual lease and start-grace checks instead, so a live one is `HELD`, not revoked.
|
|
900
|
+
- `status()` reads `run.json` over S3 and lists the run's delegations, plus, when its host driver makes boxes
|
|
901
|
+
(`daytonaHost`), a receipt for each box the place started, released ones included (`boxes`, `HostBox`). A receipt
|
|
902
|
+
has the provider, the kind, `from` and `to` (null while it lives). It also has the published list price per second
|
|
903
|
+
(`usdPerSecond`), the page that price was read from (`source`) and the ISO date it was read (`priced`). A box with
|
|
904
|
+
no price has none of those three, never a price of 0. No field holds the provider's id of the box.
|
|
905
|
+
- `recordHome` (when the control can list, as `archilControl`'s can): an attempt's runner records into
|
|
906
|
+
`recordHome.dir(attempt)`, `<root>/record/<attempt>`, in the run's directory, so its delegation covers it and
|
|
907
|
+
`deleteRunTree` removes it. `bringHome(into, { final })` reads that directory through the control (S3 list and get:
|
|
908
|
+
nothing takes the run's claim) and brings the recording into the supervisor's `into`.
|
|
909
|
+
- Blobs come first, each checked against the hash its name carries and written whole (a temporary file, fsync,
|
|
910
|
+
rename). One that doesn't match is skipped; everything else still comes home, then the call throws
|
|
911
|
+
`BLOB_MISMATCH`.
|
|
912
|
+
- Then the new whole lines of each `.spool/*.jsonl` segment, with an offset per file name: what `into` already
|
|
913
|
+
holds of it. So a second call, or a call from another process, brings nothing twice. A line waits while a blob it
|
|
914
|
+
names isn't home yet. On `final` it comes home anyway, and the call throws `CARRY_FAILED` naming the blob.
|
|
915
|
+
- Carriers that overlap on one `into` never append at once. The place runs one call at a time per destination, and
|
|
916
|
+
each append happens under a lock file beside its spool file (`.<name>.lock`, naming its holder's pid), after the
|
|
917
|
+
destination's size is read again, so only the lines past it are appended. A lock whose holder is gone is taken
|
|
918
|
+
over; one held past 5 s is `CARRY_FAILED`. A second `into` gets every line: what the place fetched for one
|
|
919
|
+
destination says nothing of another.
|
|
920
|
+
- A list or get that fails is `CARRY_FAILED`.
|
|
921
|
+
- The recording is on the disk, not in the box, so a final call after a box kill still brings home everything the
|
|
922
|
+
box fsynced: this place never throws `BOX_GONE`.
|
|
923
|
+
- With `ensure.create` (the run user's uid and gid), the place makes the EMPTY directories the runner writes into
|
|
924
|
+
at each start: `signals/`, `record/<attempt>/.spool/` and `record/<attempt>/blobs/`. It makes them over the
|
|
925
|
+
control, owned by the run user, mode 0755, in ensureRunning's window where no delegation holds the run. A
|
|
926
|
+
directory a process made by path on an Archil mount was seen to refuse file creates afterwards (ENOENT), and one
|
|
927
|
+
made over the control didn't. Every file in them stays the runner's. A refused put fails the start
|
|
928
|
+
(`DIRS_REFUSED`, as the cause of `HOST_START_FAILED`), and without an owner nothing is made.
|
|
929
|
+
- `requestFile(attempt)` is `<root>/signals/kill-<attempt>.json`, in the run's directory and outside `work/`, and
|
|
930
|
+
`signals(attempt)` reads `runs/<id>/signals/kill-<attempt>.json` over S3 every 500 ms (`signalPollMs`). A runner on
|
|
931
|
+
a box of its own can't reach the supervisor's files, so its request travels through the disk. The runner must write
|
|
932
|
+
it durably for S3 to see it: a temporary file, fsync, rename, fsync of the directory. A missing object is a request
|
|
933
|
+
not written yet; any other read failure is retried. Once reads have failed for `signalFailMs` (60 s) without a
|
|
934
|
+
break, the iterator yields one `{ kind: "signals-failed", code, message }` and ends, so a parked runner never waits
|
|
935
|
+
with no word of why. `code` is the failing read's own error code when it has one (`CONTROL_API_FAILED` for a read
|
|
936
|
+
that timed out), else `SIGNALS_FAILED`. The iterator never rejects.
|
|
937
|
+
- A request names `point` and `n`, and every other field it carries stays in the signal's `crossing`, except the
|
|
938
|
+
per-attempt nonce the supervisor put in the order's env. That one is lifted to the signal's own `nonce` (when it is a
|
|
939
|
+
string), where the supervisor reads it for its check, as from the local place. The place's token never goes into a
|
|
940
|
+
request: on a disk it is the mount token, and the app never holds it.
|
|
941
|
+
- `release()` stops every instance the place started and removes every token it minted. What it could not stop or
|
|
942
|
+
remove, it reports (`RELEASE_FAILED`) and tries again on the next `release()`. The run's directory stays
|
|
943
|
+
(`deleteRunTree`).
|
|
944
|
+
|
|
945
|
+
A runner's `exited` says how it ended. Locally that is the exit code or signal, and the outcome is the instance's
|
|
946
|
+
`{"outcome":{...}}` line on stdout. Without that line, or on a disk (a box's stdout is out of reach), the outcome is
|
|
947
|
+
`run.json`'s, with its generation, and `code`, `message`, `record` and `storeSha256` from the detail of the last status
|
|
948
|
+
the app set. Its `status` is the run's word, not the record's, so a supervisor needs no package vocabulary: `complete`
|
|
949
|
+
when the run is `done` and that detail carries a record, `failed` for anything else (done with no record, failed, or a
|
|
950
|
+
run its instance left `running`, `paused` or `sleeping`). So an app that runs on a box and wants its supervisor to have
|
|
951
|
+
its record puts the record there, in its final `setStatus("done", { record })`.
|
|
952
|
+
|
|
953
|
+
Either way the outcome may carry `models`, the role-to-model ids the run's live parts chose (`lead`, `eyes`, ...), on any
|
|
954
|
+
status, a failed run's too: from the outcome line's `models`, or from the final detail's (`setStatus("done", { record,
|
|
955
|
+
models })`). Both places keep it only when it is a plain object whose values are all strings; a missing or malformed one
|
|
956
|
+
is left out, never replaced by `{}`.
|
|
747
957
|
|
|
748
958
|
## Tests
|
|
749
959
|
|
package/dist/app.d.ts
CHANGED
|
@@ -26,7 +26,11 @@ export type AppOptions = OpenDurableRunOptions["harness"] & {
|
|
|
26
26
|
/** Record a parking run's wake (park.ts `ParkOptions.wake`); default run.json `sleeping` with `wakeAt`. */
|
|
27
27
|
wake?(run: DurableRun, at: number | null): Promise<void>;
|
|
28
28
|
};
|
|
29
|
-
/**
|
|
29
|
+
/**
|
|
30
|
+
* What `--app` names: `export default async (ctx) => ({ registry, models, onOpen })`. The CLI calls it once the claim is
|
|
31
|
+
* held, so `ctx.root` is the live run directory and the factory may write under it; the Harness opens with what it
|
|
32
|
+
* returns.
|
|
33
|
+
*/
|
|
30
34
|
export type AppModule = (ctx: AppContext) => AppOptions | Promise<AppOptions>;
|
|
31
35
|
/**
|
|
32
36
|
* The app module is missing, does not load, has no default export function, throws, returns no options, or its
|