@parcha/pi-durable-disk 0.1.0-beta.18 → 0.1.0-beta.20
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +31 -0
- package/README.md +176 -892
- package/dist/cli.d.ts +4 -2
- package/dist/cli.js +27 -1
- package/dist/run.d.ts +2 -0
- package/dist/run.js +3 -0
- package/dist/status.js +19 -8
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -1,109 +1,60 @@
|
|
|
1
1
|
# @parcha/pi-durable-disk
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
run's state on a disk that outlives the machine. Each run is one directory on the disk that holds pi's SQLite store and the
|
|
5
|
-
agent's workspace. One process at a time holds that directory (a second claim is refused), every commit is on the disk before
|
|
6
|
-
anything is shown, a host that is lost or hung is fenced by revoking its claim, and a stateless supervisor restarts the run on
|
|
7
|
-
another host, where pi-durable resumes from the last commit and never repeats a tool call whose intent it had already committed.
|
|
3
|
+
Run a [pi-durable](https://www.npmjs.com/package/@earendil-works/pi-durable) agent from a directory or a network disk, so any process or host can open the run and carry on where the last one released it.
|
|
8
4
|
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
release names no other backend and promises none.
|
|
5
|
+
An agent's home and its runs live on the disk, not on a machine: the conversation, the agent's files and what it was waiting
|
|
6
|
+
for are all in the run's directory, so a process is somewhere a run visits, not where it lives. This is a beta
|
|
7
|
+
(`0.1.0-beta.N`): a change that breaks a caller is listed in [CHANGELOG.md](CHANGELOG.md) when it ships.
|
|
13
8
|
|
|
14
|
-
|
|
15
|
-
machine. A Cloudflare Durable Object gives it both. This package gives it both on an Archil disk you mount on any Linux host.
|
|
9
|
+
## What it is
|
|
16
10
|
|
|
17
|
-
|
|
11
|
+
- **A run is one directory.** It holds pi's SQLite store (`store/run.sqlite`: the conversation and every task), the agent's
|
|
12
|
+
workspace (`work/`) and `run.json` (its status, generation and holder). [Layout on the disk](docs/reference.md#layout-on-the-disk).
|
|
13
|
+
- **One process at a time holds it.** A second claim is refused. The process that opens a run keeps a lease on it, and a
|
|
14
|
+
holder that lost the claim cannot write any more.
|
|
15
|
+
- **Release and open.** `release()` seals the run, and the instance ends: returning from `onOpen` after `await run.release()` is
|
|
16
|
+
enough (the examples call `process.exit(0)`, which is fine too). The next process to open the directory is the next generation: it reopens
|
|
17
|
+
the store, and pi-durable resumes the conversation and its tasks from the last commit.
|
|
18
|
+
- **Two places a run can live.** On a plain directory of this machine (`localRun`: no disk, no key, one machine), or on an
|
|
19
|
+
[Archil](https://archil.com) disk that a Linux host or a container can mount (your own disk: [below](#bring-your-own-disk)).
|
|
20
|
+
The app code is the same.
|
|
18
21
|
|
|
19
|
-
|
|
20
|
-
|---|---|---|
|
|
21
|
-
| Linux and Node 22.19 or later, nothing else | [Quickstart on one machine](#quickstart-on-one-machine-no-disk-no-key) | A durable chat on a directory of this machine, killed with `SIGKILL` and finished by a second process. One machine and no fence: the directory is the claim |
|
|
22
|
-
| Linux and Node 22.19 or later, and an app module | `pi-durable-disk run --dir PATH --app MODULE` ([The CLI](#the-cli)) | Your app, or example 01's (`examples/01-durable-chat/app.ts`), as an instance on a directory of this machine: `kill -9` it and run the same command again, and it resumes. One machine and no fence |
|
|
23
|
-
| macOS and Node 22.19 or later, for development | The same [quickstart](#quickstart-on-one-machine-no-disk-no-key) and `run --dir` | The same runs, development grade: commands keep your rights, and the file tools' confinement holds against the agent's paths but not a command racing it ([On macOS](#on-macos)). Use Linux or Docker when the agent is not trusted |
|
|
24
|
-
| Docker (macOS or Linux) and an Archil account | [Quickstart with Docker](#quickstart-with-docker) | A host killed in the middle of a paid charge, and the run resumed on another, every instance in a container |
|
|
25
|
-
| A Linux host with FUSE and an Archil account | [Quickstart on a Linux host](#quickstart-on-a-linux-host-kill-a-host-watch-the-run-resume-on-another), then [Setting up a host for production](#setting-up-a-host-for-production) | The production shape: systemd units, the supervisor, and a lost host's run back in about a second |
|
|
26
|
-
| An app or a host of your own | [The app](#the-app) and [Library](#library) | `openDurableRun`, `localRun`, the supervisor and host drivers, `serve`, parking and `fork` from your code |
|
|
22
|
+
## When to use it
|
|
27
23
|
|
|
28
|
-
|
|
29
|
-
and no key.
|
|
24
|
+
Use it when a run should not be tied to the machine it started on:
|
|
30
25
|
|
|
31
|
-
|
|
26
|
+
- **Your compute is disposable or changes.** Containers, sandboxes, VMs, deploys: whichever host opens the run next carries
|
|
27
|
+
on, with no state to copy first. Hosts: systemd units on this machine, Docker containers, Daytona sandboxes (a driver for
|
|
28
|
+
another compute is [three calls](docs/reference.md#host-drivers-and-the-supervisor)).
|
|
29
|
+
- **The agent waits.** A run whose work only sleeps in a retry or a long poll (longer than the park threshold, 60 s with the local
|
|
30
|
+
and Docker drivers) is released, holds no process, and is started again at its deadline. With `--serve`, a run with no work at
|
|
31
|
+
all is released too and a request starts it ([serve and parking](docs/reference.md#serve-parking-and-fork)).
|
|
32
|
+
- **You move or branch a run.** Hand it to another host at a step boundary, or fork a sealed run into many, each continuing
|
|
33
|
+
on its own.
|
|
34
|
+
- **You want to look at a run from outside.** Its record is a file: `readLocalRunStatus(dir)` on a directory,
|
|
35
|
+
`readRunStatus(control, id)` on a disk.
|
|
32
36
|
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
| One live instance per id | One exclusive mount of `runs/<id>/`; a second mount is refused | the Archil server |
|
|
36
|
-
| A stale instance is fenced | The supervisor revokes the claim; the old holder's next fsync fails with EIO, its SQLite commit fails, pi poisons the session, the instance kills its commands and exits 75 | Archil, pi and this package |
|
|
37
|
-
| Knowing an instance is gone | A lease (a heartbeat in `run.json`), Archil's orphan flag for a dead client, and the host driver's own status | this package |
|
|
38
|
-
| Storage bound to the instance | pi's `SqliteStorage` at `runs/<id>/store/run.sqlite`, on the claimed mount, at `synchronous = FULL` | pi |
|
|
39
|
-
| A write is durable before the response leaves | `commit()` resolves after an Archil fsync, pi publishes only after the commit, and an effect starts only after its intent is committed | pi and Archil |
|
|
40
|
-
| The input gate | pi's single commit line | pi |
|
|
41
|
-
| Eviction and resume | Any host that claims the directory reopens the store and `resume()`s | pi and this package |
|
|
42
|
-
| Alarms | On the same host, the host's process supervisor restarts the instance; across hosts, `supervise`. pi's own retry and poll waits stay in process | this package |
|
|
43
|
-
| A container to exec in | Tools run as child processes of the instance, against the run's `work/` directory on the same mount | the host |
|
|
44
|
-
| Hibernate and wake on request | The instance parks (releases the claim) while everything waits, and `requestRun` starts it again on a request: `serve`, parking, below | this package |
|
|
45
|
-
|
|
46
|
-
## Requirements
|
|
47
|
-
|
|
48
|
-
- Either Docker, on any operating system: each instance then runs in a container of this package's image
|
|
49
|
-
(`docker/Dockerfile`, see [Quickstart with Docker](#quickstart-with-docker)), and nothing else below applies to the
|
|
50
|
-
machine; or the following on a Linux host.
|
|
51
|
-
- Linux with systemd and FUSE (`/dev/fuse`, `libfuse2`, which the archil package depends on, and `fusermount` from `fuse3`,
|
|
52
|
-
which cleans a dead mount), `setpriv` (util-linux) and `sudo`, near the disk's region: a commit is one round trip to the
|
|
53
|
-
region (3 to 6 ms p50 from a VM next to it, measured).
|
|
54
|
-
- Node 22.19 or later, to run the package and its unit tests.
|
|
55
|
-
- The [`archil` client](https://docs.archil.com/mounting/linux) 0.8.40 or later (0.8.42 tested):
|
|
56
|
-
`curl -s https://archil.com/install | sh`.
|
|
57
|
-
- Root for the mounts: every `archil` verb needs it on client 0.8.42. The package reaches root only through
|
|
58
|
-
`sudo -n` of `bin/archil-scoped` (below), so the instance and the agent's tools run unprivileged.
|
|
59
|
-
- `/proc` that shows the instance root's processes: acquire's stale-daemon cleanup needs `/proc` visibility of root
|
|
60
|
-
processes, so mount `/proc` without `hidepid`, or run the instance where it can see them. Otherwise an archil daemon
|
|
61
|
-
left on a run's mountpoint fails every acquire there with `MOUNT_FAILED` (exit 1) until something else ends it.
|
|
62
|
-
- An Archil account, a disk, and an API key for the supervisor, all from the Archil console. Create the disk in the region
|
|
63
|
-
nearest the host (any size; its id looks like `dsk-0123456789abcdef` and its region like `aws-us-east-1`). Hosts and
|
|
64
|
-
instances never hold the API key.
|
|
65
|
-
- `@earendil-works/pi-durable` and `@earendil-works/chord`, `>=1.0.4 <1.4.0`, as peer dependencies. The package is developed against 1.1.0, and CI runs its whole suite on 1.0.4 too, the lowest version the range accepts, and on 1.2 and 1.3.
|
|
66
|
-
|
|
67
|
-
Not covered yet: a Kubernetes host driver (the `HostDriver` interface is below) and running the loop across machines
|
|
68
|
-
where FUSE is unavailable. Without Archil, a run can live on a plain directory of one machine
|
|
69
|
-
([A run on a local directory](#a-run-on-a-local-directory)).
|
|
37
|
+
It is not for a one-shot script with no state to keep (pi-durable's in-memory storage does that), and a disk other than
|
|
38
|
+
Archil is not supported: the package needs the disk to give an exclusive claim and a fence, and Archil is the one that does.
|
|
70
39
|
|
|
40
|
+
## Try it: no disk, no key
|
|
71
41
|
|
|
72
|
-
|
|
42
|
+
On Linux with Node 22.19 or later (or on macOS [for development](docs/reference.md#on-macos)). In an empty directory:
|
|
73
43
|
|
|
74
44
|
```sh
|
|
75
|
-
npm install @parcha/pi-durable-disk@beta @earendil-works/pi-durable
|
|
76
|
-
```
|
|
77
|
-
|
|
78
|
-
The pi packages are pinned to 1.2: pi 1.3.0 pins its own packages to exact versions, and this release's ranges stop below 1.3 until the next one.
|
|
79
|
-
|
|
80
|
-
From a checkout (this package is `packages/pi-durable-disk` of the repository; the root build compiles `dist/`):
|
|
81
|
-
|
|
82
|
-
```sh
|
|
83
|
-
git clone https://github.com/Parcha-ai/agentrun.git && cd agentrun && npm ci --ignore-scripts && npm run build
|
|
84
|
-
cd packages/pi-durable-disk
|
|
45
|
+
npm install @parcha/pi-durable-disk@beta @earendil-works/pi-durable @earendil-works/pi-ai @earendil-works/chord
|
|
85
46
|
```
|
|
86
47
|
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
instead of a key. One process answers the first messages and is killed (`SIGKILL`) after its third answer; a second
|
|
92
|
-
opens the same run and answers the rest, and the transcript, read back from the store, has every message answered once.
|
|
93
|
-
The directory is the claim, so this guards one machine only; the quickstarts below put the run on a disk that outlives
|
|
94
|
-
the machine.
|
|
95
|
-
|
|
96
|
-
In an empty directory, install the package and its peers, save this as `chat.mjs`
|
|
97
|
-
(it is [examples/01-durable-chat/on-a-directory.mjs](examples/01-durable-chat/on-a-directory.mjs)):
|
|
98
|
-
|
|
99
|
-
```sh
|
|
100
|
-
npm install @parcha/pi-durable-disk@beta @earendil-works/pi-durable@~1.2.0 @earendil-works/pi-ai@~1.2.0 @earendil-works/chord@~1.2.0
|
|
101
|
-
```
|
|
48
|
+
Save this as `pick-up.mts` (Node 22.19 or later runs the types away; `.mts` makes it an ES module whatever your
|
|
49
|
+
`package.json` says). It is [examples/pick-up-a-run/main.ts](examples/pick-up-a-run/main.ts). It starts three processes, one after another, on
|
|
50
|
+
one directory. Each opens the run, answers one question of the same chat, and releases it. pi-ai's faux model answers, so it
|
|
51
|
+
needs no key:
|
|
102
52
|
|
|
103
|
-
```
|
|
104
|
-
//
|
|
105
|
-
//
|
|
106
|
-
//
|
|
53
|
+
```ts
|
|
54
|
+
// Pick up a run: a run is a directory, and whatever process opens it carries on where the last one released it. No disk, no
|
|
55
|
+
// key (pi-ai's faux model answers). Three processes, one after another, each open the same run, answer one question of the
|
|
56
|
+
// same chat and release it; the last still knows the first question, because the chat is in the directory, not in a process.
|
|
57
|
+
// One machine: the directory is the claim.
|
|
107
58
|
import { spawn } from "node:child_process";
|
|
108
59
|
import { rmSync } from "node:fs";
|
|
109
60
|
import { join } from "node:path";
|
|
@@ -112,18 +63,20 @@ import { fileURLToPath } from "node:url";
|
|
|
112
63
|
import { BACKGROUND_CONTEXT as ctx } from "@earendil-works/chord/context";
|
|
113
64
|
import { createModels, fauxAssistantMessage, fauxProvider } from "@earendil-works/pi-ai";
|
|
114
65
|
import { createRegistry } from "@earendil-works/pi-durable";
|
|
115
|
-
import {
|
|
66
|
+
import type { EntryRecord } from "@earendil-works/pi-durable";
|
|
67
|
+
import { localRun, openDurableRun, readLocalRunStatus } from "@parcha/pi-durable-disk";
|
|
116
68
|
|
|
117
|
-
const
|
|
118
|
-
const root = join(process.cwd(), "
|
|
119
|
-
const text = (message
|
|
69
|
+
const QUESTIONS = ["Where does your transcript live?", "Who had this run before you?", "Do you remember my first question?"];
|
|
70
|
+
const root = join(process.cwd(), "pick-up-a-run-run");
|
|
71
|
+
const text = (message: { content: unknown }): string =>
|
|
72
|
+
typeof message.content === "string" ? message.content : (message.content as { text?: string }[]).map((part) => part.text ?? "").join("");
|
|
120
73
|
|
|
121
74
|
if (process.argv[2] === "instance") {
|
|
122
75
|
let generation = 0;
|
|
123
76
|
const faux = fauxProvider();
|
|
124
77
|
faux.setResponses(Array.from({ length: 100 }, () => (context) => {
|
|
125
78
|
const asked = context.messages.filter((message) => message.role === "user");
|
|
126
|
-
return fauxAssistantMessage(`Answer ${asked.length}, by generation ${generation}: "${text(asked
|
|
79
|
+
return fauxAssistantMessage(`Answer ${asked.length}, by generation ${generation}. You first asked: "${text(asked[0]!)}"`);
|
|
127
80
|
}));
|
|
128
81
|
const models = createModels();
|
|
129
82
|
models.setProvider(faux.provider);
|
|
@@ -131,843 +84,174 @@ if (process.argv[2] === "instance") {
|
|
|
131
84
|
generation = run.generation;
|
|
132
85
|
const model = faux.getModel();
|
|
133
86
|
const chat = await run.harness.root(ctx, { agent: { model: { provider: model.provider, modelId: model.id } } });
|
|
134
|
-
const answers = async () => (await chat.entries({}, 100, undefined, ctx)).items.reverse().filter((entry) => entry.kind === "pi.assistant");
|
|
135
|
-
const
|
|
136
|
-
console.log(`generation ${generation} opened the run
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
if (i >= before) console.log(`generation ${generation} answered message ${i + 1}`);
|
|
141
|
-
await new Promise((resolve) => setTimeout(resolve, 200));
|
|
142
|
-
}
|
|
143
|
-
for (const answer of await answers()) console.log(` ${text(answer.model[0])}`);
|
|
144
|
-
await run.setStatus("done");
|
|
87
|
+
const answers = async (): Promise<EntryRecord[]> => [...(await chat.entries({}, 100, undefined, ctx)).items].reverse().filter((entry) => entry.kind === "pi.assistant");
|
|
88
|
+
const n = (await answers()).length + 1; // the next question: this process answers one and lets go
|
|
89
|
+
console.log(`generation ${generation} opened the run (${n - 1} answered before)`);
|
|
90
|
+
await (await chat.submit({ type: "input", content: QUESTIONS[n - 1], requestId: `question-${n}` }, ctx)).wait(ctx);
|
|
91
|
+
console.log(` ${QUESTIONS[n - 1]}\n ${text((await answers()).at(-1)!.model![0]!)}`);
|
|
92
|
+
await run.setStatus(n === QUESTIONS.length ? "done" : "paused");
|
|
145
93
|
await run.release();
|
|
146
94
|
process.exit(0);
|
|
147
95
|
}
|
|
148
96
|
|
|
149
|
-
// The supervisor:
|
|
150
|
-
//
|
|
97
|
+
// The supervisor: one process per question, one after another, on the same directory. It reads whole lines, and waits for
|
|
98
|
+
// `close` (the process gone and all its output read), not `exit`.
|
|
151
99
|
rmSync(root, { recursive: true, force: true });
|
|
152
|
-
|
|
153
|
-
const
|
|
100
|
+
console.log(`the run is ${root}`);
|
|
101
|
+
const output: string[] = [];
|
|
102
|
+
for (const [i] of QUESTIONS.entries()) {
|
|
103
|
+
const child = spawn(process.execPath, [fileURLToPath(import.meta.url), "instance"], { stdio: ["ignore", "pipe", "inherit"] });
|
|
154
104
|
createInterface({ input: child.stdout }).on("line", (line) => {
|
|
155
105
|
console.log(line);
|
|
156
|
-
|
|
106
|
+
output.push(line);
|
|
157
107
|
});
|
|
158
|
-
const
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
const
|
|
165
|
-
const
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
const match = /^ {2}Answer (\d+), by generation (\d+)/.exec(line);
|
|
170
|
-
return match ? [{ n: Number(match[1]), generation: Number(match[2]) }] : [];
|
|
171
|
-
});
|
|
172
|
-
const once = MESSAGES.every((_, i) => answered.filter((a) => a.n === i + 1).length === 1) && answered.length === MESSAGES.length;
|
|
173
|
-
const generations = [...new Set(answered.map((a) => a.generation))].sort();
|
|
174
|
-
if (code !== 0 || !once || generations.length < 2) {
|
|
175
|
-
console.error(`not ok: exit ${code}, answers ${JSON.stringify(answered)}`);
|
|
108
|
+
const code = await new Promise<number | null>((resolve) => child.on("close", resolve));
|
|
109
|
+
if (code !== 0) {
|
|
110
|
+
console.error(`not ok: the process for question ${i + 1} exited ${code}`);
|
|
111
|
+
process.exit(1);
|
|
112
|
+
}
|
|
113
|
+
}
|
|
114
|
+
const opened = output.flatMap((line) => /^generation (\d+) opened the run \((\d+) answered before\)$/.exec(line)?.slice(1).map(Number) ?? []);
|
|
115
|
+
const status = await readLocalRunStatus(root); // the run's record, read from outside, with no process running
|
|
116
|
+
const remembered = output.at(-1)?.includes(`You first asked: "${QUESTIONS[0]}"`);
|
|
117
|
+
if (opened.join() !== "1,0,2,1,3,2" || !remembered || status?.status !== "done" || status.generation !== 3) {
|
|
118
|
+
console.error(`not ok: opened ${JSON.stringify(opened)}, remembered ${remembered}, status ${JSON.stringify(status && [status.status, status.generation])}`);
|
|
176
119
|
process.exit(1);
|
|
177
120
|
}
|
|
178
|
-
console.log(`ok:
|
|
121
|
+
console.log(`ok: three processes, one run: each picked up where the last released it, the last still knew the first question, and the run reads done at generation 3 (${root})`);
|
|
179
122
|
```
|
|
180
123
|
|
|
181
|
-
|
|
124
|
+
Run it (about two seconds; it writes the run under `./pick-up-a-run-run/`):
|
|
182
125
|
|
|
183
126
|
```sh
|
|
184
|
-
node
|
|
127
|
+
node pick-up.mts
|
|
185
128
|
```
|
|
186
129
|
|
|
187
130
|
It ends with:
|
|
188
131
|
|
|
189
132
|
```
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
Answer
|
|
193
|
-
ok:
|
|
133
|
+
generation 3 opened the run (2 answered before)
|
|
134
|
+
Do you remember my first question?
|
|
135
|
+
Answer 3, by generation 3. You first asked: "Where does your transcript live?"
|
|
136
|
+
ok: three processes, one run: each picked up where the last released it, the last still knew the first question, and the run reads done at generation 3 (<this directory>/pick-up-a-run-run)
|
|
194
137
|
```
|
|
195
138
|
|
|
196
|
-
|
|
197
|
-
|
|
198
|
-
|
|
199
|
-
account (the free plan is enough): from the Archil console, a disk for scratch runs and an API key. This machine needs no
|
|
200
|
-
root, no FUSE and no archil client: each instance of a run is a container of this package's image, and that container is
|
|
201
|
-
the instance's machine.
|
|
139
|
+
Nothing but the directory carried the conversation from one process to the next, so the third process knew the first
|
|
140
|
+
question. The directory is the claim here, which guards one machine only; the next section puts the run on a disk that hosts
|
|
141
|
+
can mount.
|
|
202
142
|
|
|
203
|
-
|
|
204
|
-
git clone https://github.com/Parcha-ai/agentrun.git && cd agentrun
|
|
205
|
-
export ARCHIL_API_KEY=... # Archil console, API keys; only the supervisor on this machine reads it
|
|
206
|
-
export ARCHIL_DISK=dsk-... # the disk's id
|
|
207
|
-
export ARCHIL_REGION=aws-us-east-1 # the disk's region
|
|
208
|
-
packages/pi-durable-disk/examples/docker-quickstart.sh
|
|
209
|
-
```
|
|
143
|
+
## Bring your own disk
|
|
210
144
|
|
|
211
|
-
The
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
3. it runs example 02 with `--host docker`. An agent charges six invoices through a fake paid API, and host A's container
|
|
216
|
-
is killed (`docker kill`) while the third charge is in flight. Host B's container takes over and finishes;
|
|
217
|
-
4. it cleans up: the run's containers, its directory on the disk, its token users, and the tarball it packed.
|
|
218
|
-
|
|
219
|
-
It ends with a table of what the paid API received against what the run recorded: each charge was received once, and the
|
|
220
|
-
charge that was in flight is recorded `interrupted` and was not sent again. A first run takes about two minutes, most of it
|
|
221
|
-
building the image (826 MB, 250 MB of it the archil client; `docker image rm pi-durable-disk:local` removes it). Later
|
|
222
|
-
runs take about 15 seconds. Measured on Linux (Docker Engine 29 on Ubuntu 24.04); the macOS case (Docker Desktop,
|
|
223
|
-
OrbStack, Colima, and the arm64 image on Apple silicon) is unverified until the script has been run on a Mac. Then try
|
|
224
|
-
`packages/pi-durable-disk/examples/docker-quickstart.sh freeze`: host A's container is frozen (`docker pause`), and once
|
|
225
|
-
the lease expires host B's supervisor stops it and takes over. From `packages/pi-durable-disk`, `node
|
|
226
|
-
examples/01-durable-chat/demo.ts --host docker` runs a chat that continues on the other host.
|
|
227
|
-
|
|
228
|
-
To run your own app the same way:
|
|
145
|
+
The run's directory moves to an Archil disk, and the same app module runs on it (`run --disk` where `run --dir` was). You need an Archil account, a disk in the region nearest your hosts, and an API key from the Archil console. The key stays with the supervisor:
|
|
146
|
+
hosts and instances never hold it. The supervisor is a function you call (`ensureRunning`) or `pi-durable-disk supervise`, which starts an
|
|
147
|
+
instance where a run has none and hands each instance a mount token for its own start. On a Linux host set up as in
|
|
148
|
+
[Requirements](docs/reference.md#requirements) (add `--host docker --image IMAGE` to start each instance in a container instead):
|
|
229
149
|
|
|
230
150
|
```sh
|
|
231
|
-
|
|
232
|
-
|
|
151
|
+
export ARCHIL_API_KEY=... # Archil console, API keys; only the supervisor reads it
|
|
152
|
+
npx pi-durable-disk supervise --disk dsk-0123456789abcdef --region aws-us-east-1 --id my-run --create --app ./my-app.ts --every 30s
|
|
233
153
|
```
|
|
234
154
|
|
|
235
|
-
|
|
236
|
-
|
|
237
|
-
`node_modules` in that directory is hidden. If your app needs other packages, build an image `FROM` this one that
|
|
238
|
-
installs them under `/opt/pda`.
|
|
239
|
-
|
|
240
|
-
## Quickstart on a Linux host: kill a host, watch the run resume on another
|
|
155
|
+
`./my-app.ts` is [an app module](docs/reference.md#the-app): a function that returns pi's Harness options. Start from one of these, by what
|
|
156
|
+
you have:
|
|
241
157
|
|
|
242
|
-
|
|
243
|
-
|
|
244
|
-
|
|
158
|
+
- **Docker, any operating system, and an Archil account:** [Quickstart with Docker](docs/reference.md#quickstart-with-docker). No root, no
|
|
159
|
+
FUSE, no archil client on your machine: each instance is a container.
|
|
160
|
+
- **A Linux host with FUSE and the archil client:** [Quickstart on a Linux host](docs/reference.md#quickstart-on-a-linux-host-two-hosts-one-run),
|
|
161
|
+
then [Setting up a host for production](docs/reference.md#setting-up-a-host-for-production).
|
|
162
|
+
- **Requirements, in full:** [Requirements](docs/reference.md#requirements).
|
|
245
163
|
|
|
246
|
-
|
|
247
|
-
|
|
248
|
-
|
|
249
|
-
|
|
164
|
+
**The disk.** The package needs two things from the disk: an exclusive claim on a run's directory, so that a second mount is
|
|
165
|
+
refused, and a fence, so that revoking the claim makes the old holder's writes fail. [Archil](https://archil.com) gives both,
|
|
166
|
+
which makes it the supported disk and the only one in this release. A disk without both cannot carry the package's
|
|
167
|
+
guarantees. This release names no other backend and promises none.
|
|
168
|
+
|
|
169
|
+
## What you can do with it
|
|
170
|
+
|
|
171
|
+
- **Hand a run to another host** at a step boundary: the old host records where the run is going and releases it, the new
|
|
172
|
+
one opens the next generation and tells the agent ([planned-move](examples/planned-move/)).
|
|
173
|
+
- **Park a run while it waits**, and wake it at its deadline or on a request ([serve-and-park](examples/serve-and-park/)).
|
|
174
|
+
- **Serve a run over HTTP**: `POST /submit` with a `requestId` that is deduplicated in the run's store, so a retry that reaches
|
|
175
|
+
another host gets the same submission ([Serve, parking and fork](docs/reference.md#serve-parking-and-fork)).
|
|
176
|
+
- **Fork a sealed run into many**, each fork a run of its own ([fork-into-many](examples/fork-into-many/)).
|
|
177
|
+
- **Seed a new run** with a store and files before anything sees it ([Seeding a new run](docs/reference.md#seeding-a-new-run)).
|
|
178
|
+
- **Run it supervised** on systemd units, Docker containers or Daytona sandboxes ([Host drivers and the
|
|
179
|
+
supervisor](docs/reference.md#host-drivers-and-the-supervisor)).
|
|
180
|
+
- **Keep what was committed.** Every commit is on the disk before anything is shown, and an effect whose intent was committed is never started twice.
|
|
181
|
+
|
|
182
|
+
[examples/](examples/README.md) lists every example, one property each, with what each needs and the first version that has
|
|
183
|
+
what it imports.
|
|
184
|
+
|
|
185
|
+
## What it costs
|
|
186
|
+
|
|
187
|
+
- **A local run costs nothing:** a directory of this machine, no account and no key. The model is yours: the examples use a
|
|
188
|
+
scripted one, so the first run spends nothing; the package adds no model calls of its own.
|
|
189
|
+
- **On a disk:** an [Archil](https://archil.com) account and the disk's storage. A
|
|
190
|
+
run is one directory (a SQLite store, a workspace and `run.json`), removed with `deleteRunTree`. Every commit is one round
|
|
191
|
+
trip to the disk's region, 3 to 6 ms p50 from a host next to it (measured), so run hosts near the disk. Claiming a run
|
|
192
|
+
takes about 0.2 s and releasing one about 0.15 s, from the live suites on a VM next to the disk's region ([where the figures
|
|
193
|
+
come from](docs/reference.md#the-failure-table)).
|
|
194
|
+
- **Hosts:** whatever you run them on, a machine you own, Docker, or a [Daytona](https://www.daytona.io) sandbox, which Daytona
|
|
195
|
+
bills; a sandbox's list price per second is in its receipt (`daytonaUsdPerSecond`). A run whose work only waits longer
|
|
196
|
+
than the park threshold (60 s by default with the local and Docker drivers) is released and holds no process; an instance with
|
|
197
|
+
no work at all parks only with `--serve` or `--park-idle` ([parking](docs/reference.md#serve-parking-and-fork)). Otherwise it
|
|
198
|
+
stays up, and so does its host.
|
|
199
|
+
- **A supervisor** is a function you call or `pi-durable-disk supervise`; it needs the Archil API key, and no instance ever
|
|
200
|
+
holds it.
|
|
201
|
+
|
|
202
|
+
## What it does not do
|
|
203
|
+
|
|
204
|
+
- **Other disks.** Archil is the only disk in this release: the package needs an exclusive claim and a fence from it. No other
|
|
205
|
+
backend is named or promised.
|
|
206
|
+
- **Guard a run on a plain directory against another machine.** `localRun` has no fence: the directory is the claim, so it
|
|
207
|
+
keeps a second process on this machine off the run and nothing else. Use it for one machine; use a disk to share a run.
|
|
208
|
+
- **Sandbox an agent.** Commands run as the run user, with `no_new_privs` and the file tools confined to `work/`, and a command
|
|
209
|
+
can reach anything that user can. Run an agent you do not trust in your own sandbox ([Security](docs/reference.md#security)).
|
|
210
|
+
- **Make a third party idempotent.** The package never starts the same effect twice, but a call your model decides to retry is
|
|
211
|
+
a new call: give a paid API an idempotency key from your own business key.
|
|
212
|
+
- **Choose your model or hold its keys.** You bring the provider and the key; they stay in your app.
|
|
213
|
+
- **Run on every platform.** Linux is supported. macOS is for development ([On macOS](docs/reference.md#on-macos)) and Windows is
|
|
214
|
+
not supported. A Kubernetes host driver is not written (a driver is [three calls](docs/reference.md#host-drivers-and-the-supervisor)).
|
|
250
215
|
|
|
251
|
-
|
|
252
|
-
export ARCHIL_API_KEY=... # from the Archil console
|
|
253
|
-
export ARCHIL_DISK=dsk-... # the disk id or name
|
|
254
|
-
export ARCHIL_REGION=aws-us-east-1
|
|
216
|
+
## Install
|
|
255
217
|
|
|
256
|
-
|
|
257
|
-
|
|
218
|
+
```sh
|
|
219
|
+
npm install @parcha/pi-durable-disk@beta @earendil-works/pi-durable @earendil-works/pi-ai @earendil-works/chord
|
|
258
220
|
```
|
|
259
221
|
|
|
260
|
-
|
|
261
|
-
|
|
262
|
-
feature": pi's store uses `node:sqlite`. It is harmless.) Next: `node examples/02-paid-effect/demo.ts freeze` (the host's
|
|
263
|
-
mount hangs instead of dying) and `node examples/01-durable-chat/demo.ts` (a chat that continues on the other host).
|
|
264
|
-
Each example's README lists the exact commands the demo runs, so you can run them by hand.
|
|
265
|
-
|
|
266
|
-
## The app
|
|
267
|
-
|
|
268
|
-
An instance is `pi-durable-disk run --app <module> ...`: `openDurableRun` with the app's options. The module is an ES
|
|
269
|
-
module whose default export is a function of where the run lives and returns pi's Harness options without `env` (the run
|
|
270
|
-
builds `env` on its claim: the workspace is `<root>/work`), plus an optional `beforeResume` and an optional `onOpen`.
|
|
271
|
-
|
|
272
|
-
The factory runs once the claim is held: the run's directory is mounted, its owner lock taken, `store/`, `work/` and
|
|
273
|
-
`tmp/` made, and the commands a killed incarnation left running ended (`openDurableRun`'s `afterClaim`). This
|
|
274
|
-
incarnation's `run.json` comes after it. So `where.root` is the live directory whenever app code runs, and a factory may
|
|
275
|
-
write under it (a recorder's spool, a `signals/` directory). Before, the factory ran before the mount, and its writes
|
|
276
|
-
landed in the bare mountpoint and were lost. The Harness opens after the factory, with what it returns. Claim options, if
|
|
277
|
-
one ever needs them, are static (CLI flags or the environment) and never returned by the factory, since the claim is
|
|
278
|
-
taken before the factory runs. A module that does not load, or a factory that throws, fails the open with `APP_FAILED`
|
|
279
|
-
(exit 1), and the claim is let go. Such a start publishes nothing: a new run still has no `run.json`, and a resumed run
|
|
280
|
-
keeps its previous one. A supervisor therefore counts it a failed start and backs off (`ensureRunning`), never starting
|
|
281
|
-
it again on every tick.
|
|
222
|
+
`@earendil-works/pi-durable` and `@earendil-works/chord` are peer dependencies, `>=1.0.4 <1.4.0`. Node 22.19 or later;
|
|
223
|
+
Linux, or macOS for development.
|
|
282
224
|
|
|
283
|
-
|
|
284
|
-
import { BACKGROUND_CONTEXT as ctx } from "@earendil-works/chord/context";
|
|
285
|
-
import { createModels } from "@earendil-works/pi-ai/models";
|
|
286
|
-
import { openaiProvider } from "@earendil-works/pi-ai/providers/openai";
|
|
287
|
-
import { createRegistry } from "@earendil-works/pi-durable";
|
|
288
|
-
import { ArchilCodingTools } from "@parcha/pi-durable-disk";
|
|
289
|
-
import type { AppContext, AppOptions } from "@parcha/pi-durable-disk";
|
|
225
|
+
From a checkout (this package is `packages/pi-durable-disk` of the repository; the root build compiles `dist/`):
|
|
290
226
|
|
|
291
|
-
|
|
292
|
-
|
|
293
|
-
|
|
294
|
-
models.setProvider(openaiProvider()); // reads OPENAI_API_KEY from the instance's environment
|
|
295
|
-
const registry = createRegistry(); // every extension the run's tasks need
|
|
296
|
-
registry.install(ArchilCodingTools); // pi's read, write, edit and bash, with `read` declared replay-safe
|
|
297
|
-
return {
|
|
298
|
-
models,
|
|
299
|
-
registry,
|
|
300
|
-
async beforeResume({ harness, context, generation, previous }) {
|
|
301
|
-
// Awaited once per incarnation, after the Harness is open and before it resumes: no resumed task takes a step
|
|
302
|
-
// until this settles, and what it admits commits before their steps do. Reconcile here what a resumed tool depends on.
|
|
303
|
-
// previous is run.json as the run was left (null when there was none): `sleeping` with its wakeAt after a park,
|
|
304
|
-
// `paused` with its detail after a planned stop or a fork, `running` with the dead holder and no new seal after a crash.
|
|
305
|
-
},
|
|
306
|
-
async onOpen(run) {
|
|
307
|
-
// Called once per incarnation, after the run is open and resumed. run.generation is 1 on a first start and
|
|
308
|
-
// higher on every resume. run.harness is pi-durable's Harness.
|
|
309
|
-
const agent = { model: { provider: "openai", modelId: "gpt-6-sol" } };
|
|
310
|
-
const root = await run.harness.root(ctx, { agent });
|
|
311
|
-
await root.submit({ type: "input", content: "Summarize work/notes.md", requestId: "job-1" }, ctx); // a retry is a no-op
|
|
312
|
-
},
|
|
313
|
-
};
|
|
314
|
-
}
|
|
227
|
+
```sh
|
|
228
|
+
git clone https://github.com/Parcha-ai/agentrun.git && cd agentrun && npm ci --ignore-scripts && npm run build
|
|
229
|
+
cd packages/pi-durable-disk
|
|
315
230
|
```
|
|
316
231
|
|
|
317
|
-
|
|
318
|
-
of its name is installed. A tool that only reads can be marked `replay: "safe"` (`markReadOnlySafe`): after a crash it is
|
|
319
|
-
rerun. Any other tool is an effect: its intent is committed before it starts, and after a crash the model is told the call
|
|
320
|
-
was `interrupted`, never run again. `run.setStatus("done" | "failed" | "paused" | "sleeping", detail)` records the run's
|
|
321
|
-
status in `run.json`; `done` and `failed` run the durability barrier first. `run.release()` closes the Harness, kills the
|
|
322
|
-
run's commands, runs the barrier, seals `run.json` with the store's last sequence, and unmounts. A module that is missing
|
|
323
|
-
or throws, or an `onOpen` that rejects, is `AppError` (exit 1, which the unit retries; a module that fails lets the claim
|
|
324
|
-
go first). A `beforeResume` that rejects
|
|
325
|
-
fails the open (its step is `before-resume`) and releases the claim. Besides the Harness, the hook gets `storage`, the
|
|
326
|
-
store the Harness was opened over as four reads (`scanConversations`, `scanTasks`, `scanDocuments`, `document`) with no
|
|
327
|
-
way to a write, and `head`, the store's last committed sequence: a recorder snapshots every document at `head` there.
|
|
328
|
-
Until the hook commits, the current state is the state at `head`, which is how a document that keeps no history reads.
|
|
329
|
-
|
|
330
|
-
## The CLI
|
|
331
|
-
|
|
332
|
-
`pi-durable-disk` is on the PATH after an npm install (`node dist/cli.js` in a checkout).
|
|
333
|
-
|
|
334
|
-
| Command | What it does |
|
|
335
|
-
|---|---|
|
|
336
|
-
| `run --dir PATH --app MODULE [--id ID]` | The instance on a local directory (`localRun`): no disk, no mount token, no key. The same app contract and exit codes as on a disk (76 when another process holds the run, or when a process its last incarnation left will not die; 75 when fenced), and the same run flags but the Archil ones (`--disk`, `--region`, `--mount-root`, `--archil`, `--token-stdin` are refused beside it, exit 2). `--id` defaults to the directory's name |
|
|
337
|
-
| `run --disk D --region R --id ID --app MODULE` | The instance a host driver starts: claim the run, open it, resume it. `--token-stdin` reads the mount token from stdin. Lease flags: `--heartbeat-ms`, `--lease-expiry-ms`, `--lease-margin-ms`. `--on-sigterm resume` (default) or `pause`. `--serve PORT`, `--park-threshold`, `--park-idle`, `--drain-timeout` (below). `--run-as USER` runs the agent's commands as USER when the instance runs as root (in a container) |
|
|
338
|
-
| `supervise --disk D --region R (--id ID ... \| --all) [--every 30s] --app MODULE` | One decision per run, once or in a loop: nothing to do, start, or revoke and start; `--app` is the module each started instance runs. `--create` makes the run directory. `--check` proves this host's fence first. `--sweep-tokens` removes expired token users. `--host docker --image IMAGE` starts each instance in a container (`--app-root`, `--fleet`, `--name-prefix`, `--run-as`, `--docker-arg`, `--apparmor`). Prints one JSON line per decision |
|
|
339
|
-
| `status --dir PATH` | `run.json` read from the directory, and whether its holder is still the process that claimed it: `running` (this boot, its pid alive with the start ticks `run --dir` recorded), `stopped`, `gone` (another boot), or `unknown` when this machine's boot or processes cannot be read |
|
|
340
|
-
| `status --disk D --region R --id ID` | `run.json` over the S3 API, the run's delegations and the holder's state |
|
|
341
|
-
| `release --id ID` | Unmount the run's mount on this host (a dead mount is cleaned) |
|
|
342
|
-
| `fork --id A --new-id B` | Copy a released, sealed run into a new run (see Serve, parking and fork) |
|
|
343
|
-
|
|
344
|
-
The supervisor passes the app on (`--app MODULE`, made absolute), other flags for the instance as `--run-arg=--heartbeat-ms=2000`,
|
|
345
|
-
and the instance's environment as `--env KEY=VALUE`. The API key is read from the environment variable named by
|
|
346
|
-
`--api-key-env` (default `ARCHIL_API_KEY`) by the supervisor commands only; `run` never needs it.
|
|
347
|
-
|
|
348
|
-
An instance's exit code tells the host's process supervisor what to do:
|
|
349
|
-
|
|
350
|
-
| Exit | Meaning | Restarted? |
|
|
351
|
-
|---|---|---|
|
|
352
|
-
| 0 | Drained on SIGTERM: sealed `sleeping` (due now, the default) or `paused`, unmounted | by the supervisor at its next tick, if sleeping |
|
|
353
|
-
| 1 | The app module failed to load or `onOpen` rejected, or a mount or CLI call failed | yes, within systemd's start limit |
|
|
354
|
-
| 2 | Usage error | no |
|
|
355
|
-
| 65 | Data error: the store is behind its seal (an acknowledged commit is missing), or `run.json` is unreadable. `run.json` says `failed` | never |
|
|
356
|
-
| 70 | The store's head cannot be read (pi's schema moved). `run.json` says `failed` | never |
|
|
357
|
-
| 75 | Fenced: the claim was revoked, its mount failed or left the mount table (`CLAIM_UNMOUNTED`), or the lease lapsed; or an incarnation given no mount token found no live mount to reuse (`CLAIM_GONE_NO_TOKEN`: a restart whose mount went with the failed incarnation) | never in place; the supervisor decides |
|
|
358
|
-
| 76 | Held: another client holds the claim (a delegation, live or orphaned; the supervisor revokes orphans), or another process on this host holds the run's owner lock | never in place |
|
|
359
|
-
|
|
360
|
-
## Host drivers and the supervisor
|
|
361
|
-
|
|
362
|
-
`ensureRunning(ref, host, { control })` is the whole supervisor: a function you call (from a timer, before you talk to a
|
|
363
|
-
run, or as `supervise --every 30s`). It holds no state, and it must run in a different process from any instance, because a
|
|
364
|
-
process with files on a hung mount cannot even fork. For one run it reads `run.json` over the S3 API and the delegations
|
|
365
|
-
over the control API:
|
|
366
|
-
|
|
367
|
-
| `run.json` and the delegation | Decision |
|
|
368
|
-
|---|---|
|
|
369
|
-
| `done` or `failed` | nothing (a failed run is never restarted) |
|
|
370
|
-
| `paused`, or `sleeping` with `wakeAt` later | nothing, unless the caller passes `demand` |
|
|
371
|
-
| no delegation on `runs/<id>` | start an instance (nothing while a start is in its grace) |
|
|
372
|
-
| every delegation orphaned (the holder's client is gone, seen in 0.2 s) | revoke, start (nothing while a start is in its grace) |
|
|
373
|
-
| a delegation still checking out | nothing: a mount is in progress |
|
|
374
|
-
| held, heartbeat within 90 s | healthy |
|
|
375
|
-
| held, heartbeat older than 90 s | nothing while a start is in its grace; then stop the holder if the driver can reach it, revoke, start |
|
|
376
|
-
|
|
377
|
-
Every start first writes a mark, `runs/<id>/start.json`: the generation the new instance will write into `run.json`, and
|
|
378
|
-
when. Until `run.json` reaches that generation or the start grace passes (`--start-grace`, `startGraceMs`; default the
|
|
379
|
-
lease expiry, 0 turns it off), a tick neither revokes nor starts. An instance takes about a second from its start to its
|
|
380
|
-
first `run.json` write (node boot, mount, open), more on a busy host, and a tick inside that window would otherwise see an
|
|
381
|
-
expired lease and fence the instance it just started. The mark is in the run's directory, so supervisors in other
|
|
382
|
-
processes see it too. A start that writes nothing within the grace has failed: the next tick revokes and starts again.
|
|
383
|
-
|
|
384
|
-
A run whose starts keep failing (each instance ends before its first `run.json` write: 76 when another client holds the
|
|
385
|
-
run, 1 on `MOUNT_FAILED`) backs off: the next start records the failure in its mark (`failures`, `lastExit`) and gets
|
|
386
|
-
the grace doubled per consecutive failure, up to `--start-backoff-max` (`startBackoffMaxMs`, default 10 min). With the
|
|
387
|
-
default 90 s grace that is starts at 0, 1.5, 4.5, 10.5 and 20.5 minutes, then one every 10 minutes, where a short tick
|
|
388
|
-
would start one per tick. `lastExit` is what the supervisor saw of the failed start (`no-delegation`, `held`,
|
|
389
|
-
`orphaned`); the instance's own exit code is in its host's log. `supervise` prints one `start-backoff` line per step.
|
|
390
|
-
The first `run.json` write at a start's generation resets the count.
|
|
391
|
-
|
|
392
|
-
Revoking a holder that is alive is safe: its next fsync fails and it exits 75, and nothing it writes after the revoke
|
|
393
|
-
reaches the new owner. No wait follows a revoke, because the fence is the server's, not a clock. Two supervisors racing
|
|
394
|
-
on one run are safe: the mount admits one claimant and the other instance exits 76.
|
|
395
|
-
|
|
396
|
-
A host driver is three calls, plus an optional fourth, and nothing else in the package knows which compute runs an instance:
|
|
232
|
+
## The low-level API
|
|
397
233
|
|
|
398
|
-
|
|
399
|
-
|
|
400
|
-
// The handle is JSON; the instance writes it into run.json. `attempt` (the generation the instance will open) lets a
|
|
401
|
-
// driver key a start, so a retry of the same attempt finds the instance it already started. `env` is what this start
|
|
402
|
-
// adds to the instance's environment, over the driver's own `env`; PATH and PDA_HOLDER are the driver's (refused).
|
|
403
|
-
start(ref: RunRef, mountToken: string, attempt?: { attempt: number; env?: Record<string, string> }): Promise<HostHandle>;
|
|
404
|
-
status(handle: HostHandle): Promise<"running" | "stopped" | "failed" | "gone" | "unknown">;
|
|
405
|
-
stop(handle: HostHandle): Promise<void>;
|
|
406
|
-
// Optional: take the instance away as a crash would. No drain, no release, nothing cleaned.
|
|
407
|
-
kill?(handle: HostHandle): Promise<void>;
|
|
408
|
-
readonly kills?: "process" | "box";
|
|
409
|
-
}
|
|
410
|
-
```
|
|
234
|
+
What the package exports for a host of your own. `src/index.ts` lists every export, and the `.d.ts` files in `dist/` are the
|
|
235
|
+
reference. [Library](docs/reference.md#library) describes each in full.
|
|
411
236
|
|
|
412
|
-
|
|
413
|
-
what a kill takes. `localHost` kills `process`: `systemctl kill --signal=SIGKILL --kill-whom=all` on the unit, or SIGKILL
|
|
414
|
-
to the child's process group. Its machine stays, and with `restart` on, systemd starts the unit again. `dockerHost` kills
|
|
415
|
-
`box` (`docker kill`; the container stays exited until the next start removes it), and so does `daytonaHost` (the
|
|
416
|
-
sandbox's delete). A handle from another fleet or boot is a no-op, and a kill that fails while the instance still runs
|
|
417
|
-
throws `KILL_FAILED`.
|
|
418
|
-
|
|
419
|
-
`localHost()` runs instances on this machine as systemd transient units with `KillMode=control-group`, so stopping or
|
|
420
|
-
restarting the unit kills every command the instance started, pi's detached ones included. The unit restarts on failure,
|
|
421
|
-
except after exit 65, 70, 75 or 76. `localHost({ mode: "child" })` runs a detached child process instead, for development,
|
|
422
|
-
without that guarantee.
|
|
423
|
-
|
|
424
|
-
`daytonaHost({ client, snapshot, ... })` runs each instance in a Daytona sandbox of its own (library only: `supervise`
|
|
425
|
-
starts local units, so pass the driver to `ensureRunning`). Its `box(handle)` is that sandbox's receipt: its size
|
|
426
|
-
(vCPUs, memory, disk), when it was made and when the driver deleted it, and its list price per second at
|
|
427
|
-
`DAYTONA_LIST_PRICES`. Those are Daytona's published rates, with the page and the date they were read
|
|
428
|
-
(https://www.daytona.io/pricing, 2026-10-16): $0.0504 per vCPU-hour, $0.0162 per GiB-hour of memory and $0.000108 per
|
|
429
|
-
GiB-hour of disk past the first 5 GiB, billed per second. The page's "first 5 free" footnote is applied to disk only (its
|
|
430
|
-
placement is ambiguous). A box of unknown size gets no price, and neither does a box with a GPU: the page prices GPUs per
|
|
431
|
-
model, which the driver doesn't map (`daytonaUsdPerSecond`). The box comes from a snapshot that holds Node, the archil client
|
|
432
|
-
(`libfuse2`), this package, the run user and the sudoers drop-in, or from a default image that the caller's `prepare` installs
|
|
433
|
-
them into. A container sandbox has no systemd (PID 1 is the Daytona daemon), so an in-box launcher keeps the instance
|
|
434
|
-
alive in place and never restarts it after exit 0, 65, 70, 75 or 76. Every incarnation of an attempt gets the attempt's
|
|
435
|
-
token on stdin. A restart whose mount is still live reuses it, and one whose mount went with the failed incarnation mounts
|
|
436
|
-
again: after a takeover, that mount is `HeldError` (76, terminal). The token stays in the launcher's memory for the attempt. The mount token is uploaded to a root-only file and
|
|
437
|
-
handed to the instance on stdin, never put in the sandbox's environment, which Daytona stores and returns on every read.
|
|
438
|
-
Status comes from the sandbox state plus the launcher's own word, and `stop` drains the instance, then deletes the box. The
|
|
439
|
-
archil client needs the sandbox's egress to Archil's mount servers (Daytona's Tier 3 or 4, full egress; a domain allowlist
|
|
440
|
-
cannot mount). Where the kernel refuses every unmount (a Sysbox box), the claim's release checks the delegation in, moves the
|
|
441
|
-
mount aside (`archil-scoped retire`) and kills its daemon. Before the move, the client must list no Active delegation
|
|
442
|
-
under the mount. Its own bookkeeping under `.archil/` doesn't count: the client makes `.archil/client-<n>` and its
|
|
443
|
-
`unlinked` whenever a file under the mount is deleted, and no checkin clears them once the mount is read-only. Measured live on Daytona's hosted `us` region, about 6 ms from
|
|
444
|
-
`aws-us-east-1`: a box takes about 15 s to start an instance, an exclusive mount 142 ms and a commit 8.55 ms p50; a
|
|
445
|
-
power-off, a freeze of the instance and its FUSE daemon, and a network partition were each taken over on another box (the
|
|
446
|
-
power-off through the orphaned delegation, the freeze and the partition through the lease, the partitioned box stopped by
|
|
447
|
-
the driver), and the thawed instance exited 75. A driver for another compute (a Kubernetes pod, another sandbox product) is the same three calls.
|
|
448
|
-
|
|
449
|
-
`dockerHost({ image })` (`supervise --host docker --image IMAGE`) runs each instance in a container of its own; one
|
|
450
|
-
container is one incarnation.
|
|
451
|
-
- **Start.** `docker create` named `pda-<run>-<disk key>-g<attempt>` (the disk key is 8 hex digits of the disk and region,
|
|
452
|
-
so equal run ids on two disks never share a container) and labeled `pda.fleet`, `pda.run`, `pda.disk` and `pda.region`.
|
|
453
|
-
A retry of the same attempt adopts a running container that carries all four labels (the supervisor then removes the
|
|
454
|
-
token it minted for that start) and replaces a dead one; a container by that name with other labels is refused.
|
|
455
|
-
Flags: `--device /dev/fuse --cap-add SYS_ADMIN` for the mount, `--security-opt no-new-privileges`, `--restart no` (the
|
|
456
|
-
supervisor decides every start; `dockerArgs` with another policy are refused), and `--security-opt apparmor=unconfined` only where the Docker daemon applies
|
|
457
|
-
AppArmor (`docker info` lists it; Docker Engine on Ubuntu and Debian), because Docker's default profile denies mount(2).
|
|
458
|
-
The supervisor prints that decision and its reason once. Docker Desktop and OrbStack apply no AppArmor and get no option.
|
|
459
|
-
- **The mount token** is copied into the created container as a root-only file, which the entrypoint removes before the
|
|
460
|
-
instance starts. It never appears in `docker inspect`, an argument or an environment variable; the API key stays with the
|
|
461
|
-
supervisor.
|
|
462
|
-
- **Inside the container** the instance runs as root, so the archil verbs need no sudo, and the agent's commands run as the
|
|
463
|
-
image's user `pda` (uid 1500, `run --run-as`) under no_new_privs, with every capability set empty. `work/` and the files
|
|
464
|
-
pi's write and edit tools create there belong to `pda`; the run's store, `run.json` and the archil daemon (which holds the
|
|
465
|
-
token) stay root's, and Archil enforces their modes.
|
|
466
|
-
- **Status** comes from `docker inspect`: exit 0 is stopped, any other exit (75 and 76 included) failed, and a paused
|
|
467
|
-
container is running, so the lease decides. **Stop** is `docker stop` (the instance drains), then `docker rm`.
|
|
468
|
-
- **Parking and the drain** are the local driver's: the instance gets `--park-threshold` (60 s, `parkThresholdMs`; null
|
|
469
|
-
or 0 off) and a `--drain-timeout` of the stop timeout less a close reserve. A parked instance exits 0 and its container
|
|
470
|
-
stays exited until the supervisor starts the next one at the wake.
|
|
471
|
-
- **A dead container takes its mount with it**: the mount lives in the container's mount namespace, nothing is left on the
|
|
472
|
-
host, and the next supervisor tick revokes the orphaned claim and starts a new container.
|
|
473
|
-
|
|
474
|
-
Measured with Docker Engine 29 on Ubuntu 24.04, about 3 ms from `aws-us-east-1`: a killed container is replaced in about
|
|
475
|
-
1 s and the run resumes about 2 s after the kill; a frozen one is replaced 7 to 8 s after the freeze with a 6 s test
|
|
476
|
-
lease, and when thawed it exits 75 within 0.5 s. A SQLite FULL commit from a container has the same p50 as from the host
|
|
477
|
-
(about 6 ms); a pi commit, through that store, about 11 ms.
|
|
478
|
-
|
|
479
|
-
Mount tokens are reusable with a 24 h TTL (`--token-ttl`), one token user per start attempt, named
|
|
480
|
-
`pda-<run id>-g<attempt>-<time>` so a user can be traced to its run. They are not single-use: the archil client
|
|
481
|
-
re-authenticates 5 minutes after the mount and every 5 minutes after, and the control plane rejects a spent single-use
|
|
482
|
-
token for good, after which every write on the mount fails. An instance whose token expires is fenced at its next refresh
|
|
483
|
-
(exit 75) and the supervisor restarts the run with a fresh token. Each supervise pass removes the token users of the runs it
|
|
484
|
-
supervises that no live mount needs: the run holds no delegation, its `run.json` says released, and the user is older than
|
|
485
|
-
`--token-grace` (15 minutes). Removing a token user under a live mount kills that mount's claim, so a running run keeps its
|
|
486
|
-
users. `supervise --sweep-tokens` also removes the expired users of any run, and with no `--id` cleans every released run under
|
|
487
|
-
the token prefix.
|
|
488
|
-
|
|
489
|
-
### Serve, parking and fork
|
|
490
|
-
|
|
491
|
-
`pi-durable-disk run --serve PORT` (0: any free port; `--serve-host`, default 127.0.0.1; any other address needs
|
|
492
|
-
`--serve-token-file`, see Security, and a wildcard such as 0.0.0.0 also needs `--serve-url`, the address clients reach) puts
|
|
493
|
-
a small HTTP front on the open run, so a client talks to the run, not to a host:
|
|
494
|
-
|
|
495
|
-
| Request | What it does |
|
|
496
|
-
|---|---|
|
|
497
|
-
| `POST /submit {requestId, content, conversationId?, whenBusy?, wait?}` | pi's `submit`. pi deduplicates `requestId` per conversation in the run's store, so a retry that reaches the next incarnation, even on another host, gets the same submission, and with `wait: true` its answer. The same `requestId` on another conversation is a new submission there |
|
|
498
|
-
| `POST /abort {submissionId? \| conversationId?}` | pi's abort |
|
|
499
|
-
| `GET /events` | pi's agent events as server-sent events |
|
|
500
|
-
| `GET /status` | the run's status |
|
|
501
|
-
|
|
502
|
-
The instance writes its address into `run.json` (`holder.serve`) and answers 503 (`OPENING`, `PARKING`, `RELEASED`) while it
|
|
503
|
-
opens, parks or is gone. The app module may add `root` (the options for pi's `root()`, which the serve root conversation
|
|
504
|
-
gets) and `wake`. `requestRun(ref, { method, path, body }, { host, ensure, token? })` is the client, the equivalent of a
|
|
505
|
-
Durable Object stub's `fetch`: it reads `run.json` over S3 and sends to the running holder while its lease is fresh (a
|
|
506
|
-
request whose holder's lease lapses meanwhile is dropped: a frozen instance accepts connections and never answers).
|
|
507
|
-
Otherwise it calls `ensureRunning` with demand, waits for the instance it started, or for the one the supervisor reports
|
|
508
|
-
`starting`, to write its generation, and retries until `timeoutMs`.
|
|
509
|
-
|
|
510
|
-
**Parking.** With `--park-threshold` (the local and docker drivers pass 60 s; `supervise --park-threshold 0` turns it off) the instance
|
|
511
|
-
classifies pi's tasks after every commit. When everything that could run only sleeps in a retry or a deferred-poll wait longer
|
|
512
|
-
than the threshold, it writes `run.json` `sleeping` with `wakeAt` (the deadline), releases the claim and exits 0, and the
|
|
513
|
-
supervisor starts it at `wakeAt`. With `--serve`, a run with no live work also parks after `--park-idle` (default: the
|
|
514
|
-
threshold) with `wakeAt` null, so only a request wakes it. Open requests and event streams keep the instance up; a blocked task
|
|
515
|
-
(one no definition can run) does not. An app's `wake` hook replaces how the wake is recorded: a refusal keeps the instance up
|
|
516
|
-
through the wait (pi's own timer ends it), while a failed `run.json` write is a fence (exit 75, resumed after the lease). On
|
|
517
|
-
SIGTERM the instance stops taking submissions and drains for up to `--drain-timeout` (the local driver: its stop timeout minus
|
|
518
|
-
the smaller of 5 s and half of it), then writes its wake (now, the wait's deadline, or null when idle) and releases. Work cut
|
|
519
|
-
at the deadline resumes on the next open: safe tools rerun, unsafe ones report the interruption. The parking follows the
|
|
520
|
-
lifecycle of Rivet's pi-durable host (rivet-dev/agents, Apache-2.0); no code was copied.
|
|
521
|
-
|
|
522
|
-
**Fork.** `pi-durable-disk fork --id A --new-id B` (or `fork(ref, newId, { control, mountRoot })` on the supervisor side)
|
|
523
|
-
copies a released, sealed run (paused, sleeping, done or failed, with no delegation) under two short exclusive mounts of its
|
|
524
|
-
own into a new run that starts `paused` at generation 0 with the source's `sealedSeq`. Its first open is generation 1, and a
|
|
525
|
-
lossy copy is refused (`STORE_BEHIND_SEAL`). The source is only read (store, `run.json` and workspace stay byte-identical;
|
|
526
|
-
only the claim probe `.claim` is rewritten by the mount); `run.json`, `owner.lock`, the supervisor's start mark
|
|
527
|
-
`start.json` and `tmp/` are not copied. A fork owns the new run's directory only while it holds that directory's mount: one
|
|
528
|
-
that fails after that empties the directory through its own mount and removes it, and one that lost the directory to another
|
|
529
|
-
fork or start leaves it alone.
|
|
530
|
-
|
|
531
|
-
On a local directory, `forkLocal(from, [{ id, root }, ...])` makes the same copy without a disk: one new run directory per
|
|
532
|
-
entry, from a directory of a run released and sealed (`SOURCE_NOT_RELEASED` for anything else, `TARGET_EXISTS` for a target
|
|
533
|
-
that is there). The source is only read; every copy is made beside its target under a temporary name and none is published until all are complete, so a failed
|
|
534
|
-
batch leaves no half copy. A target inside the source (also through a symlink) is `INVALID_ARGUMENT`. A local directory has no mount to take it exclusively, so nothing stops a process that opens the source while it
|
|
535
|
-
is being copied: fork a run after it is released, and open the forks with `localRun` like any run.
|
|
536
|
-
|
|
537
|
-
### Setting up a host for production
|
|
538
|
-
|
|
539
|
-
The Quickstart runs the supervisor as your own user. On a production host the instance and the agent's tools run as an
|
|
540
|
-
unprivileged user and only the archil client runs as root:
|
|
541
|
-
|
|
542
|
-
1. **A dedicated run user** (here `pda`) that owns `<mountRoot>/runs` (default `/mnt/archil/runs`), so mountpoints are
|
|
543
|
-
created and removed without root. The run user must be able to execute `node` and read the package and the app module:
|
|
544
|
-
a Node under your home directory (nvm) is usually not readable by another user, so install Node system-wide.
|
|
545
|
-
2. **The archil wrapper, root-owned**, because root executes it. It starts `archil mount` in its own systemd scope, outside
|
|
546
|
-
the instance's unit, so a restart in place finds its mount and claim still held, and it takes the mount token as one
|
|
547
|
-
line on stdin:
|
|
548
|
-
`sudo install -D -o root -g root -m 0755 bin/archil-scoped /usr/local/lib/pi-durable-disk/archil-scoped`
|
|
549
|
-
(from an npm install the file is `node_modules/@parcha/pi-durable-disk/bin/archil-scoped`, and `ARCHIL_SCOPED`, exported by
|
|
550
|
-
the package, is its path). Point the driver at it: `localHost({ archil: "/usr/local/lib/pi-durable-disk/archil-scoped",
|
|
551
|
-
user: "pda" })`, or `supervise --archil /usr/local/lib/pi-durable-disk/archil-scoped --user pda`.
|
|
552
|
-
3. **A sudoers drop-in for exactly that wrapper and `fusermount -u`.** It keeps no environment: the token travels on the
|
|
553
|
-
wrapper's stdin, and sudo logs every variable it preserves. Install it as `/etc/sudoers.d/pi-durable-disk`, mode 0440,
|
|
554
|
-
after `visudo -cf` accepts it:
|
|
555
|
-
|
|
556
|
-
```
|
|
557
|
-
pda ALL=(root) NOPASSWD: /usr/local/lib/pi-durable-disk/archil-scoped, /usr/bin/fusermount -u /mnt/archil/runs/*
|
|
558
|
-
```
|
|
559
|
-
|
|
560
|
-
4. **The supervisor runs as root** (it creates units, stops them, and writes each start's token under
|
|
561
|
-
`/run/pi-durable-disk`, root-only), with `--user pda`, so instances never run as root; `localHost` refuses to start
|
|
562
|
-
them as root without a named user. Give it the API key from a root-only environment file that its own systemd unit
|
|
563
|
-
loads (`EnvironmentFile=`), never through `sudo`: sudo logs the command line and the variables it preserves.
|
|
564
|
-
5. **Check the host once.** `pi-durable-disk supervise --check --disk D --region R` proves the fence on this host (a
|
|
565
|
-
second exclusive mount is refused, a revoked mount's fsync fails) and warns if the wrapper is not root-owned.
|
|
566
|
-
|
|
567
|
-
`examples/verify-production-path.sh` performs and checks steps 1 to 4 on a disposable root machine (`setup`, `run`,
|
|
568
|
-
`teardown`, or `all`; `plan` prints every step without running it): it creates the run user, the wrapper, the drop-in and
|
|
569
|
-
the key file, runs example 02 with the supervisor as root and `--user`, and checks that the instances never run as root,
|
|
570
|
-
that the run user can sudo the wrapper and nothing else, that `no_new_privs` takes even that away from a command, that no
|
|
571
|
-
mount token reaches the journal, and that nothing is left behind. It needs systemd; do not run it on a shared machine.
|
|
572
|
-
|
|
573
|
-
## The failure table
|
|
574
|
-
|
|
575
|
-
| Failure | What happens | Effects |
|
|
237
|
+
| Name | What it is | Example |
|
|
576
238
|
|---|---|---|
|
|
577
|
-
|
|
|
578
|
-
|
|
|
579
|
-
|
|
|
580
|
-
|
|
|
581
|
-
|
|
|
582
|
-
|
|
|
583
|
-
|
|
|
584
|
-
|
|
585
|
-
|
|
586
|
-
|
|
587
|
-
|
|
588
|
-
|
|
589
|
-
|
|
590
|
-
|
|
591
|
-
|
|
592
|
-
|
|
593
|
-
| The control API lists the run's holder without a path (it resolves paths best-effort) | The supervisor maps pathless delegations' inodes to run directories with one `Disk.exec` per pass (`find -inum`) and counts one on the run's directory as the run's: a live one keeps the run healthy, an orphaned one is revoked and the run restarted. Where `exec` is unavailable or fails, the pass is `CONTROL_API_FAILED` and starts nothing (the supervisor never forces); a dead holder then stays until Archil lists its path again or an operator revokes it by client and inode | no second instance; nothing started over a holder it cannot see |
|
|
594
|
-
| A client dies holding its own private directories (`.archil/client-<clientId>` and its `unlinked/`, which the control API lists with no path and on no run directory) | They block no run. Every revoke that fences or deletes a run (`revoke`, `takeOver`, the supervisor's takeover, `deleteRunTree`) also revokes, from the same listing, the private-directory delegations of the clients it revokes on the run, confirmed by one `Disk.exec` (`stat`), unless the client holds anything else; best effort, so one left stays for an operator to revoke by client and inode | none: they hold no run directory |
|
|
595
|
-
| The store is behind its `run.json` seal | `STORE_BEHIND_SEAL`: `run.json` is marked failed and the instance exits 65; the supervisor does not restart it | none started |
|
|
596
|
-
| SIGTERM (a deploy) | The instance stops taking submissions and drains for up to `--drain-timeout`, writes `sleeping` with its wake (now; the deadline when everything only waits; null when idle and woken by requests), then closes, kills its commands, barriers, seals and unmounts, exit 0; the supervisor starts it again at the wake | work cut at the drain's deadline resumes on the next open: safe tools rerun, unsafe ones report the interruption |
|
|
597
|
-
| Everything the run does is a retry or poll wait longer than the park threshold | The instance writes `sleeping` with `wakeAt` (the deadline), releases and exits 0; the supervisor starts it at `wakeAt` and pi's timer sleeps whatever is left; a `run.json` write that fails is a fence (exit 75) and the lease path resumes the run | nothing is lost: the wait is a checkpoint in the store |
|
|
598
|
-
|
|
599
|
-
**On a local directory** (`localRun`) nothing is mounted or delegated and no supervisor watches the run, so the rows above
|
|
600
|
-
about revocation, takeover and the lease's replacement do not apply. These do:
|
|
601
|
-
|
|
602
|
-
| Failure | What happens | Effects |
|
|
603
|
-
|---|---|---|
|
|
604
|
-
| The instance crashes or is SIGKILLed | Its owner lock goes with the process. The next open of the directory (by the app, or whatever restarts it) takes the run, ends the commands the killed incarnation left running (its `commands` step, before any code of the new incarnation runs), and `resume()`s from the last commit | running tasks go back to pending; effects in flight are `interrupted`, never rerun; no command of the killed incarnation still runs once the run resumes. pi makes each command the leader of its own process group, so the SIGKILL does not reach them and a local run has no unit cgroup: the instance records each command's group under `commands/` (by pid, start ticks and boot) and the next open kills what still matches. A command started within about 0.1 s before the kill can be missed |
|
|
605
|
-
| A process the killed incarnation's commands started does not die at the next open (an uninterruptible wait, or one the run's user may not signal) | The open is refused: `HeldError` `COMMANDS_ALIVE`, exit 76, with the message below; the records stay | none: the run does not resume beside it. End those processes and open the run again |
|
|
606
|
-
| Power loss or an OS crash | The store's commits are durable (SQLite at `synchronous = FULL`); `work/` is the operating system's to flush, since the claim's barrier syncs nothing | the store resumes from its last commit; files the agent wrote in `work/` can be older than what the store recorded |
|
|
607
|
-
| A second process opens the run on this machine | The owner lock is refused (`HeldError`, exit 76) | none |
|
|
608
|
-
| A process on another machine reaches the directory (a network share) | Nothing stops it: a local directory has no fence | unsupported, since two writers can corrupt the store. A directory another machine can reach needs a disk with a claim and a fence |
|
|
609
|
-
|
|
610
|
-
The refusal names each process, as the open writes it (here one, in a run at `<root>`):
|
|
611
|
-
|
|
612
|
-
```
|
|
613
|
-
1 process(es) a killed incarnation's commands started are still alive after SIGKILL: pid 4242 (tar -xf /mnt/slow/archive.tar). The run does not resume beside them. End them (kill -9 4242, as their owner, or wait for them) and open the run again; they stay recorded in <root>/commands until then.
|
|
614
|
-
```
|
|
615
|
-
|
|
616
|
-
What a revoke cannot do: no fence reaches a third party. A paid API that already has a request has it. The package
|
|
617
|
-
guarantees that this runtime does not start an effect twice; an effect your model decides to retry is a new call, so give a
|
|
618
|
-
paid API an idempotency key from your own business key.
|
|
619
|
-
|
|
620
|
-
Measured on a VM about 3 ms from `aws-us-east-1`, archil 0.8.42: acquire 222 ms p50, release 139 ms, takeover by revoke
|
|
621
|
-
252 ms, commit 3.75 to 5.63 ms p50, a killed host's run back to its first commit in 1 to 1.5 s, a hung host replaced in 6
|
|
622
|
-
to 6.3 s with a 6 s test lease (90 to 120 s with the defaults). A start that never reaches running costs its start grace: live on Archil with a 6 s grace and a 3 s tick,
|
|
623
|
-
starts that all failed were retried six times in five minutes, each grace doubled (`test/live/supervise-slow.live.test.ts`),
|
|
624
|
-
and a takeover whose start was slowed 5 s before its first `run.json` write started once and was never revoked, 5 of 5
|
|
625
|
-
(`test/live/supervise.live.test.ts`). On Docker and Daytona hosts, a start lost before its instance runs is not measured.
|
|
626
|
-
|
|
627
|
-
## Security
|
|
628
|
-
|
|
629
|
-
- **The mount token goes on the wrapper's stdin.** It is never in argv or in an environment that `sudo` sees. Some setup
|
|
630
|
-
guides pass the mount token with `sudo --preserve-env=ARCHIL_MOUNT_TOKEN archil mount ...`; sudo logs every variable it
|
|
631
|
-
preserves to the journal and `auth.log`, so that form leaves the token readable to anyone who can read those logs. The wrapper reads
|
|
632
|
-
one line on stdin and sets `ARCHIL_MOUNT_TOKEN` only for the exec of `archil`. The sudoers drop-in must never use
|
|
633
|
-
`env_keep` for it. The local driver hands the token to the instance on stdin from a root-only file that is emptied once
|
|
634
|
-
the unit has started; the instance reads it and closes stdin, so no process of the run's user can reopen it.
|
|
635
|
-
- **Tokens cover the whole disk.** A leaked one could mount or force-take any run on that disk until it expires, so the
|
|
636
|
-
supervisor mints one per start attempt, removes it after the instance releases, and sweeps expired ones. The tenant
|
|
637
|
-
boundary is the disk (Archil's own advice): give each tenant its own disk.
|
|
638
|
-
- **The API key stays with the supervisor.** Hosts and instances never hold it, and the local driver never copies its own
|
|
639
|
-
environment into a unit.
|
|
640
|
-
- **No privilege for the agent.** The instance, pi's tools and the agent's commands run as an unprivileged user. Every
|
|
641
|
-
command starts under `setpriv --no-new-privs` (on Linux; macOS has no equivalent, see [On macOS](#on-macos)), so a
|
|
642
|
-
command cannot use the run user's sudo rule (sudo refuses: "The no new privileges flag is set"). An app that turns
|
|
643
|
-
this off (`archilEnv(claim, { noNewPrivs: false })`) or runs commands another way must confine them itself. Commands
|
|
644
|
-
run next to the FUSE client on the host that holds the claim: if the agent is not trusted with the host, sandbox its
|
|
645
|
-
commands (for example with bubblewrap) and keep the FUSE client outside the sandbox.
|
|
646
|
-
- **The file tools stay inside `work/`.** pi's `read`, `write` and `edit` run in the loop's process, not in a command
|
|
647
|
-
sandbox, so every file operation of the environment resolves its path to the file the kernel would reach (`..`,
|
|
648
|
-
absolute paths, symlinks, dangling links, a missing tail) and refuses it unless it lies in `work/` (reads may also use
|
|
649
|
-
`tmp/`, where the bash tool tells the model its long output went, and `readRoots` the app names; writes may also go
|
|
650
|
-
inside a directory the environment's `createTempDir` returned, for the tool it was handed to). An agent cannot `write
|
|
651
|
-
../store/run.sqlite`, `../run.json` or `../owner.lock`: the call fails with a `PathOutsideWorkError` the model sees as
|
|
652
|
-
a failed tool result. A path is opened one component at a time from a held handle of `work/`, without following links,
|
|
653
|
-
so a link a command swaps in after the check is refused too (on Linux; on macOS the walk is by path, see [On
|
|
654
|
-
macOS](#on-macos)). This covers the file tools only; a command still runs as the run user and can reach anything that
|
|
655
|
-
user can, which is what an app's own command sandbox is for. `archilEnv(claim, { confineFiles: false })` turns it off
|
|
656
|
-
where `/proc/self/fd` is unavailable.
|
|
657
|
-
- **`--serve` authenticates only with a bearer token.** On 127.0.0.1 (the default) the token is optional. A non-loopback
|
|
658
|
-
`--serve-host` requires `--serve-token-file`, a file of mode 0600 holding the token (never argv or the environment); every
|
|
659
|
-
request must send `authorization: Bearer <token>`, and it is compared in constant time. The file is readable by the run
|
|
660
|
-
user, so by the agent's own commands: it guards the network, not the agent. Put TLS in front of anything that leaves the
|
|
661
|
-
machine.
|
|
662
|
-
- **In a container** (`dockerHost`) the instance runs as root and holds `CAP_SYS_ADMIN` (with AppArmor unconfined where the
|
|
663
|
-
daemon applies AppArmor), which `archil mount` needs. The agent's commands do not: they run as uid 1500 with every
|
|
664
|
-
capability set empty, the bounding set included, and no_new_privs, so `mount`, `unshare` and `nsenter` fail from a
|
|
665
|
-
command. With `fs.protected_hardlinks` on (the live suite checks it), a command cannot hard-link the store or `run.json`
|
|
666
|
-
into `work/` either. Anything passed with `--env` is visible to `docker inspect` on the machine and to the agent's
|
|
667
|
-
commands, so never pass a secret there that the agent must not read.
|
|
668
|
-
- **A revocation is the most dangerous operation.** It happens only for an orphaned client, a dead host or an expired
|
|
669
|
-
lease, and every takeover writes the new `generation` and the previous holder into `run.json` and the supervisor's log.
|
|
670
|
-
|
|
671
|
-
## Layout on the disk
|
|
672
|
-
|
|
673
|
-
```
|
|
674
|
-
<disk>/runs/<id>/ the claimed subtree, mounted at <mountRoot>/runs/<id> on every host (default /mnt/archil)
|
|
675
|
-
run.json status, generation, holder, heartbeat, seal
|
|
676
|
-
owner.lock the same-host owner lock
|
|
677
|
-
store/run.sqlite pi's store (exclusive profile: no -shm file)
|
|
678
|
-
work/ the workspace the tools write
|
|
679
|
-
tmp/ pi's temp files and long-output spills, so a path a result names still exists after a takeover
|
|
680
|
-
```
|
|
681
|
-
|
|
682
|
-
A run on a local directory (`localRun`) has the same layout, plus `commands/`: the process groups of the commands its
|
|
683
|
-
instance started, which the next open ends before it resumes (a local run has no unit cgroup to end them).
|
|
684
|
-
|
|
685
|
-
The mountpoint is the same absolute path on every host of a deployment, so paths inside the run are identical wherever
|
|
686
|
-
it resumes.
|
|
687
|
-
|
|
688
|
-
### Removing a run
|
|
689
|
-
|
|
690
|
-
A run is the subtree `runs/<id>/` of the disk. To remove one: let its instance release (or stop it), check that nothing
|
|
691
|
-
holds a delegation on it (`pi-durable-disk status` lists them; an orphaned one is revoked through the control API), delete
|
|
692
|
-
its objects, and remove its token users (`supervise --id ID --sweep-tokens --token-grace 0s` on a run that is done).
|
|
693
|
-
|
|
694
|
-
**Revoke before you delete.** Deleting the objects of a run through the S3 API while a delegation on the path is still
|
|
695
|
-
held, even an orphaned one left by a killed host, does not take effect: the same run id came back with its old store when a
|
|
696
|
-
supervisor next started it. `deleteRunTree(control, id)` does it in order: it revokes the run's delegations and its holders' private directories,
|
|
697
|
-
deletes, and requires the prefix to be empty. Doing it by hand, revoke first (`revoke(control, id)` takes the same set), delete,
|
|
698
|
-
then list the prefix and check it is empty.
|
|
699
|
-
|
|
700
|
-
## Library
|
|
701
|
-
|
|
702
|
-
`openDurableRun(ref, options)` is what `run` calls: mount, verify the claim, take the owner lock, make `store/`, `work/`
|
|
703
|
-
and `tmp/`, end a killed incarnation's commands, write `run.json`, open the store, check the seal, open and resume the
|
|
704
|
-
Harness, keep the lease. Its `afterClaim(claim)` is awaited once the claim is held and before `run.json`, with the
|
|
705
|
-
claim's `ref`, `root`, `work` and `store`. What it returns (`harness`, `beforeResume`, `seed`) goes into the open, so
|
|
706
|
-
`harness` may come from there instead of the options, and a factory can give a new run its seed (over the options'
|
|
707
|
-
`seed`); the seed runs right after `afterClaim` and before `run.json`, on a new run only. A rejection fails the open with `run.json` as the open found it and lets the
|
|
708
|
-
claim go. `acquire`, `ensureRunning`, `localHost`,
|
|
709
|
-
`archilEnv` and the typed errors (`FencedError`, `HeldError`, `StoreBehindSealError`) are exported for apps that compose
|
|
710
|
-
their own lifecycle, and so are serve (`serveRun`, `requestRun`), parking (`watchParking`, `drain`, `busyState`, and
|
|
711
|
-
`leaseParkTarget` for a Harness a host opened over `openRunLease`) and `fork`. `src/index.ts` lists every export; the
|
|
712
|
-
`.d.ts` files in `dist/` are the reference.
|
|
713
|
-
|
|
714
|
-
A supervisor of your own (a driver `supervise` does not start, such as `daytonaHost`) gets its `control` from
|
|
715
|
-
`archilControl({ apiKey, disk, region })`: the disk's S3 and control API, the same object `supervise` builds from its flags.
|
|
716
|
-
|
|
717
|
-
```ts
|
|
718
|
-
const control = await archilControl({ apiKey: process.env.ARCHIL_API_KEY!, disk: "dsk-...", region: "aws-us-east-1" });
|
|
719
|
-
await ensureRunning({ disk: "dsk-...", region: "aws-us-east-1", id: "job-1" }, daytonaHost({ client, snapshot }), { control });
|
|
720
|
-
```
|
|
721
|
-
|
|
722
|
-
A host that owns its own pi-durable Harness and store connections imports the narrow entry instead:
|
|
723
|
-
`@parcha/pi-durable-disk/lease` holds `openRunLease` (the claim, the owner lock, `run.json` with its heartbeat, the lease
|
|
724
|
-
self-fence, the seal and the release, without a store or a Harness), `storeHead`, the store's fence and pragma helpers
|
|
725
|
-
(`Fence`, `FencedDatabase`, `assertPragmas`, `PROFILE_PRAGMAS`, `openArchilStore`), parking (`leaseParkTarget`,
|
|
726
|
-
`watchParking`, `drain`, `busyState`), and the typed errors and exit codes. It leaves out the supervisor, the host drivers,
|
|
727
|
-
the CLI and the app contract, and the package declares `"sideEffects": false`, so a bundler keeps only what you use: a
|
|
728
|
-
bundle of the app with `await import("@parcha/pi-durable-disk/lease")` is about 63 KB minified against about 620 KB
|
|
729
|
-
for `await import("@parcha/pi-durable-disk")` (esbuild `--bundle --minify --platform=node --format=esm`, which inlines
|
|
730
|
-
pi-durable, chord and their dependencies).
|
|
731
|
-
|
|
732
|
-
`@parcha/pi-durable-disk/internals` holds the claim, status and Daytona plumbing this repository's examples and demos use:
|
|
733
|
-
`runPath`, `unmountClaim`, `createRunDir`, the mount-token and delegation helpers, `isLeaseFresh`, and the Daytona labels,
|
|
734
|
-
names and `sweepSandboxes`. It carries no compatibility promise: a name there can change or go in any release.
|
|
735
|
-
|
|
736
|
-
### Seeding a new run
|
|
737
|
-
|
|
738
|
-
`seed` (in `openDurableRun`'s options, and `openRunLease`'s) writes what a new run starts with before anything sees the
|
|
739
|
-
run: its store file, a SQLite database the Harness then opens (saved sign-in profiles, say), and files in its
|
|
740
|
-
workspace (`work/.agents/skills/`). `beforeResume` runs after the store is open, too late for that.
|
|
741
|
-
|
|
742
|
-
```ts
|
|
743
|
-
await openDurableRun(ref, {
|
|
744
|
-
...localRun("/var/runs/run-1"), // or a disk's mountToken
|
|
745
|
-
harness: { models, registry },
|
|
746
|
-
async seed({ storePath, workDir }) { // also root: the claim's root
|
|
747
|
-
const db = new DatabaseSync(storePath);
|
|
748
|
-
db.exec("CREATE TABLE IF NOT EXISTS x_agent_profiles (name TEXT PRIMARY KEY, profile BLOB)");
|
|
749
|
-
db.close(); // closed before it returns: the store opens it next
|
|
750
|
-
await cp(skillsDir, join(workDir, ".agents/skills"), { recursive: true });
|
|
751
|
-
},
|
|
752
|
-
});
|
|
753
|
-
```
|
|
754
|
-
|
|
755
|
-
- **When it runs.** Once, on the open that finds no `run.json`: inside the run's first claim, with the owner lock held,
|
|
756
|
-
after `store/`, `work/` and `tmp/` exist, a killed incarnation's commands are ended and `afterClaim` has run, before
|
|
757
|
-
`run.json` is written and before the store opens. Its step is `seed`. What it wrote is made durable (the claim's
|
|
758
|
-
barrier) before `run.json` publishes the run.
|
|
759
|
-
- **From the app.** `afterClaim` may return `seed` (as `AfterClaim`), over the options' one, so a factory that runs on the
|
|
760
|
-
claimed directory can give the run its starting state.
|
|
761
|
-
- **When it doesn't.** Never for a run that has `run.json`, whatever its generation, and never on a fork's first open
|
|
762
|
-
(`fork` writes `run.json`).
|
|
763
|
-
- **When it fails.** A seed that throws fails the open with its own error. A process killed while it seeds leaves the run
|
|
764
|
-
unpublished in the same way: no `run.json`, the claim let go. The next open seeds again, so a seed overwrites what an
|
|
765
|
-
earlier attempt wrote.
|
|
766
|
-
- **How long it may take.** A new run has no `run.json` heartbeat while it seeds. A supervisor keeps a start in its grace
|
|
767
|
-
(`--start-grace`, `startGraceMs`; default the lease expiry, 90 s) only while its start mark (`start.json`) is younger
|
|
768
|
-
than the grace. So while the seed runs, the instance rewrites the mark's time every heartbeat, through its claim, when a
|
|
769
|
-
supervisor's mark for generation 1 is there. It does so on a thread of its own, so a seed that holds its thread (a
|
|
770
|
-
synchronous import through `DatabaseSync`) is covered too. A live seed of any length stays `starting`. If the mark
|
|
771
|
-
can't be rewritten, the instance says so once, as a process warning with code `PDA_START_MARK`. A seeder that dies stops
|
|
772
|
-
rewriting, the grace runs out as for any start, and the claim is revoked. If the claim is revoked while the seed still
|
|
773
|
-
runs, its next write or its barrier is refused, the instance is fenced (exit 75) with nothing published, and the next
|
|
774
|
-
start seeds again.
|
|
775
|
-
- **On a local directory** (`localRun`) it is the same. As with `work/`, the files it writes are the operating system's
|
|
776
|
-
to flush; a SQLite database it commits with `synchronous = FULL` is durable when the commit returns.
|
|
777
|
-
|
|
778
|
-
### A run on a local directory
|
|
779
|
-
|
|
780
|
-
`localRun(root)` returns the `acquire` and `claimDir` that put a run on a plain directory of this machine, with no mount
|
|
781
|
-
and no mount token:
|
|
782
|
-
|
|
783
|
-
```ts
|
|
784
|
-
import { localRun, openDurableRun } from "@parcha/pi-durable-disk";
|
|
785
|
-
|
|
786
|
-
const run = await openDurableRun({ disk: "local", region: "local", id: "run-1" }, {
|
|
787
|
-
...localRun("/var/runs/run-1"), // made when it is missing; work/ and store/ inside it
|
|
788
|
-
harness: { models, registry },
|
|
789
|
-
});
|
|
790
|
-
```
|
|
791
|
-
|
|
792
|
-
The run's status is its `run.json`, and `readLocalRunStatus(root)` (the directory `localRun` takes) reads it, read-only and safe beside a live instance, the way `readRunStatus` reads a disk's: the
|
|
793
|
-
same `RunRecord` (`status`, `generation`, `wakeAt`, `holder`, `sealedSeq`, `detail`), `null` when the run has no `run.json`
|
|
794
|
-
yet, and `RunRecordError` (`RUN_JSON_INVALID`) when it is malformed.
|
|
795
|
-
|
|
796
|
-
```ts
|
|
797
|
-
const status = await readLocalRunStatus("/var/runs/run-1"); // { status: "sleeping", wakeAt: "...", ... } or null
|
|
798
|
-
```
|
|
799
|
-
|
|
800
|
-
It is for one machine (a laptop, a CI job, a test), and says so: the claim is the directory itself, so nothing but this
|
|
801
|
-
machine's owner lock keeps a second process off the run (a second open while one holds it is refused), and nothing at
|
|
802
|
-
all keeps out a process on another machine that can reach the directory. Its barrier refuses once the run is fenced,
|
|
803
|
-
like a mount's, but syncs nothing: the store's commits are made durable by SQLite, and the files the agent wrote in
|
|
804
|
-
`work/` are the operating system's to flush, so a power loss can lose what it had not written yet. `mountToken` is
|
|
805
|
-
required only with the default `acquire`; without either, the open is refused (`INVALID_ARGUMENT`) before anything mounts.
|
|
806
|
-
|
|
807
|
-
#### On macOS
|
|
808
|
-
|
|
809
|
-
A local run works on macOS for development, and is weaker there than on Linux in four ways. Use Linux, or the
|
|
810
|
-
[Docker quickstart](#quickstart-with-docker), when the agent is not trusted.
|
|
811
|
-
|
|
812
|
-
- **Commands keep your rights: `no_new_privs` has no macOS equivalent.** On Linux every command starts under
|
|
813
|
-
`setpriv --no-new-privs`, so it cannot use `sudo` or gain privilege from a setuid program. macOS has no such flag and nothing
|
|
814
|
-
that does its job (`sandbox-exec` is deprecated), so `noNewPrivs` defaults to false there and no `setpriv` runs. That
|
|
815
|
-
leaves open: a command can use `sudo` wherever sudo asks it for no password (a `NOPASSWD` rule, for one), and setuid
|
|
816
|
-
programs keep their privilege. `runAs` is not available either (it needs `setpriv`). As on Linux, a command runs as
|
|
817
|
-
you and can reach anything you can.
|
|
818
|
-
- **The file tools' confinement is path-based.** On Linux it walks a path through `/proc/self/fd`, one held directory at
|
|
819
|
-
a time, so no component can be swapped once it was checked; macOS has no `/proc/self/fd`. There it resolves the path
|
|
820
|
-
(`realpath`) and refuses it unless it lies in `work/` or another root the file tools may use, as on Linux, opens each
|
|
821
|
-
component by its full path with `O_NOFOLLOW`, then checks that the path still resolves to itself and names the
|
|
822
|
-
directory it opened (same device and inode). It refuses `../store/run.sqlite`, absolute paths and links out of
|
|
823
|
-
`work/`, a link already in place included. That leaves open: a command that swaps a directory above the target for a
|
|
824
|
-
link between that check and the operation can win, and the file tool then acts where the link points.
|
|
825
|
-
- **The run's root is addressed by its path.** On Linux, `run.json` and the run's directories are written through the
|
|
826
|
-
directory opened at acquire. On macOS each write is checked first against that directory, and a root renamed and
|
|
827
|
-
replaced between the check and the write gets the write.
|
|
828
|
-
- **Commands are recorded from `ps`.** With no `/proc`, the instance reads one `ps` listing every 0.5 s while commands run,
|
|
829
|
-
`ps -E` for a command's mark and `sysctl kern.bootsessionuuid` for the boot. A command started within about 0.5 s
|
|
830
|
-
before a kill can be missed, and a process is told from a later one with the same pid by the second it started. The
|
|
831
|
-
lease watchdog reads the same listing.
|
|
832
|
-
|
|
833
|
-
`fsync` on macOS does not flush the drive's cache, so the store also sets SQLite's `fullfsync` and
|
|
834
|
-
`checkpoint_fullfsync` there (SQLite then syncs with `F_FULLFSYNC`), and refuses to open unless they read back. CI runs
|
|
835
|
-
the no-key quickstart, the local run's tests (its command records, `run --dir` and `status --dir`, the lease watchdog,
|
|
836
|
-
the file tools' confinement) and the keyless examples on macOS.
|
|
837
|
-
|
|
838
|
-
### A short claim on any path, and a read of what it released
|
|
839
|
-
|
|
840
|
-
`withClaim` holds one path of a disk exclusively for a moment, outside any run: pi-durable-agent carries an agent's home
|
|
841
|
-
(`agent/agent.sqlite`) back at a run's end this way. `readReleased` reads a file such a claim released over the S3 API,
|
|
842
|
-
with no mount, so a reader never blocks a writer. Both are supervisor side: they take the `control` of
|
|
843
|
-
`archilControl({ apiKey, disk, region })`.
|
|
844
|
-
|
|
845
|
-
```ts
|
|
846
|
-
await withClaim({ control, disk: "dsk-...", region: "aws-us-east-1", path: "agent" }, async (held) => {
|
|
847
|
-
const db = new DatabaseSync(join(held.root, "agent.sqlite"));
|
|
848
|
-
db.exec("INSERT INTO runs (id) VALUES ('job-1'); PRAGMA wal_checkpoint(TRUNCATE)");
|
|
849
|
-
db.close();
|
|
850
|
-
});
|
|
851
|
-
const store = await readReleased({ control, path: "agent" }, "agent.sqlite", { sqlite: true }); // null: none yet
|
|
852
|
-
```
|
|
853
|
-
|
|
854
|
-
A claim is one exclusive mount, about 0.4 to 1 s with its release, so it is for a run's end, never a turn.
|
|
855
|
-
- **Exclusive by the mount.** Archil refuses a second client's mount of the path, an ancestor or a descendant. The call
|
|
856
|
-
mounts at a fresh `<mountRoot>/runs/<pid>-<uuid>` (default `/mnt/archil/.claims`; the shape `archil-scoped`'s cleanup
|
|
857
|
-
takes on a host that cannot unmount) and never shares a mount. Archil mounts no path whose directory is missing, so it
|
|
858
|
-
first makes the path's directories the disk lacks, over S3, as the calling process's own (0700; `owner` sets them). It writes
|
|
859
|
-
`.claim` with its deadline (`holdMs`, default 60 s), runs `fn`, then barriers, unmounts and gives its mount token back,
|
|
860
|
-
whether `fn` resolved or threw.
|
|
861
|
-
- **Lost holders are revoked first.** A holder whose client is gone (orphaned) is revoked before the mount, and so is a
|
|
862
|
-
holder of exactly the path whose `.claim` names a deadline that has passed and stays the same for 3 s (a holder that
|
|
863
|
-
just mounted rewrites it at once; a holder of a path above or below is never revoked by this path's `.claim`). A live holder within its deadline is waited for (`waitMs`, default 30 s), never
|
|
864
|
-
revoked. A holder revoked while still alive fails its next fsync, so its commit rolls back and its own call is FENCED:
|
|
865
|
-
the deadline never decides what is durable, the fence does.
|
|
866
|
-
- **What it refuses.** A path that is not relative and of safe segments, or that is under `runs/` (a run's own) or
|
|
867
|
-
`.archil/`, is `ClaimError` `INVALID_ARGUMENT`, before any call. A live holder after `waitMs` is `HeldError`
|
|
868
|
-
`CLAIM_HELD` (exit 76). The control API failing, or a delegation with no path it cannot attribute, is
|
|
869
|
-
`CONTROL_API_FAILED`. A claim revoked from under `fn` is `FencedError` (exit 75), and `fn`'s writes are then not
|
|
870
|
-
durable.
|
|
871
|
-
- **`CLAIM_HELD` is retryable.** The path is not released yet: retry within `waitMs`, and never fall back to an older
|
|
872
|
-
release.
|
|
873
|
-
|
|
874
|
-
`readReleased` returns the bytes only when the read is of a released file. Nothing holds the path (or an ancestor or a
|
|
875
|
-
descendant) before or after the read, and the object is unchanged across it (its etag, else its size and time). For
|
|
876
|
-
SQLite (`sqlite: true`), its `-wal` must also be absent or empty, so its writer checkpointed with `TRUNCATE` and closed. A
|
|
877
|
-
held path is `CLAIM_HELD`, retryable as above, and a WAL with content is `NOT_RELEASED`. A file that changed during every
|
|
878
|
-
one of `attempts` reads (default 3) is `CLAIM_HELD`. Open the bytes as a local file, read-only (`?immutable=1`).
|
|
879
|
-
|
|
880
|
-
### Places: a run a supervisor can kill and bring back
|
|
881
|
-
|
|
882
|
-
A place is where one run lives and how its instance starts, is killed and comes back, for a supervisor that decides when
|
|
883
|
-
to kill (agentrun's reference demo). Each factory takes its options and returns `(ctx) => Place`; `ctx.paths` is the run's
|
|
884
|
-
id and directory as the supervisor computed them.
|
|
885
|
-
|
|
886
|
-
- `localPlace({ app, root? })` runs `pi-durable-disk run --dir <root> --id <id> --app <app>` as a child in its own
|
|
887
|
-
process group, with `{ PATH, ...order.env }` and nothing else of the caller's environment. It needs no disk, no key and
|
|
888
|
-
no token. A process kill is SIGKILL to the group, and the next runner resumes the run from the directory. A box kill
|
|
889
|
-
also calls `ctx.destroyBoxes` (a deletion that failed is tried again by the next kill or recover). `release()` kills
|
|
890
|
-
what still runs, then, under the run's owner lock, ends the commands the run recorded (pi starts each in a process
|
|
891
|
-
group of its own, which the instance's kill does not reach). While another process holds the run, they are its own,
|
|
892
|
-
and release leaves them. With `ctx.requestFile`, `signals(attempt)` yields the kill request the runner writes
|
|
893
|
-
there when it parks at its planned crossing, and `requestFile` is that same file. `return()` ends a pending `next()`.
|
|
894
|
-
- `archilPlace({ disk, region, control, host })` runs the instance on the disk through a host driver set up with the app.
|
|
895
|
-
- `token(attempt)` mints that attempt's mount token through `control`, and the runner handed that token mounts with it.
|
|
896
|
-
- A start is `ensureRunning` with the token (`mountToken`) and the order's environment (`StartAttempt.env`).
|
|
897
|
-
- A kill is the driver's `kill`, and `kills` is the driver's.
|
|
898
|
-
- A recovery is the takeover: the killed holder's delegations are revoked whatever its lease says, and the next
|
|
899
|
-
instance starts. It reports `revoked` and `ms`. A newer incarnation than the one the place killed (another
|
|
900
|
-
supervisor's) gets the usual lease and start-grace checks instead, so a live one is `HELD`, not revoked.
|
|
901
|
-
- `status()` reads `run.json` over S3 and lists the run's delegations, plus, when its host driver makes boxes
|
|
902
|
-
(`daytonaHost`), a receipt for each box the place started, released ones included (`boxes`, `HostBox`). A receipt
|
|
903
|
-
has the provider, the kind, `from` and `to` (null while it lives). It also has the published list price per second
|
|
904
|
-
(`usdPerSecond`), the page that price was read from (`source`) and the ISO date it was read (`priced`). A box with
|
|
905
|
-
no price has none of those three, never a price of 0. No field holds the provider's id of the box.
|
|
906
|
-
- `recordHome` (when the control can list, as `archilControl`'s can): an attempt's runner records into
|
|
907
|
-
`recordHome.dir(attempt)`, `<root>/record/<attempt>`, in the run's directory, so its delegation covers it and
|
|
908
|
-
`deleteRunTree` removes it. `bringHome(into, { final })` reads that directory through the control (S3 list and get:
|
|
909
|
-
nothing takes the run's claim) and brings the recording into the supervisor's `into`.
|
|
910
|
-
- Blobs come first, each checked against the hash its name carries and written whole (a temporary file, fsync,
|
|
911
|
-
rename). One that doesn't match is skipped; everything else still comes home, then the call throws
|
|
912
|
-
`BLOB_MISMATCH`.
|
|
913
|
-
- Then the new whole lines of each `.spool/*.jsonl` segment, with an offset per file name: what `into` already
|
|
914
|
-
holds of it. So a second call, or a call from another process, brings nothing twice. A line waits while a blob it
|
|
915
|
-
names isn't home yet. On `final` it comes home anyway, and the call throws `CARRY_FAILED` naming the blob.
|
|
916
|
-
- Carriers that overlap on one `into` never append at once. The place runs one call at a time per destination, and
|
|
917
|
-
each append happens under a lock file beside its spool file (`.<name>.lock`, naming its holder's pid), after the
|
|
918
|
-
destination's size is read again, so only the lines past it are appended. A lock whose holder is gone is taken
|
|
919
|
-
over; one held past 5 s is `CARRY_FAILED`. A second `into` gets every line: what the place fetched for one
|
|
920
|
-
destination says nothing of another.
|
|
921
|
-
- A list or get that fails is `CARRY_FAILED`.
|
|
922
|
-
- The recording is on the disk, not in the box, so a final call after a box kill still brings home everything the
|
|
923
|
-
box fsynced: this place never throws `BOX_GONE`.
|
|
924
|
-
- With `ensure.create` (the run user's uid and gid), the place makes the EMPTY directories the runner writes into
|
|
925
|
-
at each start: `signals/`, `record/<attempt>/.spool/` and `record/<attempt>/blobs/`. It makes them over the
|
|
926
|
-
control, owned by the run user, mode 0755, in ensureRunning's window where no delegation holds the run. A
|
|
927
|
-
directory a process made by path on an Archil mount was seen to refuse file creates afterwards (ENOENT), and one
|
|
928
|
-
made over the control didn't. Every file in them stays the runner's. A refused put fails the start
|
|
929
|
-
(`DIRS_REFUSED`, as the cause of `HOST_START_FAILED`), and without an owner nothing is made.
|
|
930
|
-
- `requestFile(attempt)` is `<root>/signals/kill-<attempt>.json`, in the run's directory and outside `work/`, and
|
|
931
|
-
`signals(attempt)` reads `runs/<id>/signals/kill-<attempt>.json` over S3 every 500 ms (`signalPollMs`). A runner on
|
|
932
|
-
a box of its own can't reach the supervisor's files, so its request travels through the disk. The runner must write
|
|
933
|
-
it durably for S3 to see it: a temporary file, fsync, rename, fsync of the directory. A missing object is a request
|
|
934
|
-
not written yet; any other read failure is retried. Once reads have failed for `signalFailMs` (60 s) without a
|
|
935
|
-
break, the iterator yields one `{ kind: "signals-failed", code, message }` and ends, so a parked runner never waits
|
|
936
|
-
with no word of why. `code` is the failing read's own error code when it has one (`CONTROL_API_FAILED` for a read
|
|
937
|
-
that timed out), else `SIGNALS_FAILED`. The iterator never rejects.
|
|
938
|
-
- A request names `point` and `n`, and every other field it carries stays in the signal's `crossing`, except the
|
|
939
|
-
per-attempt nonce the supervisor put in the order's env. That one is lifted to the signal's own `nonce` (when it is a
|
|
940
|
-
string), where the supervisor reads it for its check, as from the local place. The place's token never goes into a
|
|
941
|
-
request: on a disk it is the mount token, and the app never holds it.
|
|
942
|
-
- `release()` stops every instance the place started and removes every token it minted. What it could not stop or
|
|
943
|
-
remove, it reports (`RELEASE_FAILED`) and tries again on the next `release()`. The run's directory stays
|
|
944
|
-
(`deleteRunTree`).
|
|
945
|
-
|
|
946
|
-
A runner's `exited` says how it ended. Locally that is the exit code or signal, and the outcome is the instance's
|
|
947
|
-
`{"outcome":{...}}` line on stdout. Without that line, or on a disk (a box's stdout is out of reach), the outcome is
|
|
948
|
-
`run.json`'s, with its generation, and `code`, `message`, `record` and `storeSha256` from the detail of the last status
|
|
949
|
-
the app set. Its `status` is the run's word, not the record's, so a supervisor needs no package vocabulary: `complete`
|
|
950
|
-
when the run is `done` and that detail carries a record, `failed` for anything else (done with no record, failed, or a
|
|
951
|
-
run its instance left `running`, `paused` or `sleeping`). So an app that runs on a box and wants its supervisor to have
|
|
952
|
-
its record puts the record there, in its final `setStatus("done", { record })`.
|
|
953
|
-
|
|
954
|
-
Either way the outcome may carry `models`, the role-to-model ids the run's live parts chose (`lead`, `eyes`, ...), on any
|
|
955
|
-
status, a failed run's too: from the outcome line's `models`, or from the final detail's (`setStatus("done", { record,
|
|
956
|
-
models })`). Both places keep it only when it is a plain object whose values are all strings; a missing or malformed one
|
|
957
|
-
is left out, never replaced by `{}`.
|
|
958
|
-
|
|
959
|
-
## Tests
|
|
960
|
-
|
|
961
|
-
```sh
|
|
962
|
-
npm test # unit tests: no Archil, no network, no model
|
|
963
|
-
npm run typecheck
|
|
964
|
-
```
|
|
965
|
-
|
|
966
|
-
The package and its unit tests run on Node 22.19 or later.
|
|
967
|
-
|
|
968
|
-
The live suites (`npm run test:live:claim`, `:env`, `:run`, `:store`, `:supervise`) mount a real scratch disk, and
|
|
969
|
-
`npm run test:live:docker` runs the Docker driver's takeover test against this machine's Docker daemon. See
|
|
970
|
-
`test/live/README.md` for what they need and what they clean up.
|
|
239
|
+
| `openDurableRun(ref, options)` | Open a run: take the claim and the owner lock, open the store and the Harness, resume, keep the lease. `run.release()` seals it | [pick-up-a-run](examples/pick-up-a-run/main.ts) |
|
|
240
|
+
| `localRun(root)` | The `acquire` and `claimDir` that put a run on a plain directory of this machine, with no disk and no key | [pick-up-a-run](examples/pick-up-a-run/main.ts) |
|
|
241
|
+
| `seed` (an option of `openDurableRun`) | Write what a new run starts with, its store file and workspace files, once, before anything sees the run ([Seeding a new run](docs/reference.md#seeding-a-new-run)) | [seed-a-run](examples/seed-a-run/) |
|
|
242
|
+
| `localPlace({ app })` and `archilPlace({ disk, region, control, host })` | Where one run lives and how its instance starts and ends, for a supervisor of your own ([Places](docs/reference.md#places-where-a-runs-instance-starts-and-ends)) | [place-outcome](examples/place-outcome/) (`localPlace`, no key); `archilPlace` needs Archil: [`test/place.test.ts`](test/place.test.ts) |
|
|
243
|
+
| `PlaceOutcome` | How a runner ended: the run's `status` (`complete` or `failed`), its `generation`, and the app's `record` or `code` and `message`, and the `models` it used | [place-outcome](examples/place-outcome/) |
|
|
244
|
+
| `withClaim(where, fn)` and `readReleased(where, file)` | Hold one path of a disk exclusively for a moment, outside any run; read what it released over S3, with no mount ([A short claim](docs/reference.md#a-short-claim-on-any-path-and-a-read-of-what-it-released)) | Archil only: [`test/with-claim.test.ts`](test/with-claim.test.ts) (fakes) and [`test/live/with-claim.live.test.ts`](test/live/with-claim.live.test.ts) |
|
|
245
|
+
| `recordHome` (a field of a `Place`) | The directory an attempt's runner records into, and `bringHome(into, { final })`, which reads it back over the control with no claim ([Places](docs/reference.md#places-where-a-runs-instance-starts-and-ends)) | Archil only: [`test/place.test.ts`](test/place.test.ts), [`test/live/record-home.live.test.ts`](test/live/record-home.live.test.ts) |
|
|
246
|
+
|
|
247
|
+
Also exported: `ensureRunning` and the host drivers (`localHost`, `dockerHost`, `daytonaHost`), `serveRun` and `requestRun`,
|
|
248
|
+
parking, `fork` and `forkLocal`, `readLocalRunStatus` and `readRunStatus`, `archilControl`, and the typed errors.
|
|
249
|
+
`@parcha/pi-durable-disk/lease` is the narrow entry for a host that owns its Harness; `@parcha/pi-durable-disk/internals` has
|
|
250
|
+
no compatibility promise.
|
|
251
|
+
|
|
252
|
+
## Reference
|
|
253
|
+
|
|
254
|
+
[docs/reference.md](docs/reference.md) has the rest: [requirements](docs/reference.md#requirements), the quickstarts on a disk ([Docker](docs/reference.md#quickstart-with-docker), [a Linux host](docs/reference.md#quickstart-on-a-linux-host-two-hosts-one-run)), [the app](docs/reference.md#the-app), [the CLI](docs/reference.md#the-cli), [host drivers and the supervisor](docs/reference.md#host-drivers-and-the-supervisor), [the failure table](docs/reference.md#the-failure-table), [security](docs/reference.md#security), [the layout on the disk](docs/reference.md#layout-on-the-disk) and [the library](docs/reference.md#library).
|
|
971
255
|
|
|
972
256
|
## License
|
|
973
257
|
|