@parcha/pi-durable-disk 0.1.0-beta.12 → 0.1.0-beta.14

package/README.md

@@ -14,6 +14,20 @@ release names no other backend and promises none.
 pi-durable assumes two things it does not provide: exactly one live process per store, and a store that outlives the
 machine. A Cloudflare Durable Object gives it both. This package gives it both on an Archil disk you mount on any Linux host.
 
+## Choose your path
+
+| You have | Start with | What you see |
+|---|---|---|
+| Linux and Node 22.19 or later, nothing else | [Quickstart on one machine](#quickstart-on-one-machine-no-disk-no-key) | A durable chat on a directory of this machine, killed with `SIGKILL` and finished by a second process. One machine and no fence: the directory is the claim |
+| Linux and Node 22.19 or later, and an app module | `pi-durable-disk run --dir PATH --app MODULE` ([The CLI](#the-cli)) | Your app, or example 01's (`examples/01-durable-chat/app.ts`), as an instance on a directory of this machine: `kill -9` it and run the same command again, and it resumes. One machine and no fence |
+| macOS and Node 22.19 or later, for development | The same [quickstart](#quickstart-on-one-machine-no-disk-no-key) and `run --dir` | The same runs, development grade: commands keep your rights, and the file tools' confinement holds against the agent's paths but not a command racing it ([On macOS](#on-macos)). Use Linux or Docker when the agent is not trusted |
+| Docker (macOS or Linux) and an Archil account | [Quickstart with Docker](#quickstart-with-docker) | A host killed in the middle of a paid charge, and the run resumed on another, every instance in a container |
+| A Linux host with FUSE and an Archil account | [Quickstart on a Linux host](#quickstart-on-a-linux-host-kill-a-host-watch-the-run-resume-on-another), then [Setting up a host for production](#setting-up-a-host-for-production) | The production shape: systemd units, the supervisor, and a lost host's run back in about a second |
+| An app or a host of your own | [The app](#the-app) and [Library](#library) | `openDurableRun`, `localRun`, the supervisor and host drivers, `serve`, parking and `fork` from your code |
+
+[examples/](examples/README.md) lists the examples, one property each, with what each needs: several run with no disk
+and no key.
+
 ## How it maps to a Durable Object
 
 | Durable Object | This package | Supplied by |
@@ -48,17 +62,21 @@ machine. A Cloudflare Durable Object gives it both. This package gives it both o
 - An Archil account, a disk, and an API key for the supervisor, all from the Archil console. Create the disk in the region
   nearest the host (any size; its id looks like `dsk-0123456789abcdef` and its region like `aws-us-east-1`). Hosts and
   instances never hold the API key.
-- `@earendil-works/pi-durable` and `@earendil-works/chord`, `^1.0.4`, as peer dependencies.
+- `@earendil-works/pi-durable` and `@earendil-works/chord`, `>=1.0.4 <1.4.0`, as peer dependencies. The package is developed against 1.1.0, and CI runs its whole suite on 1.0.4 too, the lowest version the range accepts, and on 1.2 and 1.3.
+
+Not covered yet: a Kubernetes host driver (the `HostDriver` interface is below) and running the loop across machines
+where FUSE is unavailable. Without Archil, a run can live on a plain directory of one machine
+([A run on a local directory](#a-run-on-a-local-directory)).
 
-Not covered yet: a Kubernetes host driver (the `HostDriver` interface is below) and running the loop where FUSE is
-unavailable.
 
 ## Install
 
 ```sh
-npm install @parcha/pi-durable-disk@beta @earendil-works/pi-durable @earendil-works/pi-ai @earendil-works/chord
+npm install @parcha/pi-durable-disk@beta @earendil-works/pi-durable@~1.2.0 @earendil-works/pi-ai@~1.2.0 @earendil-works/chord@~1.2.0
 ```
 
+The pi packages are pinned to 1.2: pi 1.3.0 pins its own packages to exact versions, and this release's ranges stop below 1.3 until the next one.
+
 From a checkout (this package is `packages/pi-durable-disk` of the repository; the root build compiles `dist/`):
 
 ```sh
@@ -66,6 +84,115 @@ git clone https://github.com/Parcha-ai/agentrun.git && cd agentrun && npm ci --i
 cd packages/pi-durable-disk
 ```
 
+## Quickstart on one machine: no disk, no key
+
+The fastest look, on Linux with Node 22.19 or later (or on macOS, [for development](#on-macos)): a durable chat on a
+directory of this machine, with [`localRun`](#a-run-on-a-local-directory) instead of a disk and pi-ai's faux model
+instead of a key. One process answers the first messages and is killed (`SIGKILL`) after its third answer; a second
+opens the same run and answers the rest, and the transcript, read back from the store, has every message answered once.
+The directory is the claim, so this guards one machine only; the quickstarts below put the run on a disk that outlives
+the machine.
+
+In an empty directory, install the package and its peers, save this as `chat.mjs`
+(it is [examples/01-durable-chat/on-a-directory.mjs](examples/01-durable-chat/on-a-directory.mjs)):
+
+```sh
+npm install @parcha/pi-durable-disk@beta @earendil-works/pi-durable@~1.2.0 @earendil-works/pi-ai@~1.2.0 @earendil-works/chord@~1.2.0
+```
+
+```js
+// A durable chat on a directory of this machine: no disk, no key (pi-ai's faux model answers). One process answers the
+// first messages and is killed (SIGKILL) after its third answer; a second opens the same run and answers the rest, and
+// the transcript, read back from the store, has every message answered once. One machine: the directory is the claim.
+import { spawn } from "node:child_process";
+import { rmSync } from "node:fs";
+import { join } from "node:path";
+import { createInterface } from "node:readline";
+import { fileURLToPath } from "node:url";
+import { BACKGROUND_CONTEXT as ctx } from "@earendil-works/chord/context";
+import { createModels, fauxAssistantMessage, fauxProvider } from "@earendil-works/pi-ai";
+import { createRegistry } from "@earendil-works/pi-durable";
+import { localRun, openDurableRun } from "@parcha/pi-durable-disk";
+
+const MESSAGES = ["Where does your transcript live?", "What happens if this process is killed now?", "Who answers after that?", "What was my first question?", "Are you the same process?", "Say goodbye."];
+const root = join(process.cwd(), "chat-run");
+const text = (message) => (typeof message.content === "string" ? message.content : message.content.map((part) => part.text ?? "").join(""));
+
+if (process.argv[2] === "instance") {
+  let generation = 0;
+  const faux = fauxProvider();
+  faux.setResponses(Array.from({ length: 100 }, () => (context) => {
+    const asked = context.messages.filter((message) => message.role === "user");
+    return fauxAssistantMessage(`Answer ${asked.length}, by generation ${generation}: "${text(asked.at(-1))}"`);
+  }));
+  const models = createModels();
+  models.setProvider(faux.provider);
+  const run = await openDurableRun({ disk: "local", region: "local", id: "chat" }, { ...localRun(root), harness: { models, registry: createRegistry() } });
+  generation = run.generation;
+  const model = faux.getModel();
+  const chat = await run.harness.root(ctx, { agent: { model: { provider: model.provider, modelId: model.id } } });
+  const answers = async () => (await chat.entries({}, 100, undefined, ctx)).items.reverse().filter((entry) => entry.kind === "pi.assistant");
+  const before = (await answers()).length;
+  console.log(`generation ${generation} opened the run in ${root} (${before} answered before)`);
+  for (const [i, content] of MESSAGES.entries()) {
+    // The request id makes a message a no-op once it was admitted: a new generation walks the list from the top.
+    await (await chat.submit({ type: "input", content, requestId: `message-${i + 1}` }, ctx)).wait(ctx);
+    if (i >= before) console.log(`generation ${generation} answered message ${i + 1}`);
+    await new Promise((resolve) => setTimeout(resolve, 200));
+  }
+  for (const answer of await answers()) console.log(`  ${text(answer.model[0])}`);
+  await run.setStatus("done");
+  await run.release();
+  process.exit(0);
+}
+
+// The supervisor: an instance, killed after its third answer, then another on the same directory (a fresh one each time).
+// It reads whole lines, and waits for `close` (the process gone and all its output read), not `exit`.
+rmSync(root, { recursive: true, force: true });
+const start = () => spawn(process.execPath, [fileURLToPath(import.meta.url), "instance"], { stdio: ["ignore", "pipe", "inherit"] });
+const eachLine = (child, onLine) =>
+  createInterface({ input: child.stdout }).on("line", (line) => {
+    console.log(line);
+    onLine(line);
+  });
+const first = start();
+eachLine(first, (line) => {
+  if (/answered message 3\b/.test(line)) first.kill("SIGKILL");
+});
+await new Promise((resolve) => first.on("close", resolve));
+console.log("the first process was killed (SIGKILL); a second opens the same run");
+const output = [];
+const second = start();
+eachLine(second, (line) => output.push(line));
+const code = await new Promise((resolve) => second.on("close", resolve));
+const answered = output.flatMap((line) => {
+  const match = /^ {2}Answer (\d+), by generation (\d+)/.exec(line);
+  return match ? [{ n: Number(match[1]), generation: Number(match[2]) }] : [];
+});
+const once = MESSAGES.every((_, i) => answered.filter((a) => a.n === i + 1).length === 1) && answered.length === MESSAGES.length;
+const generations = [...new Set(answered.map((a) => a.generation))].sort();
+if (code !== 0 || !once || generations.length < 2) {
+  console.error(`not ok: exit ${code}, answers ${JSON.stringify(answered)}`);
+  process.exit(1);
+}
+console.log(`ok: ${MESSAGES.length} messages answered once each, by generations ${generations.join(" and ")}, from the store in ${root}`);
+```
+
+Then run it (about five seconds; it writes the run under `./chat-run/`):
+
+```sh
+node chat.mjs
+```
+
+It ends with:
+
+```
+  Answer 1, by generation 1: "Where does your transcript live?"
+  ...
+  Answer 6, by generation 2: "Say goodbye."
+ok: 6 messages answered once each, by generations 1 and 2, from the store in <this directory>/chat-run
+```
+
 ## Quickstart with Docker
 
 You need Docker (Docker Desktop, OrbStack or Colima on macOS, Docker Engine on Linux), Node 22.19 or later, and an Archil
@@ -140,7 +267,7 @@ Each example's README lists the exact commands the demo runs, so you can run the
 
 An instance is `pi-durable-disk run --app <module> ...`: `openDurableRun` with the app's options. The module is an ES
 module whose default export is a function of where the run lives and returns pi's Harness options without `env` (the run
-builds `env` on its claim: the workspace is `<root>/work`), plus an optional `onOpen`:
+builds `env` on its claim: the workspace is `<root>/work`), plus an optional `beforeResume` and an optional `onOpen`:
 
 ```ts
 import { BACKGROUND_CONTEXT as ctx } from "@earendil-works/chord/context";
@@ -159,6 +286,12 @@ export default async function app(where: AppContext): Promise<AppOptions> {
   return {
     models,
     registry,
+    async beforeResume({ harness, context, generation, previous }) {
+      // Awaited once per incarnation, after the Harness is open and before it resumes: no resumed task takes a step
+      // until this settles, and what it admits commits before their steps do. Reconcile here what a resumed tool depends on.
+      // previous is run.json as the run was left (null when there was none): `sleeping` with its wakeAt after a park,
+      // `paused` with its detail after a planned stop or a fork, `running` with the dead holder and no new seal after a crash.
+    },
     async onOpen(run) {
       // Called once per incarnation, after the run is open and resumed. run.generation is 1 on a first start and
       // higher on every resume. run.harness is pi-durable's Harness.
@@ -176,7 +309,11 @@ rerun. Any other tool is an effect: its intent is committed before it starts, an
 was `interrupted`, never run again. `run.setStatus("done" | "failed" | "paused" | "sleeping", detail)` records the run's
 status in `run.json`; `done` and `failed` run the durability barrier first. `run.release()` closes the Harness, kills the
 run's commands, runs the barrier, seals `run.json` with the store's last sequence, and unmounts. A module that is missing
-or throws, or an `onOpen` that rejects, is `AppError` (exit 1, which the unit retries).
+or throws, or an `onOpen` that rejects, is `AppError` (exit 1, which the unit retries). A `beforeResume` that rejects
+fails the open (its step is `before-resume`) and releases the claim. Besides the Harness, the hook gets `storage`, the
+store the Harness was opened over as four reads (`scanConversations`, `scanTasks`, `scanDocuments`, `document`) with no
+way to a write, and `head`, the store's last committed sequence: a recorder snapshots every document at `head` there.
+Until the hook commits, the current state is the state at `head`, which is how a document that keeps no history reads.
 
 ## The CLI
 
@@ -184,8 +321,10 @@ or throws, or an `onOpen` that rejects, is `AppError` (exit 1, which the unit re
 
 | Command | What it does |
 |---|---|
+| `run --dir PATH --app MODULE [--id ID]` | The instance on a local directory (`localRun`): no disk, no mount token, no key. The same app contract and exit codes as on a disk (76 when another process holds the run, or when a process its last incarnation left will not die; 75 when fenced), and the same run flags but the Archil ones (`--disk`, `--region`, `--mount-root`, `--archil`, `--token-stdin` are refused beside it, exit 2). `--id` defaults to the directory's name |
 | `run --disk D --region R --id ID --app MODULE` | The instance a host driver starts: claim the run, open it, resume it. `--token-stdin` reads the mount token from stdin. Lease flags: `--heartbeat-ms`, `--lease-expiry-ms`, `--lease-margin-ms`. `--on-sigterm resume` (default) or `pause`. `--serve PORT`, `--park-threshold`, `--park-idle`, `--drain-timeout` (below). `--run-as USER` runs the agent's commands as USER when the instance runs as root (in a container) |
 | `supervise --disk D --region R (--id ID ... \| --all) [--every 30s] --app MODULE` | One decision per run, once or in a loop: nothing to do, start, or revoke and start; `--app` is the module each started instance runs. `--create` makes the run directory. `--check` proves this host's fence first. `--sweep-tokens` removes expired token users. `--host docker --image IMAGE` starts each instance in a container (`--app-root`, `--fleet`, `--name-prefix`, `--run-as`, `--docker-arg`, `--apparmor`). Prints one JSON line per decision |
+| `status --dir PATH` | `run.json` read from the directory, and whether its holder is still the process that claimed it: `running` (this boot, its pid alive with the start ticks `run --dir` recorded), `stopped`, `gone` (another boot), or `unknown` when this machine's boot or processes cannot be read |
 | `status --disk D --region R --id ID` | `run.json` over the S3 API, the run's delegations and the holder's state |
 | `release --id ID` | Unmount the run's mount on this host (a dead mount is cleaned) |
 | `fork --id A --new-id B` | Copy a released, sealed run into a new run (see Serve, parking and fork) |
@@ -356,6 +495,12 @@ only the claim probe `.claim` is rewritten by the mount); `run.json`, `owner.loc
 that fails after that empties the directory through its own mount and removes it, and one that lost the directory to another
 fork or start leaves it alone.
 
+On a local directory, `forkLocal(from, [{ id, root }, ...])` makes the same copy without a disk: one new run directory per
+entry, from a directory of a run released and sealed (`SOURCE_NOT_RELEASED` for anything else, `TARGET_EXISTS` for a target
+that is there). The source is only read; every copy is made beside its target under a temporary name and none is published until all are complete, so a failed
+batch leaves no half copy. A target inside the source (also through a symlink) is `INVALID_ARGUMENT`. A local directory has no mount to take it exclusively, so nothing stops a process that opens the source while it
+is being copied: fork a run after it is released, and open the forks with `localRun` like any run.
+
 ### Setting up a host for production
 
 The Quickstart runs the supervisor as your own user. On a production host the instance and the agent's tools run as an
@@ -398,6 +543,8 @@ mount token reaches the journal, and that nothing is left behind. It needs syste
 |---|---|---|
 | The instance crashes or is OOM-killed, host alive | The unit restarts it in place on the same claim; the unit's cgroup kills the previous incarnation's commands; reopen, `resume()` | running tasks go back to pending; effects in flight are `interrupted`, never rerun; at most one cut model request is sent again |
 | The host is lost (power-off, VM loss, pre-emption) | The client's sockets close and Archil flags the delegation orphaned in 0.2 s; the supervisor revokes (tens of ms) and starts an instance elsewhere | every acknowledged commit is there; unflushed workspace writes are lost; no effect whose intent was committed runs again |
+| A Docker host is lost before its instance reaches running (the container killed or removed, or its daemon gone, between the start and the instance's first `run.json` write) | If the driver's start is still running, it fails typed (`START_FAILED`) and tries to remove the container it made (a removal that fails, as it does while the daemon is gone, is not retried); the supervisor's tick fails `HOST_START_FAILED`, counts no failed start, and the next tick starts again. If the start had returned, the supervisor's start mark (`runs/<id>/start.json`) keeps the run `starting` until the start grace passes (default the lease expiry, 90 s, doubled per consecutive failed start up to 10 min), so nothing is revoked or started twice; then it starts again, revoking first if the container had mounted (its FUSE daemon died with it, so its delegation is orphaned), and that start tries to remove the run's earlier dead containers. Removal is attempted, never guaranteed: once the daemon is back, look for leftovers with `docker ps -a --filter label=pda.run=<id>` | nothing is lost: an instance writes `run.json` at its generation before it opens the store, so it had committed nothing. The run has no instance for up to the grace; the time from the loss to the next instance's first commit is not measured on Docker |
+| A Daytona host is lost before its instance reaches running (the box errors, is deleted or never starts while the driver starts it, or is lost after the launcher returned and before the instance's first `run.json` write) | While the driver starts it, a box that goes to an error state or away, does not start within the start timeout, or refuses an upload gets a delete request (attempted: a failed delete is not retried) and the start fails typed (`START_FAILED`); the supervisor counts no failed start and starts again on the next tick. Lost after the start returned: as on Docker, `starting` until the grace passes, then revoke if its client had mounted, and start a new box. A box deletes itself once stopped and Daytona destroys it at its TTL (default 25 h), so a box whose delete failed leaks nothing past that | as on Docker: nothing committed, no instance for up to the grace; not measured on Daytona |
 | The host hangs or is partitioned (a zombie) | It is not orphaned, so the lease expires at 90 s; the supervisor stops the host if its driver can, revokes, and starts elsewhere | the zombie's next commit fails (`SQLITE_IOERR_FSYNC`) and it exits 75; a call it already has in flight may complete, but it cannot start a new one, because a new call needs an intent commit |
 | A zombie that never commits again | Its lease watchdog (a worker thread, so a main thread stuck in a FUSE request does not matter) kills its commands at 75 s without touching the mount | an in-flight command runs at most until the self-fence |
 | The mount is taken away under a live instance (an operator's `umount -l`, a cleanup's `fusermount -u`) | The instance writes `run.json` and creates its directories through the directory it opened at acquire, and checks before each write that the run's path is still that mount; at the next heartbeat the path names the local directory under the mountpoint, so it exits 75 (`CLAIM_UNMOUNTED`) | nothing of its own is written to the local disk; as host lost |
@@ -409,17 +556,38 @@ mount token reaches the journal, and that nothing is left behind. It needs syste
 | A second instance on the same host | The owner lock (an exclusive SQLite lock on `owner.lock`) is refused, exit 76 | none |
 | Two supervisors race | The mount admits one claimant; the other instance exits 76 | none |
 | The control API lists the run's holder without a path (it resolves paths best-effort) | The supervisor maps pathless delegations' inodes to run directories with one `Disk.exec` per pass (`find -inum`) and counts one on the run's directory as the run's: a live one keeps the run healthy, an orphaned one is revoked and the run restarted. Where `exec` is unavailable or fails, the pass is `CONTROL_API_FAILED` and starts nothing (the supervisor never forces); a dead holder then stays until Archil lists its path again or an operator revokes it by client and inode | no second instance; nothing started over a holder it cannot see |
+| A client dies holding its own private directories (`.archil/client-<clientId>` and its `unlinked/`, which the control API lists with no path and on no run directory) | They block no run. Every revoke that fences or deletes a run (`revoke`, `takeOver`, the supervisor's takeover, `deleteRunTree`) also revokes, from the same listing, the private-directory delegations of the clients it revokes on the run, confirmed by one `Disk.exec` (`stat`), unless the client holds anything else; best effort, so one left stays for an operator to revoke by client and inode | none: they hold no run directory |
 | The store is behind its `run.json` seal | `STORE_BEHIND_SEAL`: `run.json` is marked failed and the instance exits 65; the supervisor does not restart it | none started |
 | SIGTERM (a deploy) | The instance stops taking submissions and drains for up to `--drain-timeout`, writes `sleeping` with its wake (now; the deadline when everything only waits; null when idle and woken by requests), then closes, kills its commands, barriers, seals and unmounts, exit 0; the supervisor starts it again at the wake | work cut at the drain's deadline resumes on the next open: safe tools rerun, unsafe ones report the interruption |
 | Everything the run does is a retry or poll wait longer than the park threshold | The instance writes `sleeping` with `wakeAt` (the deadline), releases and exits 0; the supervisor starts it at `wakeAt` and pi's timer sleeps whatever is left; a `run.json` write that fails is a fence (exit 75) and the lease path resumes the run | nothing is lost: the wait is a checkpoint in the store |
 
+**On a local directory** (`localRun`) nothing is mounted or delegated and no supervisor watches the run, so the rows above
+about revocation, takeover and the lease's replacement do not apply. These do:
+
+| Failure | What happens | Effects |
+|---|---|---|
+| The instance crashes or is SIGKILLed | Its owner lock goes with the process. The next open of the directory (by the app, or whatever restarts it) takes the run, ends the commands the killed incarnation left running (its `commands` step, before the Harness opens), and `resume()`s from the last commit | running tasks go back to pending; effects in flight are `interrupted`, never rerun; no command of the killed incarnation still runs once the run resumes. pi makes each command the leader of its own process group, so the SIGKILL does not reach them and a local run has no unit cgroup: the instance records each command's group under `commands/` (by pid, start ticks and boot) and the next open kills what still matches. A command started within about 0.1 s before the kill can be missed |
+| A process the killed incarnation's commands started does not die at the next open (an uninterruptible wait, or one the run's user may not signal) | The open is refused: `HeldError` `COMMANDS_ALIVE`, exit 76, with the message below; the records stay | none: the run does not resume beside it. End those processes and open the run again |
+| Power loss or an OS crash | The store's commits are durable (SQLite at `synchronous = FULL`); `work/` is the operating system's to flush, since the claim's barrier syncs nothing | the store resumes from its last commit; files the agent wrote in `work/` can be older than what the store recorded |
+| A second process opens the run on this machine | The owner lock is refused (`HeldError`, exit 76) | none |
+| A process on another machine reaches the directory (a network share) | Nothing stops it: a local directory has no fence | unsupported, since two writers can corrupt the store. A directory another machine can reach needs a disk with a claim and a fence |
+
+The refusal names each process, as the open writes it (here one, in a run at `<root>`):
+
+```
+1 process(es) a killed incarnation's commands started are still alive after SIGKILL: pid 4242 (tar -xf /mnt/slow/archive.tar). The run does not resume beside them. End them (kill -9 4242, as their owner, or wait for them) and open the run again; they stay recorded in <root>/commands until then.
+```
+
 What a revoke cannot do: no fence reaches a third party. A paid API that already has a request has it. The package
 guarantees that this runtime does not start an effect twice; an effect your model decides to retry is a new call, so give a
 paid API an idempotency key from your own business key.
 
 Measured on a VM about 3 ms from `aws-us-east-1`, archil 0.8.42: acquire 222 ms p50, release 139 ms, takeover by revoke
 252 ms, commit 3.75 to 5.63 ms p50, a killed host's run back to its first commit in 1 to 1.5 s, a hung host replaced in 6
-to 6.3 s with a 6 s test lease (90 to 120 s with the defaults).
+to 6.3 s with a 6 s test lease (90 to 120 s with the defaults). A start that never reaches running costs its start grace: live on Archil with a 6 s grace and a 3 s tick,
+starts that all failed were retried six times in five minutes, each grace doubled (`test/live/supervise-slow.live.test.ts`),
+and a takeover whose start was slowed 5 s before its first `run.json` write started once and was never revoked, 5 of 5
+(`test/live/supervise.live.test.ts`). On Docker and Daytona hosts, a start lost before its instance runs is not measured.
 
 ## Security
 
@@ -435,19 +603,22 @@ to 6.3 s with a 6 s test lease (90 to 120 s with the defaults).
 - **The API key stays with the supervisor.** Hosts and instances never hold it, and the local driver never copies its own
   environment into a unit.
 - **No privilege for the agent.** The instance, pi's tools and the agent's commands run as an unprivileged user. Every
-  command starts under `setpriv --no-new-privs`, so a command cannot use the run user's sudo rule (sudo refuses: "The no new
-  privileges flag is set"). An app that turns this off (`archilEnv(claim, { noNewPrivs: false })`) or runs commands another
-  way must confine them itself. Commands run next to the FUSE client on the host that holds the claim: if the agent is not
-  trusted with the host, sandbox its commands (for example with bubblewrap) and keep the FUSE client outside the sandbox.
+  command starts under `setpriv --no-new-privs` (on Linux; macOS has no equivalent, see [On macOS](#on-macos)), so a
+  command cannot use the run user's sudo rule (sudo refuses: "The no new privileges flag is set"). An app that turns
+  this off (`archilEnv(claim, { noNewPrivs: false })`) or runs commands another way must confine them itself. Commands
+  run next to the FUSE client on the host that holds the claim: if the agent is not trusted with the host, sandbox its
+  commands (for example with bubblewrap) and keep the FUSE client outside the sandbox.
 - **The file tools stay inside `work/`.** pi's `read`, `write` and `edit` run in the loop's process, not in a command
-  sandbox, so every file operation of the environment resolves its path to the file the kernel would reach (`..`, absolute
-  paths, symlinks, dangling links, a missing tail) and refuses it unless it lies in `work/` (reads may also use `tmp/`, where
-  the bash tool tells the model its long output went, and `readRoots` the app names). An agent cannot `write
+  sandbox, so every file operation of the environment resolves its path to the file the kernel would reach (`..`,
+  absolute paths, symlinks, dangling links, a missing tail) and refuses it unless it lies in `work/` (reads may also use
+  `tmp/`, where the bash tool tells the model its long output went, and `readRoots` the app names; writes may also go
+  inside a directory the environment's `createTempDir` returned, for the tool it was handed to). An agent cannot `write
   ../store/run.sqlite`, `../run.json` or `../owner.lock`: the call fails with a `PathOutsideWorkError` the model sees as
   a failed tool result. A path is opened one component at a time from a held handle of `work/`, without following links,
-  so a link a command swaps in after the check is refused too. This covers the file tools only; a command still runs as
-  the run user and can reach anything that user can, which is what an app's own command sandbox is for. `archilEnv(claim, {
-  confineFiles: false })` turns it off where `/proc/self/fd` is unavailable.
+  so a link a command swaps in after the check is refused too (on Linux; on macOS the walk is by path, see [On
+  macOS](#on-macos)). This covers the file tools only; a command still runs as the run user and can reach anything that
+  user can, which is what an app's own command sandbox is for. `archilEnv(claim, { confineFiles: false })` turns it off
+  where `/proc/self/fd` is unavailable.
 - **`--serve` authenticates only with a bearer token.** On 127.0.0.1 (the default) the token is optional. A non-loopback
   `--serve-host` requires `--serve-token-file`, a file of mode 0600 holding the token (never argv or the environment); every
   request must send `authorization: Bearer <token>`, and it is compared in constant time. The file is readable by the run
@@ -473,6 +644,9 @@ to 6.3 s with a 6 s test lease (90 to 120 s with the defaults).
   tmp/                     pi's temp files and long-output spills, so a path a result names still exists after a takeover
 ```
 
+A run on a local directory (`localRun`) has the same layout, plus `commands/`: the process groups of the commands its
+instance started, which the next open ends before it resumes (a local run has no unit cgroup to end them).
+
 The mountpoint is the same absolute path on every host of a deployment, so paths inside the run are identical wherever
 it resumes.
 
@@ -484,26 +658,92 @@ its objects, and remove its token users (`supervise --id ID --sweep-tokens --tok
 
 **Revoke before you delete.** Deleting the objects of a run through the S3 API while a delegation on the path is still
 held, even an orphaned one left by a killed host, does not take effect: the same run id came back with its old store when a
-supervisor next started it. `deleteRunTree(control, id)` does it in order: it revokes the run's delegations, deletes, and
-requires the prefix to be empty. Doing it by hand, revoke the delegation first, delete, then list the prefix and check it is
-empty.
+supervisor next started it. `deleteRunTree(control, id)` does it in order: it revokes the run's delegations and its holders' private directories,
+deletes, and requires the prefix to be empty. Doing it by hand, revoke first (`revoke(control, id)` takes the same set), delete,
+then list the prefix and check it is empty.
 
 ## Library
 
 `openDurableRun(ref, options)` is what `run` calls: mount, verify the claim, take the owner lock, write `run.json`, open
 the store, check the seal, open and resume the Harness, keep the lease. `acquire`, `ensureRunning`, `localHost`,
-`archilEnv`, `openArchilStore` and the typed errors (`FencedError`, `HeldError`, `StoreBehindSealError`) are exported
-for apps that compose their own lifecycle, and so are serve (`serveRun`, `requestRun`), parking (`watchParking`, `drain`,
-`busyState`, and `leaseParkTarget` for a Harness a host opened over `openRunLease`) and `fork`. The `.d.ts` files in `dist/`
-are the reference.
+`archilEnv` and the typed errors (`FencedError`, `HeldError`, `StoreBehindSealError`) are exported for apps that compose
+their own lifecycle, and so are serve (`serveRun`, `requestRun`), parking (`watchParking`, `drain`, `busyState`, and
+`leaseParkTarget` for a Harness a host opened over `openRunLease`) and `fork`. `src/index.ts` lists every export; the
+`.d.ts` files in `dist/` are the reference.
+
+A supervisor of your own (a driver `supervise` does not start, such as `daytonaHost`) gets its `control` from
+`archilControl({ apiKey, disk, region })`: the disk's S3 and control API, the same object `supervise` builds from its flags.
+
+```ts
+const control = await archilControl({ apiKey: process.env.ARCHIL_API_KEY!, disk: "dsk-...", region: "aws-us-east-1" });
+await ensureRunning({ disk: "dsk-...", region: "aws-us-east-1", id: "job-1" }, daytonaHost({ client, snapshot }), { control });
+```
 
 A host that owns its own pi-durable Harness and store connections imports the narrow entry instead:
 `@parcha/pi-durable-disk/lease` holds `openRunLease` (the claim, the owner lock, `run.json` with its heartbeat, the lease
 self-fence, the seal and the release, without a store or a Harness), `storeHead`, the store's fence and pragma helpers
-(`Fence`, `FencedDatabase`, `assertPragmas`, `PROFILE_PRAGMAS`, `openArchilStore`), and the typed errors and exit codes. It
-leaves out the supervisor, the host drivers, the CLI and the app contract, and the package declares `"sideEffects": false`, so
-a bundler keeps only what you use: a bundle of the app with `await import("@parcha/pi-durable-disk/lease")` is about 52 KB
-minified against about 493 KB for `await import("@parcha/pi-durable-disk")` (measured with esbuild, pi-durable inlined).
+(`Fence`, `FencedDatabase`, `assertPragmas`, `PROFILE_PRAGMAS`, `openArchilStore`), parking (`leaseParkTarget`,
+`watchParking`, `drain`, `busyState`), and the typed errors and exit codes. It leaves out the supervisor, the host drivers,
+the CLI and the app contract, and the package declares `"sideEffects": false`, so a bundler keeps only what you use: a
+bundle of the app with `await import("@parcha/pi-durable-disk/lease")` is about 63 KB minified against about 620 KB
+for `await import("@parcha/pi-durable-disk")` (esbuild `--bundle --minify --platform=node --format=esm`, which inlines
+pi-durable, chord and their dependencies).
+
+`@parcha/pi-durable-disk/internals` holds the claim, status and Daytona plumbing this repository's examples and demos use:
+`runPath`, `unmountClaim`, `createRunDir`, the mount-token and delegation helpers, `isLeaseFresh`, and the Daytona labels,
+names and `sweepSandboxes`. It carries no compatibility promise: a name there can change or go in any release.
+
+### A run on a local directory
+
+`localRun(root)` returns the `acquire` and `claimDir` that put a run on a plain directory of this machine, with no mount
+and no mount token:
+
+```ts
+import { localRun, openDurableRun } from "@parcha/pi-durable-disk";
+
+const run = await openDurableRun({ disk: "local", region: "local", id: "run-1" }, {
+  ...localRun("/var/runs/run-1"),          // made when it is missing; work/ and store/ inside it
+  harness: { models, registry },
+});
+```
+
+It is for one machine (a laptop, a CI job, a test), and says so: the claim is the directory itself, so nothing but this
+machine's owner lock keeps a second process off the run (a second open while one holds it is refused), and nothing at
+all keeps out a process on another machine that can reach the directory. Its barrier refuses once the run is fenced,
+like a mount's, but syncs nothing: the store's commits are made durable by SQLite, and the files the agent wrote in
+`work/` are the operating system's to flush, so a power loss can lose what it had not written yet. `mountToken` is
+required only with the default `acquire`; without either, the open is refused (`INVALID_ARGUMENT`) before anything mounts.
+
+#### On macOS
+
+A local run works on macOS for development, and is weaker there than on Linux in four ways. Use Linux, or the
+[Docker quickstart](#quickstart-with-docker), when the agent is not trusted.
+
+- **Commands keep your rights: `no_new_privs` has no macOS equivalent.** On Linux every command starts under
+  `setpriv --no-new-privs`, so it cannot use `sudo` or gain privilege from a setuid program. macOS has no such flag and nothing
+  that does its job (`sandbox-exec` is deprecated), so `noNewPrivs` defaults to false there and no `setpriv` runs. That
+  leaves open: a command can use `sudo` wherever sudo asks it for no password (a `NOPASSWD` rule, for one), and setuid
+  programs keep their privilege. `runAs` is not available either (it needs `setpriv`). As on Linux, a command runs as
+  you and can reach anything you can.
+- **The file tools' confinement is path-based.** On Linux it walks a path through `/proc/self/fd`, one held directory at
+  a time, so no component can be swapped once it was checked; macOS has no `/proc/self/fd`. There it resolves the path
+  (`realpath`) and refuses it unless it lies in `work/` or another root the file tools may use, as on Linux, opens each
+  component by its full path with `O_NOFOLLOW`, then checks that the path still resolves to itself and names the
+  directory it opened (same device and inode). It refuses `../store/run.sqlite`, absolute paths and links out of
+  `work/`, a link already in place included. That leaves open: a command that swaps a directory above the target for a
+  link between that check and the operation can win, and the file tool then acts where the link points.
+- **The run's root is addressed by its path.** On Linux, `run.json` and the run's directories are written through the
+  directory opened at acquire. On macOS each write is checked first against that directory, and a root renamed and
+  replaced between the check and the write gets the write.
+- **Commands are recorded from `ps`.** With no `/proc`, the instance reads one `ps` listing every 0.5 s while commands run,
+  `ps -E` for a command's mark and `sysctl kern.bootsessionuuid` for the boot. A command started within about 0.5 s
+  before a kill can be missed, and a process is told from a later one with the same pid by the second it started. The
+  lease watchdog reads the same listing.
+
+`fsync` on macOS does not flush the drive's cache, so the store also sets SQLite's `fullfsync` and
+`checkpoint_fullfsync` there (SQLite then syncs with `F_FULLFSYNC`), and refuses to open unless they read back. CI runs
+the no-key quickstart and the local run's tests (its command records, `run --dir` and `status --dir`, the lease watchdog,
+the file tools' confinement) on macOS.
 
 ## Tests
 

package/dist/app.d.ts

@@ -11,6 +11,11 @@ export interface AppContext {
 }
 /** pi's HarnessOptions without `env` (registry, models, settings, conversationCreated, now, onReport), plus the hooks below. */
 export type AppOptions = OpenDurableRunOptions["harness"] & {
+    /**
+     * Awaited once per incarnation after the Harness is open and before it resumes (`OpenDurableRunOptions.beforeResume`):
+     * no resumed task takes a step until it settles. A rejection fails the open (exit 1, retried).
+     */
+    beforeResume?: OpenDurableRunOptions["beforeResume"];
     /**
      * Called once per incarnation after the run is open and resumed; `run.generation` tells a first start from a resume.
      * Work that must keep the instance up belongs in pi's tasks and tools: parking sees only those.

package/dist/app.js

@@ -8,8 +8,9 @@ var __rewriteRelativeImportExtension = (this && this.__rewriteRelativeImportExte
 };
 // The app an instance hosts (`pi-durable-disk run --app <module>`): an ES module whose default export is a function
 // of where the run lives that returns pi's Harness options (without `env`, which the run builds on its claim), an
-// optional `onOpen` for the app to submit or resume work once the run is open and resumed, how `serve` creates the root
-// conversation, and how a parking run records its wake.
+// optional `beforeResume` for what must settle before the run resumes, an optional `onOpen` for the app to submit or
+// resume work once the run is open and resumed, how `serve` creates the root conversation, and how a parking run records
+// its wake.
 import { resolve } from "node:path";
 import { pathToFileURL } from "node:url";
 import { PdaError } from "./errors.js";

package/dist/claim.d.ts

@@ -55,7 +55,7 @@ export interface ArchilHost {
         cli?: number;
         staleGrace?: number;
     };
-    /** File access on the mount; replaceable so tests can inject a dead mount (ENOTCONN) or a fence (EIO). */
+    /** File access on the mount; replaceable so tests can inject a dead mount (ENOTCONN, ECONNABORTED) or a fence (EIO). */
     fs?: {
         stat?(path: string): Promise<unknown>;
         persist?(path: string, data: string): Promise<void>;
@@ -92,6 +92,12 @@ export interface Claim {
     readonly fenced: boolean;
     /** Mark the claim lost (the store's facade calls this on an I/O-class error); later barriers throw at once. */
     markFenced(cause?: unknown): void;
+    /**
+     * Where the run's environment records the commands it starts, so that the next open of this claim ends what a killed
+     * incarnation left running (`reapCommands`, before the Harness opens). Set by a claim that nothing else cleans up after,
+     * as a local run's; a mount's host ends them with its unit's cgroup or its container.
+     */
+    readonly commands?: string;
     /** `archil sync`: every pending write on the mount is durable when it resolves. A failure fences the claim. */
     barrier(): Promise<{
         ms: number;
@@ -174,10 +180,36 @@ export type PathlessResolver = (pathless: readonly Delegation[]) => Promise<Read
  * removes them.
  */
 export declare function pathlessResolver(control: Pick<ControlApi, "exec">): PathlessResolver;
-/** Revoke every delegation on the run through the control API. The old holder's next fsync returns EIO. */
+/**
+ * The delegations `held`'s clients also hold on their own private directories (`privateDirs`: inode to the client whose
+ * private directory it is). A killed client leaves them orphaned, and nothing ties them to a run: they block no run, but
+ * they pile up and resurface in listings. A client that holds anything else beyond `held` may hold another run, so none
+ * of its delegations is a companion.
+ */
+export declare function companionsOf(all: readonly Delegation[], held: readonly Delegation[], privateDirs: ReadonlyMap<number, string>): Delegation[];
+/**
+ * Revoke the companions of `held` (`companionsOf`): only for clients the caller is already revoking because it fences or
+ * deletes a run, never as a sweep. From `all` or a fresh listing. Best effort, and hygiene only: a companion blocks no
+ * run, so one left behind (or a failed listing or `exec`) costs a listing entry. Returns the ones revoked.
+ */
+export declare function revokeCompanions(control: Pick<ControlApi, "listDelegations" | "revokeDelegation" | "exec">, held: readonly Delegation[], all?: readonly Delegation[]): Promise<Delegation[]>;
+/**
+ * Revoke every delegation on the run through the control API, then its holders' private directories
+ * (`revokeCompanions`). The old holder's next fsync returns EIO. Returns the run's own delegations.
+ */
 export declare function revoke(control: ControlApi, id: string): Promise<Delegation[]>;
+/**
+ * Revoke what `revoke` revokes, skipping every revoke that fails: for cleanup, where deleting the tree next checks what
+ * is left. Only the listing (and a pathless resolution) can fail it. Returns the run's own delegations.
+ */
+export declare function revokeBestEffort(control: Pick<ControlApi, "listDelegations" | "revokeDelegation" | "exec">, id: string): Promise<Delegation[]>;
 /** Revoke through the control API, then mount plainly; if the API path fails, mount with `--force`. */
 export declare function takeOver(control: ControlApi, opts: AcquireOptions): Promise<Claim>;
+/**
+ * The errno codes of a stat on a FUSE mount whose daemon is gone: ENOTCONN once its connection is torn down, ECONNABORTED
+ * for a stat already in flight while it is torn down. Codes, never messages: a message is translated by the locale.
+ */
+export declare const DEAD_CONNECTION: ReadonlySet<string>;
 /**
  * Unmount whatever the run has at `mountpoint`: `archil unmount` flushes and checks the delegation in; a dead mount
  * (which `archil unmount` refuses) is cleaned with `fusermount -u`; a live mount refused with ENOENT (Sysbox) is checked