flyloft 0.2.2__tar.gz → 0.3.0__tar.gz

{flyloft-0.2.2 → flyloft-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: flyloft
-Version: 0.2.2
+Version: 0.3.0
 Summary: Agent swarms for empirical research on Slurm clusters
 Requires-Python: >=3.11
 Requires-Dist: httpx>=0.27

{flyloft-0.2.2 → flyloft-0.3.0}/docs/ARCHITECTURE.md

@@ -1,6 +1,6 @@
 # flyloft Architecture
 
-Status: approved 2026-09-21. Stage 1 complete on 2026-09-23 (see [design/stage-1.md](design/stage-1.md)); stage 2 complete on 2026-09-23 (tiers one, two, and three passed; see [design/stage-2.md](design/stage-2.md)); stage 3a complete (see [design/stage-3a.md](design/stage-3a.md)). Stage 3b complete (see [design/stage-3b.md](design/stage-3b.md)). Stage 4a complete on 2026-09-25 (tiers one, two, and three passed; see [design/stage-4a.md](design/stage-4a.md)). Stage 4b complete on 2026-09-25 (tiers one, two, and three passed; see [design/stage-4b.md](design/stage-4b.md)). Stage 5a complete on 2026-09-27 (tiers one, two, and three passed; see [design/stage-5a.md](design/stage-5a.md)). Stage 5b complete on 2026-09-27 (tiers one and three passed; see [design/stage-5b.md](design/stage-5b.md)). Stage 5c complete on 2026-09-28 (tiers one and three passed; see [design/stage-5c.md](design/stage-5c.md)). Stage 5d complete on 2026-09-28 (tiers one and three passed; see [design/stage-5d.md](design/stage-5d.md)). Stage 6a complete on 2026-09-29 (tiers one and three passed; see [design/stage-6a.md](design/stage-6a.md)). Stage 6b complete on 2026-09-30 (tiers one and three passed; see [design/stage-6b.md](design/stage-6b.md)). Stage 7 complete on 2026-09-30 (tiers one, two, and three passed; see [design/stage-7.md](design/stage-7.md)). Every staged part is built. The vision's revision of 2026-10-13 widens flyloft to two modes, investigate and optimize, with a lab-wide toolshed; what comes next is in §6, *What comes next*. The Projects rework's board flow and template repositories are built and closed on 2026-10-19 (tiers one, two, and three passed; see [design/projects.md](design/projects.md)); the investigate mode is next, in four parts, the first built (see [design/investigate.md](design/investigate.md)); then the agent interface, where the researcher's own coding agent starts work and the board is where they come back to (see [design/agent-interface.md](design/agent-interface.md)). A critic-judged campaign's 0-GPU runs go to its cluster's CPU partition, sized within a CPU-run limit (see [design/cpu-partition.md](design/cpu-partition.md)).
+Status: approved 2026-09-21. Stage 1 complete on 2026-09-23 (see [design/stage-1.md](design/stage-1.md)); stage 2 complete on 2026-09-23 (tiers one, two, and three passed; see [design/stage-2.md](design/stage-2.md)); stage 3a complete (see [design/stage-3a.md](design/stage-3a.md)). Stage 3b complete (see [design/stage-3b.md](design/stage-3b.md)). Stage 4a complete on 2026-09-25 (tiers one, two, and three passed; see [design/stage-4a.md](design/stage-4a.md)). Stage 4b complete on 2026-09-25 (tiers one, two, and three passed; see [design/stage-4b.md](design/stage-4b.md)). Stage 5a complete on 2026-09-27 (tiers one, two, and three passed; see [design/stage-5a.md](design/stage-5a.md)). Stage 5b complete on 2026-09-27 (tiers one and three passed; see [design/stage-5b.md](design/stage-5b.md)). Stage 5c complete on 2026-09-28 (tiers one and three passed; see [design/stage-5c.md](design/stage-5c.md)). Stage 5d complete on 2026-09-28 (tiers one and three passed; see [design/stage-5d.md](design/stage-5d.md)). Stage 6a complete on 2026-09-29 (tiers one and three passed; see [design/stage-6a.md](design/stage-6a.md)). Stage 6b complete on 2026-09-30 (tiers one and three passed; see [design/stage-6b.md](design/stage-6b.md)). Stage 7 complete on 2026-09-30 (tiers one, two, and three passed; see [design/stage-7.md](design/stage-7.md)). Every staged part is built. The vision's revision of 2026-10-13 widens flyloft to two modes, investigate and optimize, with a lab-wide toolshed; what comes next is in §6, *What comes next*. The Projects rework's board flow and template repositories are built and closed on 2026-10-19 (tiers one, two, and three passed; see [design/projects.md](design/projects.md)); the investigate mode followed, in four parts, the first built (see [design/investigate.md](design/investigate.md)), the second parked on 2026-10-22 (§8: a bar fixed at launch), so the toolshed is its next part; then the agent interface, where the researcher's own coding agent starts work and the board is where they come back to (see [design/agent-interface.md](design/agent-interface.md)). A critic-judged campaign's 0-GPU runs go to its cluster's CPU partition, sized within a CPU-run limit (see [design/cpu-partition.md](design/cpu-partition.md)). On a cluster that names an object storage `work_dir_archive`, each ended run's directory and each ended campaign's folder move to the bucket (see [design/run-storage.md](design/run-storage.md)). The toolshed is built, and its tiers one and two passed on 2026-10-27 (see [design/toolshed.md](design/toolshed.md)); tier three follows.
 
 This document settles the shape of flyloft: the record that holds its state, the lifecycle and authority rules, the agent turn, the subsystem boundaries, and the stages by which the system grows. It fixes what is expensive to change later and leaves everything else to the design document of the stage that needs it. [VISION.md](VISION.md) says what flyloft is for; [AGENTS.md](../AGENTS.md) says how to work on it. This document is the bridge between them.
 
@@ -75,9 +75,9 @@ An assessment must link to the run whose result it judges. It exists so an agent
 
 Stage two changes the last step: a finding entry leaves the hypothesis in testing, and a critique that accepts the finding moves it to supported. Stage 4a inserts screened and admitted between proposed and testing, and merged beside them for a duplicate; replication before review is a rule at the finding, not a state. States are text and transitions are code, so this growth is not shape.
 
-**Run lifecycle.** preparing, queued, running, finished, failed, canceled, unknown. The application state is derived from observed scheduler state. A job missing from the queue is unknown until confirmed, never assumed finished. A finished job without its expected result files yields an incomplete result, not a conclusion.
+**Run lifecycle.** preparing, queued, running, finished, failed, canceled, interrupted, unknown. The application state is derived from observed scheduler state. A job missing from the queue is unknown until confirmed, never assumed finished. A finished job without its expected result files yields an incomplete result, not a conclusion. A preempted job is requeued by the cluster and waited on; it is interrupted when it is never requeued or its attempts have used twice its time limit. Interrupted is infrastructure, not a verdict, and is charged for the time it used ([design/preemption.md](design/preemption.md)).
 
-**Campaign lifecycle.** active, stopped, done. No campaign waits on a human (stage 5d): a failed turn is retried and, after three in a row, the agent is benched; a stall wakes the decision-maker and then finishes the campaign; a failed baseline stops it with its reason. Stopped means a human stopped new work, or the baseline failed; outstanding runs finish and are collected.
+**Campaign lifecycle.** active, stopped, done. No campaign waits on a human (stage 5d): a failed turn is retried and, after three in a row, the agent is benched; a stall wakes the decision-maker and then finishes the campaign; a baseline that does not finish stops it with its reason, an interrupted one naming the preemption. Stopped means a human stopped new work, or the baseline failed; outstanding runs finish and are collected.
 
 **Agent state.** idle, active, benched. The active state is the claim that prevents a second concurrent turn for the same agent. It lives on the agent, not the campaign.
 
@@ -169,7 +169,7 @@ Two processes write to the record from the first turn, the service and the tool
 5. UUID identities.
 6. Agents act only through the tool surface, with tool lists per role.
 7. A substrate interface, justified by two implementations.
-8. A worktree per agent.
+8. A worktree per agent, and in a campaign with a toolshed, a worktree of the shed beside it.
 9. PostgreSQL with numbered SQL migrations.
 10. What the researcher fixes at start is immutable: the question and, for an optimization, the baseline, metric, direction, and target, as campaign columns.
 11. Observed external state lives on entity rows with a last-observed timestamp.
@@ -197,7 +197,7 @@ Two processes write to the record from the first turn, the service and the tool
 
 **The Compute page.** After campaign-level compute, its own change ([design/compute.md](design/compute.md)): a sidebar page where each cluster says what each GPU type its file lists has (free GPUs, flyloft's running and queued jobs, nodes, CPUs, memory, longest run, and most free on a node), cached for 30 seconds and asked per cluster over the runs module's SSH client; the launch page's GPU type select shows each type's free count.
 
-**What comes next.** The vision's revision of 2026-10-13 widens flyloft from optimization to two modes, investigate and optimize, and adds a toolshed shared across the lab. Stages 1 through 7 built the optimize mode. What follows is staged like everything before it, each part with its own scenario and a design document written when it begins: the Projects rework, which makes bringing a project one flow on the board and turns templates into GitHub template repositories registered with flyloft (see [design/projects.md](design/projects.md)); then the investigate mode, in four parts: a solo, exploratory investigation whose project may be only a brief ([design/investigate.md](design/investigate.md)); a test fixed in advance, making findings confirmed; the toolshed; and investigation at swarm size. Then the agent interface ([design/agent-interface.md](design/agent-interface.md)): the researcher's coding agent is flyloft's front door (sign-in from a laptop, `submit` of a brief that flyloft turns into a repository and a campaign, a `flyloft` client with every initiating action, a skill), the host's commands move under `flyloft host` and admin actions under `flyloft admin`, and the board becomes where people monitor and steer; its second part adds tasks, a third label on the investigation engine ([design/tasks.md](design/tasks.md)); both are built and closed (2026-10-22). Their mechanics are decided in their design documents, not here.
+**What comes next.** The vision's revision of 2026-10-13 widens flyloft from optimization to two modes, investigate and optimize, and adds a toolshed shared across the lab. Stages 1 through 7 built the optimize mode. What follows is staged like everything before it, each part with its own scenario and a design document written when it begins: the Projects rework, which makes bringing a project one flow on the board and turns templates into GitHub template repositories registered with flyloft (see [design/projects.md](design/projects.md)); then the investigate mode, in four parts: a solo, exploratory investigation whose project may be only a brief ([design/investigate.md](design/investigate.md)); a test fixed in advance, making findings confirmed; the toolshed; and investigation at swarm size. The second part was parked on 2026-10-22 (§8): the researcher's own agent is the front door, so what follows a campaign is decided there, from the three campaign types and the client's commands, and a bar the researcher already knows goes into the brief, which the critic judges against today. flyloft builds primitives any workflow can be assembled from, not the workflows; the toolshed is the next primitive, and it is built ([design/toolshed.md](design/toolshed.md)): a lab-owned, public repository of tools, each a self-contained image with a prose skill; a campaign's run may name a tool and runs inside its image, and a campaign that needs a tool the shed lacks builds it through the shed's pull request, its own test run of the candidate image, and the critic's review, with no human in the loop. Then the agent interface ([design/agent-interface.md](design/agent-interface.md)): the researcher's coding agent is flyloft's front door (sign-in from a laptop, `submit` of a brief that flyloft turns into a repository and a campaign, a `flyloft` client with every initiating action, a skill), the host's commands move under `flyloft host` and admin actions under `flyloft admin`, and the board becomes where people monitor and steer; its second part adds tasks, a third label on the investigation engine ([design/tasks.md](design/tasks.md)); both are built and closed (2026-10-22). Their mechanics are decided in their design documents, not here.
 
 Recovery is built when a stage loses work, most likely stage three, when a service restart first interrupts many turns at once. It is not scheduled.
 
@@ -275,7 +275,7 @@ Ideas that have arrived before their stage. Adding a line here is how an impulse
 | Replication policy | Done at stage 4a: a campaign's `replication` is the seeds a finding needs at one commit, enforced at `post_finding`; the job script exports `FLYLOFT_SEED` |
 | Replication beyond a seed count (a second dataset, a held-out split), and the coordinator ordering it | When a project's evaluation policy asks for more than seeds |
 | Replicating integration runs | With [design/replication.md](design/replication.md): a merged tree is measured at `replication` seeds |
-| A tool-builder role | With the toolshed (the vision's revision of 2026-10-13), where a tool-builder adds tools for every project |
+| A tool-builder role | Done: the critic-judged implementer builds tools for the shed (see [design/toolshed.md](design/toolshed.md)) |
 | The swarm map's island row scrolling horizontally, its rule and stubs with it | When a campaign opens more islands than fit across the map (four at the board's width) |
 | A run slot reserved for consolidation (integration and root-island runs), so they never wait for a running test to end | When consolidation is seen waiting a whole run's length for a slot on a project whose runs take hours |
 | Backpressure on exploration: no new admissions while a campaign's queue of waiting runs is deep | When queued test runs are seen going stale (their island has moved on) before a slot reaches them |
@@ -305,6 +305,7 @@ Ideas that have arrived before their stage. Adding a line here is how an impulse
 | A per-project smoke time limit (a validation's is 30 minutes) | When a project's smoke run needs longer |
 | Cancelling a validation | When one is seen holding a GPU it should not |
 | A launch refused without a passed validation (the verdict is shown: see [design/projects.md](design/projects.md)) | When launches on unvalidated projects are seen failing |
+| A run at a higher tier than the lab's lowest, for work that cannot be split under the preemption cap | When a campaign hits the cap on work that cannot be shortened or made to resume |
 | A run an outage kills mid-run (Slurm's NODE_FAIL) is a failed test run: it is charged, and its implementer may read a lost node as a refuted change; Slurm's `--requeue`, or flyloft resubmitting a NODE_FAIL run uncharged, are the likely fixes. The power outage under ESM-2 8427bd25 killed none: its jobs had finished, and the job script's `finished` marker recovered them | When an outage is seen killing runs |
 | An implementer whose request ran nothing because another line was measuring its tree is not woken if those runs fail: its hypothesis stays testing, unmeasured, with nothing in flight. The integrator in the same place re-requests the missing seeds at its verdict turn | When it is seen |
 | Two requests for one tree that arrive together both run: measure-once checks for a run in flight outside any lock, so concurrent requests each see none (budget billing's tier two: three seats requested one tree within 2 ms and all three ran, at three times the cost) | When the duplicate runs are seen costing a real campaign's budget |
@@ -337,7 +338,7 @@ Ideas that have arrived before their stage. Adding a line here is how an impulse
 | Artifact retention and garbage collection; W&B synchronization | When storage fills or a project asks |
 | Cost estimate on proposals | Done at stage 4a: the proposal's `cost` line, read by the coordinator |
 | Harness environment scrubbed of git and cluster credentials | Stage 3a, by allowlist |
-| Observation of pull request state, including merged or squash-merged promotion pull requests (stage 6b) | When something reads it |
+| Observation of pull request state, including merged or squash-merged promotion pull requests (stage 6b) | Done for tool builds: the checks on a build's pushed and merge commits (see [design/toolshed.md](design/toolshed.md)); a campaign's own pull requests when something reads them |
 | A chart of cost and the metric over time on the board | When stage 6b's cost-per-promotion table stops being enough to read a long campaign |
 | An `.flyloft/` export at each promotion, not only at the stop | When a researcher needs turn transcripts before an indefinite campaign stops |
 | Timeout for runs and validations in unknown | When one is encountered |
@@ -377,10 +378,22 @@ Ideas that have arrived before their stage. Adding a line here is how an impulse
 | Searching or filtering the template list in New project | When a lab has more templates than fit on a screen |
 | Memory reading investigations: an earlier investigation's account and findings in a later campaign's assignment (today memory skips investigations both ways) | When an optimize campaign follows an investigation on the same project |
 | A size cap on an investigation run's collected `out/` | When a run is seen copying something large back to the host |
+| Raising a running campaign's GPU-hour ceiling (the client, the host CLI, or the board; today it is set once at launch) | Done once by hand on the live database, 2026-10-23, for a task whose work outgrew its first estimate |
 | Computed completion for tasks: a checklist the ask is parsed into | When critics are seen accepting incomplete outputs |
 | Two campaigns writing a project's work directory at once | When two are seen colliding there |
 | An "Empty project" choice on New project that writes the brief on the board and creates the repository | Superseded by `flyloft submit`: see [design/agent-interface.md](design/agent-interface.md) |
-| The toolshed (part 3 of investigate) and a data equivalent: where the lab's datasets live, how they are versioned, what has been fetched and prepared | Near-term, after the agent interface; each its own design |
+| The toolshed (part 3 of investigate) and a data equivalent: where the lab's datasets live, how they are versioned, what has been fetched and prepared | The toolshed done (see [design/toolshed.md](design/toolshed.md)); the data equivalent its own design |
+| Two campaigns building the same tool at once | When a second campaign is seen submitting a name another's build holds |
+| Abandoning a failed or blocked tool build to start another in the same campaign | When an implementer is seen stuck on a tool it should give up on |
+| A tool whose test needs more than one GPU or 30 minutes | When one is built |
+| A private tool (a license that forbids redistribution): a squashfs on the cluster's shared storage | When one arrives |
+| A `toolshed` command for laptops that picks podman or docker and mounts the working directory | When the skill's `podman run` line proves insufficient |
+| A tool build whose branch falls behind the shed's `main` under its strict protection: the merge is refused and retried each tick with a log line | When seen: update the branch through GitHub's API, then the checks again |
+| Declared parameters per tool command | When agents are seen misusing commands from prose |
+| A tool's image per GPU architecture (Arm) | When an Arm partition is wanted |
+| `flyloft tools` and the catalogue in the researcher's skill | When a researcher's agent needs the shed's contents while writing a brief |
+| A tool run on a cluster without pyxis | When a cluster without it is configured |
+| A bar fixed at launch (what is measured, on what data, what counts as support) as a field of an investigation, and findings that answer it marked confirmed in place of exploratory; today the bar goes into the brief and the critic judges claims against it, without the label | When a researcher comes to a question knowing what settles it and wants the label |
 | A Claude Code plugin and marketplace for the flyloft skill (the skill's directory plus a manifest) | When a lab member wants it from a marketplace, or a hook or second component is needed |
 | The flyloft skill for Codex and OpenCode | Done 2026-10-22: `flyloft install` serves every harness found |
 | `flyloft logout`, several hosts per laptop, refreshing a session before it expires | When one is missed |
@@ -392,6 +405,13 @@ Ideas that have arrived before their stage. Adding a line here is how an impulse
 | A CPU budget ([design/cpu-partition.md](design/cpu-partition.md) §6) | When CPU work on the shared node needs a bound beyond run slots and the CPU-run limit |
 | Choosing a GPU type per run within a campaign; budgeting must then weigh the types ([design/cpu-partition.md](design/cpu-partition.md) §6) | When a campaign needs two GPU types |
 | The request and the render of a CPU run disagreeing when `cpu_partition` changes while a run is preparing; the fix is to key `_is_cpu_run` on `run.cpus is not None` | When it is observed |
+| Watching archive jobs: a failed one leaves its directory and log on the cluster and tells no one ([design/run-storage.md](design/run-storage.md) §6) | When a failed archive goes unnoticed |
+| An object storage key scoped to `flyloft/`, under a CoreWeave principal of flyloft's own, in place of Sarah's key ([design/run-storage.md](design/run-storage.md) §6) | When the lab wants the exposure closed |
+| A cap on the disk that live runs and campaigns hold on an archiving cluster ([design/run-storage.md](design/run-storage.md) §6) | When a campaign fills `/mnt/data` |
+| Archiving validations and the shared project work directory; object storage on-prem ([design/run-storage.md](design/run-storage.md) §6) | When either fills its volume |
+| A lab's own job-script settings: first extra `#SBATCH` directives per cluster and GPU type (`--qos`, `--constraint`, `--reservation`), then a host-side prologue before the srun step (module loads), then a container runtime other than pyxis; a full template override only if those fall short, checked by doctor against what flyloft relies on (the run directory's output with append, the single srun step that ignores TERM, the `finished` marker, the `FLYLOFT_*` variables) | When a lab or a cluster needs a directive or runtime flyloft does not write |
+| The archive's transfer per cluster (endpoint, tool, key names) in place of CoreWeave's s5cmd fork, `http://cwlota.com` and `COREWEAVE_S3_*` ([design/run-storage.md](design/run-storage.md) §2) | When a cluster outside CoreWeave needs a `work_dir_archive` |
+| Archiving a run interrupted at the preemption cap only once squeue no longer lists its job: today it is archived in the tick that cancels it, so a file written during the kill grace can fail the check (the directory then stays, marked archived) ([design/run-storage.md](design/run-storage.md)) | When a cap-interrupted run's archive is seen to fail |
 
 ## References
 

{flyloft-0.2.2 → flyloft-0.3.0}/docs/NEW_PROJECT.md

@@ -23,22 +23,26 @@ campaign's metric. A project is one GitHub repository plus one registration on t
 | `environment_setup` | Shell lines run before the job enters the repository |
 
 Every run (a campaign's baseline, each test, a validation) is one Slurm job. Its script, as flyloft writes it (abridged:
-`--partition`, `--account`, `--cpus-per-task`, `--mem`, and `--container-mounts` appear when set):
+`--partition`, `--account`, `--cpus-per-task`, `--mem`, and `--container-mounts` appear when set, and the lines that
+pass a preemption's SIGTERM to the run command are left out):
 
 ```bash
 #!/bin/bash
 #SBATCH --job-name=flyloft
 #SBATCH --output=<run dir>/slurm-%j.out
+#SBATCH --open-mode=append              # a restarted run adds to the log
 #SBATCH --time=<the campaign's time limit; 00:30:00 for a validation>
 #SBATCH --gres=gpu:<count>              # with the GPU type's gres name, as the cluster's file says
-#SBATCH --container-image=<image>       # when the cluster runs jobs in a container
+srun --ntasks=1 --container-image=<image> bash -c '   # one step, in the cluster's container
 set -euo pipefail
+export FLYLOFT_PROJECT_WORK="<the project's directory on the cluster>"
 <environment_setup>
 export FLYLOFT_SEED=<seed>              # each replicate's seed; 0 for a validation
 export FLYLOFT_SMOKE=1                  # a validation only
 cd "<run dir>/code"
 <run_command>
 touch "<run dir>/finished"
+'
 ```
 
 What follows from it:
@@ -53,6 +57,10 @@ What follows from it:
 - **`environment_setup` runs before the `cd`.** Its paths are absolute, or values the command resolves after the
   `cd`, such as `export DATA_DIR=data`.
 - **One job is the whole run.** Training, evaluation, and writing results all happen inside the time limit.
+- **A run may restart.** Flyloft runs at the cluster's lowest tier, where a job running longer than 30 minutes can
+  be preempted and restarted from the top. The run command resumes from its own checkpoints, saved more often than
+  every 30 minutes, or starts over cleanly, writing outputs to a scratch path and moving them into place only when
+  complete. A preempted run gets SIGTERM five minutes before it is stopped, time enough to save a checkpoint.
 
 ## 2. The contract
 
@@ -186,6 +194,7 @@ Reading a validation's report:
 | passed | The job finished and wrote a flat object of numbers, which the report lists | The project runs: launch |
 | failed | The job exited with an error, ran out of time, or the cluster refused it; the report shows the log's last lines | A file nobody committed, a dependency the environment lacks, a wrong path, a crash in the evaluation |
 | incomplete | The job finished, but `results_path` is missing or is not a flat object of numbers | Results written somewhere else, a nested value, a metric written as a string |
+| interrupted | The cluster preempted the job and never restarted it; this says nothing about the project | Validate again |
 
 A failed or incomplete validation means fix the repository, push, and validate again; the report is written to be
 pasted back to you. A passed validation shows that the code runs at all, not that a campaign will succeed. A smoke

{flyloft-0.2.2 → flyloft-0.3.0}/docs/design/investigate.md

@@ -7,7 +7,7 @@ An investigation is a campaign whose researcher fixes a question and nothing els
 The investigate mode is staged in four parts, each with its own design when it begins:
 
 1. **Investigate, solo and exploratory.** This document.
-2. **A test fixed in advance.** The researcher fixes what will be measured, on what data, and what would count as support; findings answering it are confirmed, and critics judge claims against it.
+2. **A test fixed in advance.** The researcher fixes what will be measured, on what data, and what would count as support; findings answering it are confirmed, and critics judge claims against it. Parked on 2026-10-22: a bar the researcher knows goes into the brief, and the critic judges against the brief today; the *confirmed* label waits on the deferred list ([ARCHITECTURE.md](../ARCHITECTURE.md) §8).
 3. **The toolshed.** Tools shared across the lab, a tool-builder role, review before any agent calls a tool, agents looking there before they build.
 4. **Investigate at swarm size.** Generators, islands, and posture in an investigation, and per-hypothesis branches with an integrator, when a real campaign needs them.
 
@@ -164,7 +164,7 @@ tests/e2e/test_live.py                          tier two
 
 ## 9. Out of scope
 
-- A test fixed in advance and confirmed findings (part 2).
+- A test fixed in advance and confirmed findings (part 2, parked; deferred).
 - The toolshed and the tool-builder (part 3).
 - Generators, islands, posture, several implementers, per-hypothesis branches, and an integrator in an investigation (part 4).
 - Memory reading investigations (deferred).

flyloft-0.3.0/docs/design/preemption.md

@@ -0,0 +1,218 @@
+# Preemption and Interrupted Runs: Design
+
+Status: presented 2026-10-20; revised and approved 2026-10-23 with the restart contract, the preemption cap, attempt-aware timing, and the appended log; built 2026-10-23, tiers one and two passed. Charging was decided in conversation on 2026-10-20: an interrupted run is charged its used GPU-time, as any ended run. On 2026-10-23 that was carried across attempts: the time-limit cap applies per attempt (§6).
+
+Flyloft runs at each cluster's lowest scheduling tier, where every other tier preempts it: bcr's `low` QOS (slinky-config #177), on its GPU partitions and its `cpu` partition alike, and CoreWeave dev's `hpc-low`. Both clusters run `PreemptMode=REQUEUE`. On bcr, read live on 2026-10-23: `mid`, `service`, and `high` preempt `low`, and `low` is exempt for its first 30 minutes. `hpc-low` documents REQUEUE with a 300-second grace. Flyloft's job script does not opt out, so a preempted job restarts under the same job id and runs its script from the top.
+
+Three things go wrong today:
+
+- `runs/observe.py` maps `PREEMPTED` to `failed`, so a preemption is recorded as a failed experiment: negative evidence that never existed.
+- Nothing tells a project that its run may restart. A run that cannot pick up where it stopped begins again from zero. A long run on a busy tier can begin again forever, the livelock the lab hit on `hpc-low` on 2026-07-29: six training runs died between 29 and 76 minutes in, five of them before their first checkpoint.
+- The record measures a requeued run from its first start. The over-limit warning and an in-flight run's charge both count the time the job then waited in the queue.
+
+This change does four things:
+
+- It makes a requeued run something waited on, and a lost one a run with no verdict.
+- It states what a run owes a restart.
+- It bounds how much time a restarting run may use.
+- It times and charges each attempt by what the cluster reports.
+
+## 1. Scenario
+
+A campaign on a low tier submits a run whose job is preempted mid-flight.
+
+- **Requeued and finishes.** Slurm requeues the job, and the run reads queued, then running, again. The run command resumes from its own checkpoint, or starts over cleanly. It finishes, and the campaign records its metrics as usual.
+- **Preempted again and again.** Once the run's attempts together have used twice its time limit, flyloft cancels the job and the run ends `interrupted`, naming the time used and the preemption count. The agent that requested it wakes, and its assignment names the interruption as infrastructure. It re-requests the tree, splits the work into shorter steps, or makes the run resume from a checkpoint. The measure-once rule does not block a re-request, since nothing was measured.
+- **The requeue never reappears.** The run ends `interrupted` the same way, without the count.
+- **Charging.** Every attempt is charged the time the cluster reports it used, as any ended run (§6).
+- **Visibility.** The board's run pages and `flyloft campaign show` show `interrupted`, the raw scheduler state, and the number of preemptions. The log tail keeps every attempt.
+
+**Tier one** scripts all three outcomes in the fake cluster.
+
+**Tier two** is a live drill on bcr. A small `low`-QOS job, whose command resumes from a step counter it writes, is preempted by a `mid`-QOS job and watched end to end. The drill also records four behaviors code cannot settle:
+
+- what squeue shows inside the kill window;
+- what `sacct --duplicates` writes for a requeued job's attempts;
+- whether Slurm truncates the job's output file on restart;
+- whether the command resumed from its counter.
+
+**Tier three:** the researcher sees the interrupted run on the board.
+
+## 2. The map
+
+`PREEMPTED` maps to `interrupted` instead of `failed`. Nothing else in the map changes. A job the queue still lists wins over accounting, as today, so a requeued job reads running (COMPLETING), then queued, then running again, and stays active throughout. Accounting's last word is PREEMPTED only when the queue no longer lists the job and no requeue record follows: the job is gone for good, and the run ends `interrupted` at once.
+
+The drill (§14) showed bcr keeping a requeued job in the queue from the kill to its restart, and CoreWeave's accounting records the same requeue shape. A first design waited ten observations on a `preempted` state before giving up, in case a cluster dropped the job between the kill and the requeue. No cluster has been seen doing that, so the state and its wait were removed (decided 2026-10-23).
+
+TIMEOUT, FAILED, and OUT_OF_MEMORY stay `failed`. A run past its own limit, or dying on its own, is evidence about the run, and `settled()` keeps rescuing a job the controller lost after the command exited 0. NODE_FAIL and BOOT_FAIL stay on the deferred list; their entries are the same one-line change when their trigger fires.
+
+## 3. What a run owes a restart
+
+A requeued job runs `job.sh` again from the top, in the same run directory, with the same `$FLYLOFT_OUT` and `$FLYLOFT_PROJECT_WORK`. Slurm sets `SLURM_RESTART_COUNT` on a restart. Every run command must do one of two things:
+
+- **Resume.** It checkpoints into a directory that survives the restart and, on start, continues from the latest complete checkpoint. A run's checkpoints go under the campaign's own directory in `$FLYLOFT_PROJECT_WORK` (`campaigns/<id8>/`), in a subdirectory named by `SLURM_JOB_ID`, which a requeue keeps. A run longer than the tier's exempt window checkpoints more often than that window, 30 minutes on both clusters, or it can lose every attempt to the next preemption.
+- **Start over cleanly.** It writes to a scratch path and moves its outputs into place only when complete, so a restart never finds a half-written file and never appends to one.
+
+**The warning.** A preemption on either cluster starts a five-minute grace (`low` and `hpc-low` both set `GraceTime` 300). At its start Slurm sends SIGTERM to the job's steps, not to its batch shell; at its end it sends SIGTERM to the batch shell, and SIGKILL after `KillWait`. So the job script runs the work as one `srun` step, which hears the first SIGTERM, and the batch shell ignores TERM (`trap '' TERM`) so it outlives the step. The container goes on the `srun` line, because pyxis removes a batch-level container when the batch shell exits, even with the work still running (§14). Slurm signals only the step's own process, the shell that wraps the run command, so that shell traps TERM and sends it to its whole process group (`kill -TERM 0`), which is the run and nothing outside the step. It then waits until every process of the run has exited, not only its direct child: the run holds a lock on an inherited file descriptor, and the shell waits for the lock (`flock`). Without that wait, a shell between the step's shell and the work dies on the TERM, the step ends, and pyxis removes the container under the work still saving. A run command that saves a checkpoint on SIGTERM has the five minutes. One that ignores it is killed at the grace's end.
+
+Training runs resume. Data preparation, evaluation, and other runs that cannot checkpoint start over. A run under 30 minutes is exempt from preemption and owes nothing. No run declares which kind it is: the rule is the same for both, and the run command is where it is kept.
+
+This is prose, not code:
+
+- `NEW_PROJECT.md` gains a contract line beside "One job is the whole run". An optimization's `run_command` is the project's, so the project keeps the rule.
+- The implementer's rules (`_cj_implementer_rules.md.j2`) gain one bullet, beside the two directories. An investigation's or task's run command is the implementer's, so the implementer keeps the rule.
+
+## 4. The preemption cap
+
+A run whose attempts together have used twice its time limit is ended (decided 2026-10-23, in place of a count of preemptions). When a preempted run's summed elapsed time (§6) reaches twice its time limit, the service cancels the job, records the run `interrupted` itself rather than reading the cancellation back as `canceled`, collects it, and posts it as any terminal run. The factor is a constant, not configuration. A run that has never been preempted is bounded by Slurm's own limit and never reaches the cap.
+
+The cap is on time rather than on a count of preemptions, because flyloft cannot see whether a run resumed. Time used tells the two kinds of run apart:
+
+- **A run that resumes** loses at most its checkpoint interval and its startup per preemption, so its total stays near the work it needs. A 72-hour training run preempted eight times, losing about 35 minutes each, uses about 77 hours and finishes.
+- **A run that starts over** repeats its work on every attempt, so its total climbs past its limit. A 10-hour clustering run preempted at 9 hours finishes at 19. One that keeps being preempted reaches 20 and is ended, and its agent splits the work.
+
+The preemptions are counted too, for the record: §6's sacct call lists every attempt, and those that ended PREEMPTED are counted. The run row stores the count, and the board and the interruption's diagnostics show it.
+
+A run is never moved to a higher QOS. Which tier flyloft may use is the lab's decision (slinky-config #177), and a higher tier preempts other people's jobs. The deferred list gains the row in §11.
+
+## 5. A requeue that never comes
+
+A preempted job the queue no longer lists ends its run `interrupted` on the observation that sees it (§2), its reason `preempted and not requeued`. Nothing is waited on and nothing is counted, so nothing persists across a service restart.
+
+## 6. Timing and charging
+
+**Each attempt is its own clock.** Slurm gives every attempt a fresh `--time`. The record follows:
+
+- `started_at` is the current attempt's start: a run is stamped again when it reads running after reading queued, or when accounting shows a new preemption, which catches a requeue that restarts within one tick.
+- The stored preemption count never falls: a tick on which accounting answers nothing keeps the count it had.
+- The over-limit warning measures the current attempt.
+- The board shows the preemption count beside the start, so the reset is visible.
+
+**Elapsed is summed across attempts.** By default sacct reports only a requeued job's latest attempt, so the observation's sacct call gains `--duplicates`. `parse_observation` then sums `ElapsedRaw` over a job's records, and keeps the last record's state. A run that finishes after a requeue charges every attempt.
+
+**An in-flight run is charged its observed elapsed time.** `charge()` uses the run's last observed `elapsed_seconds` while it is active, falling back to time since `started_at` when there is none. Time spent waiting in the queue after a requeue is never charged. The observation is at most one tick old.
+
+**Charged as today otherwise.** An interrupted run bills the GPU time the cluster reports it used. The consequence is accepted and named here: a preemption-heavy low-tier campaign spends budget on attempts that taught nothing. Both the ledger's node-hours and the campaign's charge show that time.
+
+**The cap is per attempt (decided 2026-10-23).** Each attempt may use up to the time limit, so `charge()` caps a run at the limit times its number of attempts. A run with a 10-hour limit that runs 8 hours, is preempted, and then runs 9 more is charged 17 hours, not 10. Every attempt is billed what it used, as the 2026-10-20 decision says. The preemption cap (§4) ends a run near twice its limit, so in practice no run is charged much more than that.
+
+## 7. The log
+
+The job script gains `#SBATCH --open-mode=append`, so a restart adds to `slurm-<job>.out` instead of replacing it. The log tail then shows the preemption and what each attempt did. The drill confirms the truncation it guards against.
+
+## 8. The record and the agents
+
+`interrupted` joins `canceled` as a terminal `RunStatus` that carries no verdict. The result body carries:
+
+- status `interrupted`, with metrics null;
+- the raw scheduler state and the preemption count, beside the log tail. `scheduler_state` already persists the raw word.
+
+The agent that requested the run wakes on the result entry, as on any result. The roles that judge results, implementer and critic, gain one sentence: an interrupted run is infrastructure, not evidence about the hypothesis. A run interrupted by the cap is too long for the tier; the move is to re-request it shorter or resuming (§3).
+
+The integrator's merged tree is measured by integration runs. When a seed's integration run is interrupted, the integrator requests the same merge commit again instead of reverting the finding, but only when every seed without the metric was interrupted, and each of those only once. A seed that failed or reported no metric is evidence about the merge, and a seed interrupted twice does not fit the tier; either way the integrator reverts, as today.
+
+A baseline has no requester. One that does not finish stops the campaign, as today, but an interrupted one stops it with its own reason (decided 2026-10-23): "the baseline run was interrupted: preempted on the cluster's lowest tier; nothing is wrong with the project; start the campaign again". It is not resubmitted (§13).
+
+The population and result renderings print the state beside the run, as they print `failed` today. No assessment rule changes: assessment is the agents' judgment, and nothing auto-refutes today.
+
+## 9. Validations
+
+`_validate` shares the map and `RUN_ACTIVE`, so today a preempted smoke run reads `failed`: a project blamed for the cluster's tier. A smoke run finishes well inside 30 minutes, inside the exempt window, so preemption reaches it only in the kill window of a job that started before it. It gets the same treatment:
+
+- A preempted smoke run that is requeued stays queued or running, as a run does.
+- One that is never requeued ends `interrupted`, a new validation verdict beside passed, failed and incomplete, its log tail naming the preemption. `incomplete` keeps its one meaning: the job finished without readable results.
+- Re-running is the existing path: `start_validation` from the CLI or the board.
+
+The preemption cap does not apply to validations: a 30-minute job inside its own exempt window cannot be preempted often enough to use an hour.
+
+## 10. Failure behavior
+
+| Condition | Behavior |
+|---|---|
+| A run's job is preempted and requeued | The run stays active and reads queued or running again. The command resumes or starts over (§3), and the run finishes or fails on its own merits |
+| A preempted run's attempts have used twice its time limit | Flyloft cancels it, and the run ends `interrupted` with the time used and the count. No metrics; the requester is woken and can re-request shorter or resuming |
+| A run that resumes is preempted many times | It keeps running: each preemption costs it little time, so it finishes under the cap |
+| A preempted job is never requeued | The run ends `interrupted` at once; no metrics; the requester is woken and can re-request |
+| An observation lands inside the kill window | The job reads COMPLETING, so the run stays running; no result is posted |
+| Accounting answers nothing for a tick | The preemption count and charge keep what they had |
+| An integration run is interrupted | The integrator measures the same merge again, once per seed |
+| A requeued run waits in the queue | Not charged, not over its limit; its next attempt is timed from its own start |
+| A run that starts over is restarted | It finds no half-written output, by the §3 rule |
+| The cluster stops answering | The observation pass fails and counts toward nothing; the over-limit and unanswered warnings govern, as today |
+| A run's job dies on its own (TIMEOUT, FAILED, OUT_OF_MEMORY) | `failed`, as today, with `settled()`'s rescue unchanged |
+| A campaign's baseline is interrupted | The campaign stops, its reason naming the preemption, not the project |
+| A validation's smoke run is preempted | Requeued, it runs on; never requeued, it ends `interrupted` with the reason in its log tail |
+| A researcher cancels | `canceled`, unchanged |
+
+## 11. The docs
+
+- **`ARCHITECTURE.md`, the run lifecycle paragraph.** The state list gains `interrupted`. The paragraph says a preempted job is requeued by the cluster and waited on, and is interrupted when it is never requeued or its attempts have used twice its time limit. Interrupted is infrastructure, not a verdict, and is charged for the time it used.
+- **`ARCHITECTURE.md`, the deferred list.** It gains one row: "A run at a higher tier than the lab's lowest, for work that cannot be split under the preemption cap", triggered "when a campaign hits the cap on work that cannot be shortened or made to resume".
+- **`NEW_PROJECT.md`, the contract.** It gains the §3 line: a run resumes from its own checkpoint or starts over cleanly.
+- **`NEW_PROJECT.md`, the validation verdict table.** It gains an `interrupted` row: the cluster preempted the job and never restarted it, which says nothing about the project; validate again.
+
+## 12. Code layout
+
+```
+src/flyloft/runs/observe.py                       the map entry: PREEMPTED → interrupted
+src/flyloft/runs/ssh.py                           sacct --duplicates; elapsed summed, preemptions counted
+src/flyloft/runs/staging.py                       --open-mode=append; the work in one srun step, the shell ignoring TERM (§3)
+src/flyloft/record/models.py                      RunStatus, RUN_TERMINAL, RUN_ACTIVE
+src/flyloft/record/actions.py                     charge(): observed elapsed in flight; the cap per attempt
+src/flyloft/migrations/                           the run's preemptions
+src/flyloft/service/tick.py                       the _observe branch, started_at per attempt, the cap, _validate
+src/flyloft/record/collect.py                     build_result_body's status literal and preemption count
+src/flyloft/turns/roles/_cj_implementer_rules.md.j2   the restart rule (§3)
+src/flyloft/turns/roles/implementer.md.j2, critic.md.j2   the interrupted-run sentence
+src/flyloft/turns/assembly.py, integrator.md.j2  measuring an interrupted merge again
+src/flyloft/web/                                  RunStatus and the preemption count where they render; types.ts regenerated
+docs/ARCHITECTURE.md, docs/NEW_PROJECT.md         §11
+tests/fakes/cluster.py                            scripted preemptions, requeues, and attempt records
+tests/                                            §1's scenarios, the cap, the parse sum, per-attempt timing, the integrator
+```
+
+## 13. Out of scope, and where it goes
+
+- **Flyloft-side resubmission of interrupted runs.** This is already the NODE_FAIL row's idea on the deferred list. Slurm requeues on both clusters, and the requester re-requests, so nothing builds it until preemptions are seen eating a real campaign.
+- **A higher tier for work that cannot be split.** This is the deferred row in §11.
+- **An earlier warning (`--signal=B:USR1@<seconds>`).** The `srun` step's SIGTERM at grace start gives five minutes (§3). A longer warning would need `--signal` in the job script and, on bcr, `PreemptParameters=send_user_signal`, which runs on dev alone (slinky-config #204). Flyloft adds nothing until a run is seen needing more than five minutes.
+- **A preemption-aware budget policy, charging interrupted runs nothing.** Decided against on 2026-10-20: node-hours and the campaign's charge both record the used time.
+
+## 14. Brought in line at close
+
+**The drill (tier two), bcr, 2026-10-23.** Two `low` jobs on `cpu` (128 CPUs each, the counter in `scripts/preemption_drill.sh`), one with `--open-mode=append` and one without, were preempted 37 minutes in by one `mid --exclusive` job. It was submitted at 16:14 UTC and preempted them at 16:19:30.
+
+- **The kill window.** squeue showed both jobs COMPLETING for about 10 seconds, then PENDING. The job never left the queue: a requeued job stays visible as running, then queued, then running. This is what removed the `preempted` state (§2).
+- **Accounting.** `sacct -D` wrote the first attempt as `PREEMPTED` with its own ElapsedRaw (2217). At the requeue it added a new record, `PENDING|0`, which became `RUNNING` with its own growing ElapsedRaw. Summing the records gives the job's time across attempts. PREEMPTED is the word counted.
+- **The restart.** The requeued jobs started again at 16:25:00, about five minutes after the kill, with `SLURM_RESTART_COUNT=1`. The counter resumed at step 174 of 174 saved.
+- **The log.** Without `--open-mode=append`, the restart truncated the log: only the second attempt's lines remained. With it, the log kept the first attempt, Slurm's `CANCELLED ... DUE TO PREEMPTION` line, and the second attempt.
+- **`MaxBatchRequeue = 10`** on bcr and on CoreWeave. slurm.conf documents it for requeues after a launch or prolog failure, not for preemption, so it does not bound a preempted run; the cap (§4) does. This has not been checked on a job preempted more than ten times.
+
+**CoreWeave.** `sacct -D` on coreweave-b200 job 25439_98 (preempted on `hpc-low` on 2026-10-12) shows the same shape: three `PREEMPTED` records with their own ElapsedRaw (38381, 2397, 2422), then `COMPLETED`. The parse counts both clusters' records alike.
+
+**Through flyloft, partlow, 2026-10-23.** This branch ran as a scratch host on partlow (a scratch database restored from a backup, its own config and data directory). An investigation's one CPU run on bcr `cpu` counted to 270 in ten-second steps, checkpointing a counter. A `mid --exclusive` job preempted it 37 minutes in.
+
+- **The record.** The run read running (COMPLETING), then queued, then running. It ended `finished` with 2 preemptions and 3449 seconds elapsed, the sum of its three attempts' records (2252, 996, 201). `started_at` restarted with each attempt. No result was posted until it finished.
+- **The work.** It resumed at step 198, and the log kept all three attempts.
+- **An extra requeue after a clean exit.** The second attempt's batch step completed with exit 0 at 19:29:19, yet Slurm recorded the attempt `PREEMPTED` and requeued it. Nothing preempted it: the controller log has no preemption decision near then, only `requeue JobId=3098 per user/system request` after the exit. This is a bug in Slurm 26.05.4. A requeue after a preemption's grace leaves the job's `GRACE_PREEMPT` flag set, so the next attempt's normal exit is taken as a preemption and requeued once more. Slurm 26.05.5 fixes it (upstream `c960622101`). It was reproduced on bcr's dev cluster on 26.05.4 and is gone there on 26.05.5. bcr prod still runs 26.05.4. CoreWeave's 25.05.3 has the same code path; that is read in source, not reproduced. The third attempt resumed at 270 and finished in seconds. A run that starts over would have repeated its work. The exempt window is per attempt: a requeue resets the job's start time.
+- **A preempted attempt outlived its container.** After Slurm's `CANCELLED ... DUE TO PREEMPTION` line, the first attempt's script ran on for seconds without the container's filesystem (`sleep`, `tee` and the checkpoint path were missing) and spun to its end without writing. The cause is how `low`'s grace signals a job. At grace start Slurm sends SIGTERM to the job's steps but not to the batch shell, and the job then ran the command in the batch shell with no steps, so the run command heard nothing. At grace end Slurm sends SIGTERM to the batch shell. `job.sh` did not trap it, so the shell died and pyxis removed the container while the run command, the shell's child, was still running. A dev probe whose shell trapped TERM kept its container until the SIGKILL 30 seconds later. Had the checkpoint path stayed writable, the orphan would have written a false last step. The job script now runs the work as an `srun` step in its own container, behind a batch shell that ignores TERM (§3). A run that resumes should still write a checkpoint only when it is complete (§3 says "latest complete checkpoint").
+
+**The live tier, partlow, 2026-10-23.**
+
+- **Passed:** a toy validation, and an investigation's CPU run and GPU run.
+- **Failed on the scratch setup, not on this branch:**
+  - The four campaign tests needed projects and scripted harnesses the scratch config lacked.
+  - The two tests that create a repository were refused by partlow's GitHub App.
+- **Untested live:** the four campaign tests were not rerun. The in-flight charge from accounting's elapsed time is therefore exercised live only by runs that are not budget-gated, because the fake cluster reports no elapsed time for a running job.
+
+**Built differently from the plan:**
+
+- **No `preempted` state** (§2, §5). The plan built it with a ten-observation wait; the drill and a review showed it guarded against a shape no cluster produces and carried three bugs of its own, so it was removed.
+- **The integrator** (§8) was not in the plan. A review found an interrupted integration run would have reverted an accepted finding.
+- **`record/views.py` is unchanged.** Stamping `started_at` per attempt makes the over-limit warning measure the current attempt.
+- **The validation verdict.** An interrupted validation first ended `incomplete`; it got its own `interrupted` verdict so `incomplete` keeps one meaning (decided 2026-10-23).
+- **The fake cluster** preempts as bcr does: COMPLETING in the queue while accounting records PREEMPTED, then PENDING. A job that is never requeued leaves the queue as PREEMPTED.
+- **The job runs as an `srun` step** (§3). The plan left the job script's shape alone; the drill's orphaned attempt showed the run command heard no warning and outlived its container. Tried on bcr `cpu` on 2026-10-26:
+  - **A real preemption (job 3182)** of a step that only wrapped the run command in `bash -c`. The step was signaled one second after the `mid --exclusive` job's submission, and the attempt was recorded `PREEMPTED` and requeued. But only the wrapping shell got the TERM. It died, the step ended, and the run command, which never heard the signal, ran on without its container. This is why the step's shell forwards TERM and waits.
+  - **Signals sent by hand** (`scancel -b --signal=TERM`, then `scancel --signal=TERM`), standing in for the grace's end and start without waiting out the 30-minute exemption. With the shipped job script (job 3189), the batch shell ignored its TERM, the step's TERM reached a run command two shells down, and the run spent five seconds saving with its container intact before the job ended. A first try that waited on the shell's child alone (job 3186) ended the step while the run was still saving.
+- **Known gap.** A template measurement's reference run time sums a preempted seed's attempts, so it includes time lost to preemption.

flyloft-0.3.0/docs/design/run-storage.md

@@ -0,0 +1,158 @@
+# Run Storage on Each Cluster: Design
+
+Status: approved 2026-10-26. Tiers one and two passed on 2026-10-26: tier two archived a run and a campaign folder on coreweave-rtx6000 to brineylab-us-east.
+
+A run's directory on the cluster holds its code checkout, its Slurm log, and whatever its command writes there: a few files for an analysis, hundreds of GB of checkpoints for an optimize campaign's training run. An investigation's or task's own products, its checkpoints included, go instead to the campaign's folder in the project work directory, `$FLYLOFT_PROJECT_WORK/campaigns/<id8>/`, where its later runs find them. flyloft copies back only the small part (`results.json`, `slurm.out`, an investigation's `out/`). Everything else stays where it was written, and nothing is ever deleted.
+
+Two things are wrong with where that is today:
+
+- **On CoreWeave it is the wrong volume.** Both clusters set `remote_work_dir = /mnt/home/flyloft-svc/flyloft`. `/mnt/home` is one 1T volume shared by every user, and on b200 it reached 80% on 2026-10-23 from one user's checkpoints. `/mnt/data` (4T) is bigger but not big enough either: b200 has under 900G free.
+- **On CoreWeave researchers cannot read it.** `/mnt/home/flyloft-svc` is `0750 flyloft-svc`. Getting a checkpoint out of a campaign would take an admin.
+
+The researcher's rule: **any lab member may read any campaign's outputs, and no admin hands data over.**
+
+This change keeps on-prem on davyjones. On CoreWeave, runs move to `/mnt/data`, each ended run's directory moves to the region's object storage bucket, and each ended campaign's folder follows.
+
+## 1. Scenario
+
+1. **On-prem (bcr), runs stay on davyjones.** `/mnt/data/users/flyloft` is `brineylab_shared/flyloft`, with no quota. Every lab member's JupyterHub mounts `brineylab_shared` read-only at `~/shared`, so runs and project work directories are readable under `~/shared/flyloft/` while they run and after. Only the layout changes (below).
+2. **On CoreWeave, a campaign runs on b200.** Its runs write under `/mnt/data/flyloft/<campaign>/<run>/`, and an investigation's checkpoints under `/mnt/data/flyloft/projects/<project>/campaigns/<id8>/`.
+3. **A run ends** (finished, failed, cancelled, or timed out). flyloft collects the small files exactly as today and posts the result. The result's artifact is `s3://brineylab-eu/flyloft/runs/<campaign>/<run>/`, not the disk path.
+4. **flyloft submits the run's archive job.** It is a 0-GPU job on the cluster that copies the run directory to that prefix with s5cmd, checks that every file arrived, and only then deletes the directory from `/mnt/data`.
+5. **The campaign ends**, either `done` (goal met, concluded, stalled, or budget spent) or `stopped` by a researcher. Once its last run has ended, flyloft submits the campaign's archive job. That job moves `campaigns/<id8>/` to `s3://brineylab-eu/flyloft/campaigns/<id8>/` the same way. The shared parts of the project work directory (`envs/`, `data/`, `weights/`) stay for later campaigns.
+6. **A lab member reads the checkpoints** from the bucket with their own CoreWeave key. The `brineylab-users` access policy already gives every lab member read access to both buckets. The run page shows the run's prefix, and the campaign page shows the campaign's.
+7. **A follow-up campaign** on the same project that starts from the pilot's checkpoints pulls them from the pilot's campaign prefix in a setup run.
+8. **On rtx6000** the same happens with `s3://brineylab-us-east/flyloft/...`.
+
+## 2. What changes
+
+**The cluster file.** `ClusterConfig` gains `work_dir_archive: str | None`, the bucket prefix (`s3://<bucket>/flyloft`) that ended runs and campaigns move to. A cluster without one keeps everything where it was written, as bcr does. The prefix must be in CoreWeave AI Object Storage: the archive job runs CoreWeave's s5cmd fork from the cluster's container against CoreWeave's in-cluster endpoint, `http://cwlota.com` (`ARCHIVE_ENDPOINT` in `runs/staging.py`), with the `COREWEAVE_S3_*` key. None of the three is configurable; a bucket elsewhere would fail at its first archive job. Config validation refuses a `work_dir_archive` on a cluster with no `partition`, because the archive job needs somewhere to run.
+
+**One layout on every cluster.** `remote_work_dir` is flyloft's own top-level directory, and everything flyloft writes on a cluster sits directly under it:
+
+```
+<remote_work_dir>/<campaign>/<run>/       a run
+<remote_work_dir>/projects/<project>/     the project work directory
+<remote_work_dir>/validations/<id>/       smoke runs
+<remote_work_dir>/archive-logs/           archive job logs, on an archiving cluster
+```
+
+The live files are in `brineylab/flyloft-config`, changed in a pull request there once this is deployed:
+
+```toml
+# clusters/bcr.toml
+remote_work_dir = "/mnt/data/users/flyloft"
+
+# clusters/coreweave-b200.toml
+remote_work_dir = "/mnt/data/flyloft"
+container_mounts = "/mnt/data:/mnt/data"
+work_dir_archive = "s3://brineylab-eu/flyloft"
+
+# clusters/coreweave-rtx6000.toml
+remote_work_dir = "/mnt/data/flyloft"
+container_mounts = "/mnt/data:/mnt/data"
+work_dir_archive = "s3://brineylab-us-east/flyloft"
+```
+
+A run's prefix is `<work_dir_archive>/runs/<campaign>/<run>/`, mirroring `remote_run_dir`. A campaign's prefix is `<work_dir_archive>/campaigns/<id8>/`, mirroring its folder in the project work directory.
+
+**The credential.** There is one object storage key. Sarah created it, and it is already in partlow's `secrets.env` as `COREWEAVE_S3_ACCESS_KEY_ID` and `COREWEAVE_S3_SECRET_ACCESS_KEY`. Both CoreWeave clusters use it, so the names are fixed rather than set per cluster. `flyloft doctor`'s secrets check requires both whenever any cluster has a `work_dir_archive`.
+
+The ssh command that submits an archive job reads the key from stdin with `read` and exports it as `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. It then runs `sbatch --export=ALL`. So the key is:
+- never on a command line, where any user on the login pod could see it in the process list;
+- never in the rendered script;
+- never in a training job's environment, and never in `flyloft-svc`'s home directory.
+
+Slurm does keep each job's environment in its state directory, where `flyloft-svc` and Slurm admins can read it back. Every job runs as `flyloft-svc`, so a training job set on finding the key could find it there. The key carries Sarah's full access to both buckets. That exposure is known and accepted (§6).
+
+**The archive job.** One script serves both a run and a campaign: given a directory and a prefix, it does the following.
+- **Resources:** 0 GPUs, 2 CPUs and 4G, a 12-hour limit, on the cluster's `partition` (`hpc-low`; neither CoreWeave cluster has a CPU-only partition). It runs in the cluster's container. `deeplearning.sqsh` installs CoreWeave's s5cmd fork, at `images/deeplearning/Dockerfile:131` in `devops/containers`. The job uses `--requeue`, so a preemption reruns it.
+- **Copy:** `s5cmd sync` copies the directory to the prefix over CoreWeave's internal endpoint, which is a constant in the script.
+- **Check:** the check lists every regular file on disk with `find` (hidden and empty files included) and compares paths and sizes against `s5cmd ls` of the prefix, independently of the copy, so a file the copy's walker skips cannot be deleted. Every file on disk must be in the bucket at the same size. The archive job's own log is left out of the check. Checking disk against bucket, not comparing totals, means a requeue that interrupts the delete cannot wedge the next attempt.
+- **Delete:** the directory is deleted only when the check passes.
+- **Log:** the job's log is written to `<remote_work_dir>/archive-logs/`, outside the directory it moves. If the job fails, the directory and its log stay on `/mnt/data`.
+
+A sync is idempotent, so a requeued or resubmitted archive job finishes what an earlier one started.
+
+**The service.** A migration adds `run.work_dir_archive` and `campaign.work_dir_archive`, each the prefix its files went to, set when its archive job is submitted. On each tick, on clusters with a `work_dir_archive`:
+- **A run:** each ended run that had a Slurm job (`job_id` set) and has no archive job gets one, after it is collected and its result posted. A run cancelled before staging, or one whose `sbatch` was refused, had no job and is not archived.
+- **A campaign:** each `done` or `stopped` campaign with no run still active and no archive job gets one for its `campaigns/<id8>/` folder. Only an investigation's or a task's campaign has its own folder; an optimize campaign's is not archived.
+- **Retries:** a failed submission, by ssh or by `sbatch`, is logged and retried on the next tick, as staging is today.
+- **Not done:** flyloft does not watch an archive job after submitting it (§6). Archive jobs do not count toward `max_outstanding_jobs` or run slots.
+
+The cluster client (`runs/client.py`) gains one method that writes the archive script and runs `sbatch` in a single ssh call, with the key on stdin. The fake cluster in `tests/fakes/cluster.py` implements it too.
+
+**The result.** On a cluster with a `work_dir_archive`, `build_result_body` names the run's prefix as its artifact instead of the disk path. On other clusters nothing changes.
+
+**The implementer's assignment.** On an archiving cluster, the shared `_cj_implementer_rules.md.j2` partial adds one sentence after the campaign's folder: when the campaign ends, the folder moves to its campaign prefix. It also says that a later campaign that needs those products pulls them from there.
+
+**What does not move.**
+- *The shared project work directory* (`envs/`, `data/`, `weights/`) stays on `/mnt/data`, because later campaigns rely on it.
+- *Validations* (`<remote_work_dir>/validations/<id>`) are 30-minute smoke runs and stay where they ran.
+
+**Before deploying.** These are admin steps, not code:
+- On each CoreWeave cluster, create `/mnt/data/flyloft` owned by `flyloft-svc`, group `sg-coreweave-slurm-users`, setgid and group-readable, so lab members can also read a live run on disk. `/mnt/data` is `0777`.
+- On bcr, with no flyloft run active, move the contents of `/mnt/data/users/flyloft/runs/` up one level as `flyloft`, then deploy `bcr.toml`'s new `remote_work_dir`. It is one ZFS dataset, so each move is a rename. flyloft computes a run's directory from the cluster file, so a run active across the change would be collected from the wrong path.
+- Check that the live database has no runs or campaigns on `coreweave-*`. Changing `remote_work_dir` would otherwise point archive jobs at paths that don't exist. No campaign has run on CoreWeave yet; if one has by then, mark its runs and campaigns as not to archive.
+
+## 3. The board
+
+- **A run's page** shows its bucket prefix as where its files are, on an archiving cluster.
+- **A campaign's page** shows its campaign prefix among the header's facts, once its archive job is submitted.
+- `npm run types` regenerates `web/src/api/types.ts`.
+
+**Where an agent finds the files.** The host API's cluster view (`ClusterView`, behind `flyloft options`) carries each cluster's `remote_work_dir` and `work_dir_archive`. `flyloft options` prints, under each cluster, where a run's files and a campaign's own folder are while they run and, on an archiving cluster, where they go after. The flyloft skill says to read that, with the campaign and run ids from `flyloft status <id> --json`, and that an `s3://` path is read with the researcher's own object storage key. The skill stays lab-neutral: it names no lab's paths.
+
+## 4. Tests
+
+**Tier one.** The fake cluster gains a `work_dir_archive`, and a stub `s5cmd` on its `PATH` maps `s3://` to a local directory, so the real archive script runs. The checks are what a researcher would check:
+- **An investigation on the archiving cluster** runs a setup run and a training run that writes fake checkpoints to its campaign folder and to its run directory, then concludes.
+  - Each run's result names its prefix.
+  - Each run's directory is under the stand-in bucket and gone from the work directory.
+  - After the campaign ends, `campaigns/<id8>/` is too, and the project's `envs/` is untouched.
+  - The board shows both prefixes.
+- **A stopped campaign** with a run in flight is archived only after that run ends.
+- **A cluster without a `work_dir_archive`** leaves runs and the campaign folder where they were written, as today.
+- **A failed check:** the stub drops one file, and the directory is not deleted.
+
+**Unit tests** cover the archive script and the submission:
+- the key appears nowhere in the rendered script or in the ssh command line;
+- deletion follows only a passing check.
+
+**Tier two** (live, scripted, no model tokens) runs on prod `hpc-low` on b200, which Sarah allowed for this test. Dev has had no Slurm nodes since 2026-10-23.
+- **Setup:** a reservation of a few hours for `flyloft-svc` is created at test time, on a node with a GPU free, after checking the shape of the jobs on it. `SBATCH_RESERVATION` goes in `flyloft-svc`'s `~/.bashrc` on the login pod for the window. Both are removed afterwards.
+- **The test:** a run writes a few hundred MB of fake checkpoints to its run directory and its campaign folder, and the campaign concludes.
+- **Checked:** `s5cmd ls` shows both prefixes holding every file at its size, and both directories are gone from `/mnt/data`.
+- **Cleanup:** the test deletes its own prefixes.
+- **Live config:** the test needs a live config naming `coreweave-b200`. `test_live.py` today takes the one cluster its config names. How it reaches a second cluster is the plan's call.
+- **As run, 2026-10-26:** on `coreweave-rtx6000` and `brineylab-us-east` instead, with no reservation: b200's `hpc-low` was full and rtx6000 had idle nodes. The run asks for no GPU, so either cluster proves the same path. It ran from partlow with partlow's database, `flyloft-serve` stopped for the run, and passed in 2 min 20 s. The test project was `flyloft-trial`: Sarah's GitHub token cannot read `flyloft-dev`.
+
+**Tier three** is the first real campaign on b200.
+
+## 5. Code layout
+
+```
+src/flyloft/migrations/0030_archive.sql             run.work_dir_archive, campaign.work_dir_archive
+src/flyloft/config/models.py, config/load.py        ClusterConfig.work_dir_archive; refused without a partition
+src/flyloft/runs/staging.py                         run and campaign prefixes; render_archive_script()
+src/flyloft/runs/client.py, runs/ssh.py             archive(): write the script, sbatch, the key on stdin
+src/flyloft/runs/collect.py                         the result's artifact is the prefix on an archiving cluster
+src/flyloft/service/tick.py                         archive ended runs, then ended campaigns
+src/flyloft/turns/roles/_cj_implementer_rules.md.j2 the campaign folder moves to its prefix at the end
+src/flyloft/host/doctor.py                          the key is present when a cluster archives
+src/flyloft/web/schemas.py, record/views.py         the run's and the campaign's prefixes
+web/src/...                                         the prefixes on the run and campaign pages
+tests/fakes/cluster.py                              archive(), and a stub s5cmd
+examples/live-config/clusters/                      a commented work_dir_archive line
+```
+
+## 6. Out of scope
+
+Each goes on the deferred list in `docs/ARCHITECTURE.md`.
+
+- **Watching archive jobs.** flyloft submits each one and moves on. If one fails, the directory stays on `/mnt/data` with its log, and nothing tells anyone. Added when a failed archive goes unnoticed.
+- **A key scoped to `flyloft/`.** The key is Sarah's, with full access to both buckets. A CoreWeave principal for flyloft, with a policy on `brineylab-*/flyloft/*` only, removes the exposure. Asked for when the lab wants it closed.
+- **A cap on live disk use.** While runs and campaigns are live, `/mnt/data` holds all of them. Added when a campaign fills it.
+- **Archiving validations and the shared project work directory.**
+- **Retention in the bucket.** Lab members can already delete under `flyloft/`, and nothing expires.
+- **Object storage on-prem.**