fwm 0.1.1__tar.gz

fwm-0.1.1/LICENSE

@@ -0,0 +1,33 @@
+/**********************************************************************************
+Copyright (c) 2026, North Carolina State University (NCSU).
+
+
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without modification,
+are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice, this
+list of conditions and the following disclaimer in the documentation and/or other
+materials provided with the distribution.
+
+3. The names “North Carolina State University”, “NCSU” and any trade‐name, personal
+name, trademark, trade device, service mark, symbol, image, icon, or any abbreviation,
+contraction or simulation thereof owned by North Carolina State University must not
+be used to endorse or promote products derived from this software without prior
+written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT
+SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
+TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
+ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
+DAMAGE.
+**************************************************************************************/

fwm-0.1.1/PKG-INFO

@@ -0,0 +1,353 @@
+Metadata-Version: 2.4
+Name: fwm
+Version: 0.1.1
+Summary: RISC-V FESVR Workload Manager: traces, checkpoints and SimPoint runs for a record/replay spike.
+Keywords: riscv,spike,fesvr,simulation,checkpoint,simpoint,anycore
+Author: Jiayang Li
+Author-email: Jiayang Li <jli95@ncsu.edu>
+License-Expression: LicenseRef-NCSU
+License-File: LICENSE
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Operating System :: MacOS
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: System :: Emulators
+Requires-Dist: colorama>=0.4.6
+Requires-Dist: click>=8.4.0
+Requires-Dist: peewee>=4.5.0
+Requires-Dist: pydantic>=2.9.0
+Requires-Python: >=3.12
+Project-URL: Homepage, https://github.com/s117/fesvr-workload-manager
+Project-URL: Repository, https://github.com/s117/fesvr-workload-manager.git
+Description-Content-Type: text/markdown
+
+# fwm: the RISC-V FESVR Workload Manager
+
+fwm manages RISC-V simulation workloads for the custom spike in [riscv-isa-sim](https://github.com/s117/riscv-isa-sim/tree/anycore-gcc_16.1.0-llvm_22.1.8), and replaces riscv-simenv.
+
+That spike's front-end server (FESVR) can **record** the device traffic of a run: every proxied system call, with the memory it read and wrote. It can then **replay** that recording without the original environment, and create and load **checkpoints** on top of a replay (see [docs/fesvr-checkpoint-and-traffic-replay.md](https://github.com/s117/riscv-isa-sim/blob/anycore-gcc_16.1.0-llvm_22.1.8/docs/fesvr-checkpoint-and-traffic-replay.md) in riscv-isa-sim). fwm keeps the pieces of that workflow in a repository and generates the Makefiles that run them.
+
+**The repository's creator** builds its content with `fwm repo`: `fwm repo bootstrap` generates the runs that record traces and create checkpoints, and `fwm repo compose` imports them:
+
+```
+sysroot ──(entry point)──> record-fesvr-trace ──> trace ──> create-checkpoints ──> checkpoints
+```
+
+**Its users,** who may have read access only, begin simulations with `fwm mkgen`: from a sysroot, from a FESVR trace, from a checkpoint, or for one SimPoint interval.
+
+## How it differs from riscv-simenv
+
+- **The sysroot is needed once.** Only the recording run (and `mkgen from-sysroot`, which runs the program for real) executes system calls on the host. A replay, or a run from a checkpoint, needs only the trace (which keeps a copy of the boot ELF as `boot.elf`), the checkpoint and the same `-m`/`-p`. There is no simenv to spawn and verify before every run.
+- **A replay prints nothing and writes no file.** Console output and output files exist only in the recording run. A replay checks that the program computes the same data, byte for byte, and stops at the first difference.
+- **Restore cost doesn't grow with the syscall history.** Loading a checkpoint seeks in the recording instead of re-executing it.
+- **Traces and checkpoints are portable.** They replay and load on any little-endian host (x86-64 or ARM64 Linux, macOS). Recording must be done on Linux.
+
+## Concepts
+
+| Term | Meaning |
+| --- | --- |
+| **sysroot** | A directory tree the program sees as `/`, imported read-only into the repository with a catalog (every file's permissions and SHA-256). |
+| **workload** | A name grouping the entry points that run the same program. |
+| **entry point** | How to launch a workload in a sysroot: the command, working directory, boot ELF (usually pk), stdin file, memory size and processor count, with optional free-form metadata. A workload can have entry points on several sysroots, and a sysroot can serve several workloads. |
+| **trace** | A recording (`+dev-traffic-record`) of a run of an entry point, identified by a UUID. It is *complete* if the program exited, or *truncated* if the run was stopped first (e.g. by spike's `-e<n>` instruction limit). |
+| **checkpoint** | A snapshot taken while replaying a trace. It loads only with that trace. A SimPoint checkpoint carries its SimPoint index, interval, weight and the analysis's maxK. |
+| **HART** | A hardware thread: a simulated processor. fwm's workloads run on one. |
+| **BBV, SimPoint** | spike's `-s<interval>` writes the program's basic block vectors (BBVs), one per interval of instructions. SimPoint picks a few intervals that represent the whole run, each with a weight: simulating only those (from a checkpoint at the start of each) estimates the whole run. |
+
+## Install
+
+fwm needs Python 3.12 or later. Install the `fwm` command from PyPI:
+
+```
+uv tool install fwm        # or: pipx install fwm, or pip install fwm in a virtual environment
+```
+
+Or from a checkout of its source:
+
+```
+git clone https://github.com/s117/fesvr-workload-manager.git fwm && cd fwm
+uv sync                # creates .venv with fwm
+uv run fwm --help
+# or install the command: uv tool install .
+```
+
+To run simulations, fwm also needs the spike built from [riscv-isa-sim](https://github.com/s117/riscv-isa-sim/tree/anycore-gcc_16.1.0-llvm_22.1.8) (see [Generated Makefiles](#generated-makefiles) for the versions it requires), or a simulator compatible with it.
+
+`fwm --version` says which fwm you have: its version, and the git commit it was installed from when installed from git, or the source tree and its commit for an editable install (`uv sync` here).
+
+## Shell completion
+
+fwm completes its commands and options on Tab, like riscv-simenv. It also completes the names in the repository: sysroots, workloads, entry points, traces and checkpoints. The names come from the repository given by `--repo-path` or `FWM_REPO_PATH`; without one, nothing is offered. Enable it in your shell's startup file:
+
+```
+eval "$(fwm completion zsh)"            # ~/.zshrc
+eval "$(fwm completion bash)"           # ~/.bashrc
+fwm completion fish | source            # ~/.config/fish/config.fish
+```
+
+- **`fwm` must be on `PATH`**, e.g. with `uv tool install .`, or with this repository's `.venv/bin` on `PATH`. `uv run fwm` alone is not enough: the shell calls `fwm` to complete.
+- **Faster shell start-up:** `eval` runs fwm each time a shell starts. To avoid that, write the script once to a file and source it, e.g. for zsh `fwm completion zsh > ~/.zfunc/_fwm` with `~/.zfunc` in `fpath` before `compinit`. Write it again after updating fwm.
+- **Checkpoints** are completed by name when the name is unique, and otherwise as `<trace UUID>:<name>`. Typing a trace prefix and `:` narrows the list to that trace's checkpoints.
+  - bash splits words at `:`, so it can't complete that form. zsh and fish can.
+  - zsh and fish also show a short description with each name.
+
+## Quick start
+
+```
+export FWM_REPO_PATH=~/fwm-repo                      # or pass --repo-path to every command
+export FWM_SIM_CMD=/path/to/spike                    # default: spike on PATH
+
+fwm repo init $FWM_REPO_PATH
+fwm repo compose add sysroot ./sysroots/gapbs-road road.sg
+fwm repo compose add entry road.sg bfs bfs-road-n64 bfs-road-n64.json
+```
+
+`bfs-road-n64.json` holds the launch fields of the entry point. In paths, `/` is the sysroot's root and `.` the working directory:
+
+```json
+{ "command": "bfs -f road.sg -n64", "working_directory": "/app",
+  "envs": null, "stdin_file": null, "boot_elf": "/boot/pk", "mem_mb": 2048, "nprocs": 1, "copy_spawn": false,
+  "metadata": { "suite": "GAPBS", "description": "Breadth-First Search on 'road'", "language": ["C++"] } }
+```
+
+- `envs` must be `null` or `{}`: FESVR passes no environment to the program.
+- `nprocs` must be 1: checkpoints can be created for a single HART only.
+- `boot_elf` and `stdin_file` must be regular files of the sysroot, not symlinks.
+- `copy_spawn` (optional, default `true`) is how a run of the program gets its sysroot (see `fwm spawn`): a copy, or, when `false`, links to the repository's read-only files. Linking takes no time or space, even for gigabytes of input, but the program can't open its files for writing, `stat` reports them read-only, and `lstat` and `readlink` see links into the repository. Use `false` only for a program that just reads the files it finds.
+- `metadata` is optional and free-form: any JSON object, such as a riscv-simenv suite's `index.yaml` metadata. fwm shows it and copies it into each run's `run.json`. Change it later with `fwm repo compose set-metadata`.
+
+Record a run, and import the recording as a trace. The recording says how it was made (`recording_provenance.json`: when, on what machine, the command line, and the spike's and FESVR's builds), which the trace keeps:
+
+```
+fwm repo bootstrap mkgen record-fesvr-trace bfs-road-n64 -o run1 && make -C run1 run SIM_FLAGS_EXTRA=-s100000000   # with BBVs
+fwm repo compose add trace bfs-road-n64 run1/trace --final-state run1/final-state.json
+# prints the trace UUID
+```
+
+Without `--final-state`, the trace has no instruction counts, and its checkpoints can't be validated. While an entry point has one trace, commands that take a trace also take the entry point's name, as below.
+
+Checkpoint files don't say how they were made, so their import takes the `spike --version` output for provenance:
+
+```
+"${FWM_SIM_CMD:-spike}" --version > sim-version.txt
+```
+
+Create checkpoints from a description file, and import them in one transaction. The description names each checkpoint and the instruction count where it is taken, e.g. `ckpts.txt` for two SimPoints:
+
+```
+bfs-road-n64.3.0.25: 300000005
+bfs-road-n64.7.0.75: 700000012
+```
+
+```
+fwm repo bootstrap mkgen create-checkpoints bfs-road-n64 ckpts.txt -o run2 && make -C run2 run
+fwm repo compose add checkpoint bfs-road-n64 run2/ckpt.desc run2/checkpoints --simpoint-info simpoints.json \
+    --sim-version sim-version.txt
+```
+
+- **A SimPoint's count is the sum of the sizes of the BBV intervals before it, not index × interval:** spike's intervals are a little longer than `-s` says. fwm-bootstrap writes the description and `simpoints.json` from SimPoint's output; see [docs/fwm-bootstrap-interface.md](https://github.com/s117/fesvr-workload-manager/blob/main/docs/fwm-bootstrap-interface.md), step 5.
+- **Names are just names.** `<entry>.<SimPoint index>.<weight>`, as here, is a readable convention; fwm reads nothing from a name.
+- **The run directory keeps a copy of the description,** `run2/ckpt.desc`, which the import reads.
+- **Every checkpoint of the description must be in `run2/checkpoints`,** which catches one spike didn't create: it creates none at or after the program's exit, without a word. Other files there are ignored.
+- **`--simpoint-info`** gives the SimPoint analysis, naming exactly the checkpoints of the description:
+  ```json
+  {"interval": 100000000, "maxK": 10,
+   "checkpoints": {"bfs-road-n64.3.0.25": {"index": 3, "weight": 0.2531}, "bfs-road-n64.7.0.75": {"index": 7, "weight": 0.7469}}}
+  ```
+
+Validate a checkpoint by running it to the end:
+
+```
+fwm mkgen from-checkpoint bfs-road-n64.3.0.25 -o run3
+make -C run3 run; fwm repo compose set-checkpoint-validation bfs-road-n64.3.0.25 run3      # passed or failed, recorded
+```
+
+## Running workloads
+
+`fwm mkgen` generates a run directory that begins a simulation; run it with `make -C <dir> run`:
+
+```
+fwm mkgen from-sysroot bfs-road-n64 -o a           # the real program in a fresh copy of its sysroot (Linux only)
+fwm mkgen from-fesvr-trace bfs-road-n64 -o b       # replay its trace from the start: any host, no output
+fwm mkgen from-checkpoint bfs-road-n64.3.0.25 -o c # from a checkpoint to the end of the program
+fwm mkgen from-simpoint bfs-road-n64.3.0.25 -o d   # one SimPoint interval from a SimPoint checkpoint
+make -C d run SIM=/path/to/721sim                  # make reports "Error 1": spike exits 1 when -e stops it
+```
+
+- **`from-sysroot`** does the program's real system calls, so it prints its console (also saved to `console.stdout` and `console.stderr`) and writes its files in `sysroot/`. Their answers come from the host, so its instruction count can differ from the trace's. For example, the program's stdout going to `/dev/null` rather than a file changed it by about 200 out of 1.5 million.
+- **`from-fesvr-trace`, `from-checkpoint` and `from-simpoint`** replay the trace: the program prints nothing and writes no file, and every run repeats the recorded one exactly, instruction counts included.
+- **`from-simpoint`** stops after the checkpoint's SimPoint interval (`-e$(INTERVAL)`, overridable with `INTERVAL=<n>`); weigh its results by the checkpoint's `simpoint_weight`.
+
+### An AI skill for fwm's users
+
+[skills/use-fwm/SKILL.md](https://github.com/s117/fesvr-workload-manager/blob/main/skills/use-fwm/SKILL.md) teaches an AI coding agent (Claude Code) how to use a repository: search it with `fwm list`, read metadata with `fwm show` and `run.json`, run simulations, judge them by `final-state.json`, and sweep SimPoint checkpoints. Install it once, from a checkout of fwm's source (see [Install](#install)):
+
+```
+ln -s "$PWD/skills/use-fwm" ~/.claude/skills/use-fwm
+```
+
+The link follows the checkout, so the skill stays in step with the fwm you pull.
+
+## Commands
+
+**Daily**, for every user of a repository (they work without write permission):
+
+| Command | Purpose |
+| --- | --- |
+| `fwm list [<kind>] [--workload\|--sysroot\|--entry\|--trace\|--checkpoint GLOB]... [--simpoint] [--json]` | List the names of a kind: workloads, sysroots, entries (the default), traces or checkpoints, filtered by the entities they are related to (see [Listing](#listing)). |
+| `fwm show <kind> <name> [--json]` | Show the details of one workload, sysroot, entry, trace or checkpoint. |
+| `fwm mkgen {from-sysroot <entry> \| from-fesvr-trace <trace> \| from-checkpoint <ckpt> \| from-simpoint <ckpt>} [-o dir]` | Generate a run directory that begins a simulation. |
+| `fwm spawn [--force] [--mode auto\|copy\|link] <entry> <dest>` | Make a working copy of an entry point's sysroot: copied, or linked to the repository's files; `auto`, the default, follows its `copy_spawn` (the `from-sysroot` and record Makefiles do it). |
+
+**Setup:** `fwm completion {bash|zsh|fish}` prints the shell completion script (see [Shell completion](#shell-completion)). It isn't offered on TAB.
+
+**`fwm repo`**, for the repository's creator. The repository as a whole:
+
+| Command | Purpose |
+| --- | --- |
+| `fwm repo init <dir>` | Create a repository. |
+| `fwm repo verify [<kind> <name>]` | Check the repository content against its catalogs (all, or one sysroot, trace or checkpoint), and for leftovers. |
+| `fwm repo gc [--dry-run]` | Remove the leftovers of interrupted commands. |
+| `fwm repo pull <source> {<entry>... \| --all} [--no-traces] [--no-checkpoints]` | Copy entry points of another repository into this one, with their sysroots, traces and checkpoints. |
+| `fwm repo push <dest> {<entry>... \| --all} [--no-traces] [--no-checkpoints]` | Copy entry points of this repository into another one. |
+| `fwm repo subrepo <new-repo> <entry>... [--no-traces] [--no-checkpoints]` | Create a repository with some entry points of this one: `repo init`, then `repo push`. |
+
+**`fwm repo compose`** changes the repository's content; each command needs write access:
+
+| Command | Purpose |
+| --- | --- |
+| `fwm repo compose add sysroot <dir> <name>` | Import a sysroot. |
+| `fwm repo compose add entry <sysroot> <workload> <name> <json_file>` | Add an entry point. |
+| `fwm repo compose add trace <entry> <recording-dir> [--final-state f.json]` | Import a recording, with its provenance; prints its UUID. |
+| `fwm repo compose add checkpoint <trace> <desc> <checkpoint-dir> [--simpoint-info j] [--sim-version f]` | Import the checkpoints of a description in one transaction; prints `<uuid>:<name>` for each. |
+| `fwm repo compose remove <kind> <name> [--recursive]` | Remove a sysroot, entry, trace or checkpoint, with what depends on it when `--recursive`. |
+| `fwm repo compose set-metadata <entry> <json_file>` | Replace an entry point's metadata (`null` removes it), keeping its traces. |
+| `fwm repo compose set-checkpoint-validation <checkpoint> <run-dir>` | Judge a `mkgen from-checkpoint` run to the end, and set the checkpoint's validation: passed or failed. |
+
+**`fwm repo bootstrap`** makes the repository's traces and checkpoints, in the pipeline's order; its commands only read the repository:
+
+| Command | Purpose |
+| --- | --- |
+| `fwm repo bootstrap mkgen record-fesvr-trace <entry> [-o dir]` | Generate a run directory that records a run of an entry point: its FESVR trace. |
+| `fwm repo bootstrap check-fesvr-trace <recording-dir> [--final-state f.json] [--json]` | Check a recording as `repo compose add trace` would, without a repository: complete, well formed, agreeing with its final state. |
+| `fwm repo bootstrap mkgen create-checkpoints <trace> <desc> [-o dir]` | Generate a run directory that creates the checkpoints of a description. |
+
+## Listing
+
+`fwm list <kind>` prints the names of the workloads, sysroots, entries (the default), traces or checkpoints, one per line, or with `--json` as a JSON array. `fwm show <kind> <name>` gives the details of each. Traces are listed by UUID, and checkpoints as `fwm show checkpoint` takes them (below).
+
+Its filters `--workload`, `--sysroot`, `--entry`, `--trace` and `--checkpoint` keep what is related to an entity they match, directly or through others:
+
+| Kind | Is related to |
+| --- | --- |
+| workload | its entry points, their sysroots, traces and checkpoints |
+| sysroot | its entry points, their workloads, traces and checkpoints |
+| entry point | its workload, sysroot, traces and their checkpoints |
+| trace | its entry point (and that one's workload and sysroot), and its checkpoints |
+| checkpoint | its trace, and that trace's entry point, workload and sysroot |
+
+```
+fwm list checkpoints --workload bfs                   # the checkpoints of bfs's entry points
+fwm list sysroots --checkpoint 'bfs-road*'            # the sysroots where those checkpoints were made
+fwm list entries --entry 'bfs-*'                      # a filter of the listed kind selects by name
+fwm list traces --sysroot s1 --sysroot s2             # a filter given twice: either
+fwm list entries --workload bfs --sysroot road.sg     # different filters: both
+fwm list checkpoints --checkpoint '8f25*:bfs*'        # a checkpoint as <trace UUID glob>:<name glob>
+fwm list checkpoints --entry bfs-road-n64 --simpoint  # those `fwm mkgen from-simpoint` can simulate
+```
+
+- **Several filters:** each keeps its own set, and the result is their intersection. `fwm list workloads --sysroot s1 --entry c` lists the workloads related to `s1` and to `c`, even through different entry points.
+- **Patterns** are shell-style globs (`*`, `?`, `[...]`, case-sensitive; quote them in the shell); a plain name matches exactly. `--trace` matches the UUID as `fwm show` prints it.
+- **`--simpoint`** keeps what is related to a checkpoint with SimPoint information: exactly the checkpoints `fwm mkgen from-simpoint` accepts. Listing other kinds, it keeps those with at least one such checkpoint.
+- **A pattern that matches nothing of its kind** prints a warning (likely a typo); the command still succeeds, with nothing listed.
+- **A sysroot without entry points** is related to nothing else: it is listed without filters, or with a matching `--sysroot`.
+
+How to name things:
+- **A trace** is its UUID (a unique prefix is enough), or the name of an entry point that has one trace.
+- **A checkpoint** is `<trace>:<name>`, or just its name when no other trace has a checkpoint of that name.
+- **A checkpoint's name** is just a name: its SimPoint data comes only from `--simpoint-info`.
+
+Every `repo compose add` takes `--exist-ok`: if the same item is already in the repository, it prints it and succeeds. An entry point that differs only in its description or metadata is the same; it keeps its own. Without it, adding what exists fails with status 5. A recording that already is a trace of the entry point counts as existing, so a rerun never creates a second trace.
+
+Exit status: 0 success, 1 unexpected error, 2 usage error or unusable repository, 3 not found, 4 failed check, 5 already exists: an item of the repository, or a destination (`spawn` into an existing directory without `--force`, `repo init` or `subrepo` into a non-empty one).
+
+## Merging and subsets
+
+`repo pull`, `repo push` and `repo subrepo` copy entry points from one repository to another. Each entry point brings its workload and sysroot, and its traces with their checkpoints, unless `--no-traces` or `--no-checkpoints` leave them out:
+
+```
+fwm repo subrepo ~/small bfs-road-n64 pr-road           # a new repository with two entry points
+fwm --repo-path ~/small repo pull /shared/repo cc-road  # add a third one to it
+fwm --repo-path /shared/repo repo push ~/merged --all   # merge everything into another, existing repository
+```
+
+- **Rows are copied as they are:** names, trace UUIDs, provenance, metadata, SimPoint data and validation results.
+- **What the receiving repository has is kept,** under the rules of `--exist-ok`:
+  - a sysroot of the same name and content;
+  - an entry point of the same name, sysroot, workload and launch fields, with its own metadata;
+  - a trace made of the same recording, with its own UUID, to which the copied checkpoints attach;
+  - a checkpoint of the same name and content, with its own validation result.
+
+  Copying again adds nothing, and its rows never change.
+- **Conflicts:** the same name with something else is a conflict. Every entry point is checked before anything is copied, and then nothing is: the command fails with status 5. If another command changes the receiving repository during the copy, a conflict found then stops it, keeping the entry points already committed.
+- **Interruptions:** each entry point is committed on its own, like an import. An interrupted copy can be run again to complete it. `subrepo` removes its new repository if the copy fails (an empty directory it was created in is left empty, as it was).
+- **Read access is enough** to copy from a repository. Completion offers the entry points of the repository copied from: for `pull`, those of its `<source>`.
+- **Output:** each item prints as `added|present <kind> <id>` (`subrepo` prints the new repository's path), and a summary goes to stderr.
+
+## Generated Makefiles
+
+Every Makefile has a single target, `run`. It first removes that run's previous outputs, so it can be run again. A run directory can be generated again for the same kind of run, but not for another kind, whose Makefile wouldn't remove the former outputs.
+
+Next to it, `run.json` describes what the run uses: the sysroot, workload, entry point (with its metadata), trace and checkpoint, as `fwm show --json` shows them when the Makefile is generated. Its `kind` says which command generated it, e.g. `from-simpoint`, or `record` for `repo bootstrap mkgen record-fesvr-trace`. A create-checkpoints run also lists the checkpoints it requests. See [docs/fwm-bootstrap-interface.md](https://github.com/s117/fesvr-workload-manager/blob/main/docs/fwm-bootstrap-interface.md#runjson).
+
+Override variables on the make command line:
+
+| Variable | Use |
+| --- | --- |
+| `SIM` | The simulator (default: `FWM_SIM_CMD` when generated, else `spike`). |
+| `SIM_FLAGS_EXTRA` | spike options such as `-e<n>` or `-s<interval>`. They come before the FESVR options. |
+| `FESVR_FLAGS_EXTRA` | FESVR options such as `+verbose`. |
+| `BOOT_ELF_FLAGS_EXTRA`, `APP_CMD_EXTRA` | (from-sysroot, record) Options of the boot ELF (pk), and extra program arguments. |
+| `FWM` | (from-sysroot, record) How to run fwm for `spawn` (default: the Python that generated the Makefile, so fwm needn't be on `PATH`). |
+| `INTERVAL` | (from-simpoint) The instructions to simulate (default: the checkpoint's SimPoint interval). |
+
+spike's exit status:
+- **Without `-e`:** the program's exit status, or 0 after creating the last checkpoint.
+- **With `-e<n>`:** 1 when the run was stopped after `n` instructions, and 0 when the program ended first.
+
+**Don't judge a run by make's exit status; let fwm judge it.** A program that legitimately exits non-zero makes `make` fail, while spike stopped by SIGTERM (e.g. a job's time limit) exits 0. `fwm repo compose add trace --final-state` judges a record run, `fwm repo compose add checkpoint` a checkpoint creation, and `fwm repo compose set-checkpoint-validation` a run from a checkpoint.
+
+`final-state.json` says why the run finished, and each hart's instruction count there: `target_exit` (with `exit_code`), `checkpoints_created`, `instret_limit` (`-e`), `debug_quit`, `interrupted` (with `signal`), `error` (with the simulator's message, and no count) or `unfinished` (spike died). fwm needs a spike that writes `finish_reason` (riscv-isa-sim eb87256 / riscv-fesvr 6f2d3b8 or later): older ones wrote counts at the program's exit that depended on where the run started, and their dumps are refused. fwm also needs the recording's provenance (`recording_provenance.json`), written from riscv-isa-sim fd01c25 / riscv-fesvr 8472993 on, and refuses a recording without it. Those versions also include the fixes of riscv-isa-sim 872d191 / riscv-fesvr ecc3df3, needed in batch jobs: before them, a run that got SIGTERM twice (from a job's time limit, under make) could leave an empty or broken dump.
+
+The Makefiles leave other simulator outputs, such as `bbv_proc_0.bb.gz` from `-s`, to the caller: use a fresh run directory, or remove them.
+
+## Clusters and shared repositories
+
+- **Commands that change the repository** (`repo init`, `gc`, `pull`, every `repo compose` command, and `push` / `subrepo` for the repository they write) should run on one host, e.g. a Snakemake controller. The database uses SQLite's rollback journal, which works on a shared filesystem, and they wait for each other.
+- **Compute jobs** run the generated Makefiles. Only the `from-sysroot` and record runs open the database, read-only, when `fwm spawn` checks the sysroot out.
+- **Read-only use:** `list`, `show`, `mkgen`, `spawn`, `repo verify`, every `repo bootstrap` command, `repo push` / `subrepo` (copying from it) and completion work in a repository you can't write to, e.g. one shared with your group.
+- **Interrupted commands:** a command killed by its scheduler (SIGTERM) cleans up after itself. A harder kill can leave staging or files that no row refers to. `fwm repo verify` reports them and `fwm repo gc` removes them; it never leaves rows whose files are missing. A command committing at the moment `verify` runs can show as leftovers too: run it again.
+
+## Development
+
+```
+uv run pytest            # unit tests
+uv run mypy fwm tests
+uv run ruff check fwm tests
+```
+
+The end-to-end test needs a spike built from `riscv-isa-sim` and a small workload:
+
+```
+FWM_TEST_SPIKE=/path/to/spike FWM_TEST_SYSROOT=/path/to/sysroot FWM_TEST_ENTRY_POINT=/path/to/entry_point.json \
+FWM_TEST_SPIKE_SRC=/path/to/riscv-isa-sim uv run pytest tests/integration
+```
+
+The bootstrap pipeline's interface to fwm is described in [docs/fwm-bootstrap-interface.md](https://github.com/s117/fesvr-workload-manager/blob/main/docs/fwm-bootstrap-interface.md).
+
+Notes for coding agents (how fwm is built, the spike behaviour it relies on, decisions, conventions) are in [docs/agent-guide.md](https://github.com/s117/fesvr-workload-manager/blob/main/docs/agent-guide.md).