static-fire 0.44.0 → 0.54.1

package/src/guide.md

@@ -14,8 +14,12 @@ safe default. How you talk with the user is yours to decide. The second part is
 format and command.
 
 Run every command as `static-fire <command>`, or `npx static-fire <command>` when `static-fire` is not
-installed. A command finds the project by looking in the current folder and each folder above it for
-`static-fire.yaml`.
+installed.
+
+**Which project a command uses.** A command uses the nearest project above the current folder (a folder
+with `static-fire.yaml`, the current one included). Outside any project it uses the home, the one project
+the user keeps all their tasks and results in, and prints `using the home project: <folder>` on stderr.
+`static-fire home` prints the home; `static-fire home <folder>` sets it.
 
 ## The rule above all: realism
 
@@ -25,6 +29,19 @@ score is worth something only when it predicts how a configuration does on the u
 difference from the real job is a known gap. Write each gap of a task in the project's `README.md` (make
 the file when the first gap comes up), so the user can see what the score does not cover.
 
+## What spends
+
+These start agent runs or judge calls, which cost money or use up the user's plan:
+
+- `static-fire run`: agent runs, and judge calls to grade each trial.
+- `static-fire rerun <batch>`: the failed trials of the batch again, with their judge calls.
+- `static-fire regrade`: judge calls only, for each trial it grades again.
+- On the dashboard: New run on the Runs page, and Regrade on the Results page.
+
+The rule: get the user's approval before each one starts, every time, with what it will run (step 6). An
+approval covers the one command it was given for. `run --plan` and `regrade --plan` print the plan and
+spend nothing.
+
 ## The path
 
 ### 1. Learn the goal, the examples and what good looks like
@@ -53,12 +70,28 @@ Do not invent the user's taste: ask what good looks like.
 
 ### 2. Set up the project and settle the block list
 
-**Why.** A project is the folder that holds the tasks, the configurations and the records. A trial runs with
-no permission prompts, so nobody is there to approve a tool that sends mail, posts a message, spends money
-or deploys. The block list stops those tools in trials.
+**Why.** A project is the folder that holds the tasks, the configurations and the records. Most users need
+one: the home. A trial runs with no permission prompts, so nobody is there to approve a tool that sends
+mail, posts a message, spends money or deploys. The block list stops those tools in trials.
+
+**Do.** Run `static-fire init --json`. It never waits for input and never changes a file that exists.
+
+- With no home, outside any project, it makes the project in `~/static-fire` and saves it as the home.
+  `static-fire init <folder> --json` makes it in that folder instead, and saves that as the home.
+- With a home, outside any project, it writes nothing and reports the home. `static-fire init <folder>`
+  then makes a second project in that folder, with its own records and dashboard. The home does not change.
+- Inside a project, it writes nothing and reports that project.
 
-**Do.** In the folder the user chose (an empty folder is best), run `static-fire init --json`. It never
-overwrites a file. It makes:
+Tell the user where the home is, or will be, before you add their task to it. `static-fire home` prints it,
+or says none is set, and writes nothing; with none set, the home will be `~/static-fire` unless the user
+names another folder. The report's `project` is the project's folder, `created` says if this run made it,
+`home` is the home's folder (or null), `isHome` says if `project` is the home, and `homeSaved` says if this
+run saved the home.
+
+**Safe default.** Put the task in the home. Make a second project only when the user asks for a separate
+one.
+
+A new project has:
 
 - `configs/<harness>.yaml`: one agent configuration for each harness it found on the `PATH` (`claude`,
   `codex`, `agy`, `pi`), with a default model.
@@ -67,15 +100,27 @@ overwrites a file. It makes:
   run until one is added (see the reference).
 - `evals/`, `suites/`, `static-fire.yaml`, a `.gitignore` for `runs/` and `.cache/`, and `AGENTS.md`.
 
-The JSON report has, for each harness: `found` and `path`, `blockable`, `tools` (each with `name`, `kind`,
-`foundIn` and `block`, the entries that block it) and `notes`. It also has `judge`, `written`, `kept` and
-`notes`. Run `static-fire init --json` again in the project at any time: it changes nothing and lists the
+The JSON report also has, for each harness: `found` and `path`, `blockable`, `tools` (each with `name`,
+`kind`, `foundIn` and `block`, the entries that block it) and `notes`. It also has `judge`, `written`, `kept`
+and `notes`. Run `static-fire init --json` again in the project at any time: it changes nothing and lists the
 tools again.
 
-**Settle the block list.** For each tool in the report, decide if it acts on an outside service: sends or
-posts something, changes data in an account, spends money, or deploys. Propose the block list to the user
-with a short reason for each entry, and show the tools you propose to leave on. The user approves or changes
-it. Then write the `block` entries under `trials.blockedTools.<harness>` in `static-fire.yaml`.
+**Settle the block list.** `init` finds the tools; it does not choose and does not ask. For each tool in the
+report, decide if it acts on an outside service: sends or posts something, changes data in an account,
+spends money, or deploys. Then the user decides. Before they decide, they need to see:
+
+- Every tool found, for each harness, with its kind and where it was found. They cannot judge a list they
+  have not seen.
+- Which tools you propose to block, each with its reason.
+- Which tools stay on.
+- Their choices: approve your list, change it, or block nothing.
+
+Blocking nothing is a valid choice. Its facts: a trial then runs with every tool of the user's setup and no
+permission prompts, which is the closest to how their agents really run. And a tool that acts on an outside
+service can then act in a trial, with nobody there to stop it.
+
+Write the entries of the user's choice under `trials.blockedTools.<harness>` in `static-fire.yaml`, copied
+from each tool's `block` list.
 
 **Facts.**
 
@@ -92,7 +137,7 @@ Run `static-fire init --json` again after the user installs a harness.
 result of a fast "yes" from the user must be safe. Tools that only read (search, docs, the local file
 system) can stay on, because the real job has them.
 
-**Configurations.** Show the user the configurations `init` wrote, and change them to what they want to
+**Configurations.** Show the user the configurations in the project, and change them to what they want to
 compare: other models, other effort levels, more than one configuration per harness. A configuration is a
 small YAML file (see the reference). `init` writes a current model id for each harness it found. For another
 model, take its id from the harness itself (its model list or its documentation). Static Fire does not check
@@ -126,20 +171,29 @@ comment on each part, and an empty `evals/<id>/inputs/`. Put the example inputs
   error and every grading of the task is invalid.
 - A check is cheap to test before any trial: write a sample output by hand and run the check's command or
   pattern on it yourself. A `regex` is a JavaScript regular expression.
-- Give each criterion a level: `floor` (most configurations pass), `strong` (good ones pass) or `ceiling`
-  (only the best pass). A mix makes the scores spread out.
+- Test each derived evidence script the same way before the first trial: run it on a sample output, see
+  that it ends well inside its 60-second limit, and open the PNG it writes. Every file in the task's
+  `grader/` folder is part of the grading version, so a fix after a trial costs a regrade (step 5).
+- Give each criterion a level: `floor` (most configurations should pass), `strong` (good ones pass) or
+  `ceiling` (only the best pass). A mix makes the scores spread out. These three names are the ones the
+  dashboard shows.
 - `static-fire tasks` warns under 15 criteria in all, under 3 in a measure, or when a level is missing.
   These are advice. With 15 criteria, one criterion moves the score by about 7 points.
 - A gate is a criterion outside the measures: when one fails, the task score is 0. "The output file exists
   and is not empty" is always a gate, added by the engine.
 
-**Safe default.** Turn each thing the user said about good and bad output into one or more criteria. Show
-the user the criteria in plain words and let them correct them before you run anything. Write each gap
-between the task and the real job in the project's `README.md`.
+**Confirm the criteria with the user.** The criteria decide every score, so the user approves them as their
+own step, apart from any approval to run. Show each criterion in plain words with its level, and explain the
+levels in plain words with their names: floor, strong, ceiling. Offer to change any criterion, and get the
+user's yes on the criteria before you go on.
+
+**Safe default.** Turn each thing the user said about good and bad output into one or more criteria. Write
+each gap between the task and the real job in the project's `README.md`.
 
 ### 4. Check the task and put it in a suite
 
-**Why.** A task with an error cannot run. The dashboard has a button only for the tasks that are in a suite.
+**Why.** A task with an error cannot run. A task counts toward the Agent Performance Index only when it is in a
+suite.
 
 **Do.** Run `static-fire tasks`. Each error stops the task from running: fix every one. Warnings are advice.
 Then add the task to a suite, a file `suites/<id>.yaml` (see the reference), and run `static-fire suites`,
@@ -153,8 +207,8 @@ suite fits.
 **Why.** The first trial shows that the task works: the agent finds its inputs, writes the output where the
 task expects it, and each verdict is right for the right reason. It is the least spend that shows this.
 
-**Before you run.** A trial spends money: the agent run, and the judge calls that grade it. Tell the user
-what will run and get their approval first.
+**Before you run.** A trial spends: the agent run, and the judge calls that grade it. Show the user its plan
+(add `--plan` to the command below, as in step 6) and get their approval first.
 
 **Do.** Run one trial of one configuration, usually the one the user works with:
 `static-fire run --tasks <task> --configs <config> --trials 1`. A trial can take several minutes (the limit
@@ -177,37 +231,52 @@ the task works. Show the user the output and the verdicts, so they can say wheth
 their own view.
 
 **Facts.** A change to the prompt, the output path, or a file in `inputs/` or `project/` makes a new task
-hash: earlier trials no longer share a chart with new ones. A change to a criterion, `context`, a reference
-or the judge makes a new grading version: earlier trials are left out of the view until a
-`static-fire regrade` grades them again with judge calls only. So settle the task before the batch.
+hash: earlier trials no longer share a chart with new ones. A change to a criterion, `context`, a reference,
+any file in `grader/` (a check's helper script or a derived evidence script) or the judge makes a new
+grading version: earlier trials are left out of the view until a `static-fire regrade` grades them again.
+That regrade spends judge calls and needs the user's approval like any other spend. So settle the task, its
+grader scripts included, before the batch.
 
 ### 6. Work out the batch cost, get approval, run the batch
 
-**Why.** The batch is where most of the money goes. The user approves it knowing what it costs.
+**Why.** The batch is where most of the spend goes. The user approves it knowing what it uses.
+
+**The plan in runs comes first.** Add `--plan` to the `run` command you mean to start, for example
+`static-fire run --suites <suite> --configs <a>,<b> --trials 3 --plan --json`. It starts nothing and writes
+nothing. It gives the trials, the agent runs for each configuration (a human configuration needs none),
+the judge calls for the judge as a range, and what the range leaves out (`judgeCalls.notCounted`). These
+counts are what a user on a subscription can judge: their plan counts runs and usage, not dollars. `static-fire regrade <flags> --plan --json` gives the plan of a
+regrade the same way. `rerun` has no plan: it runs the failed trials of the batch again, each with its
+grading.
 
 **Facts.**
 
-- Static Fire gives no cost estimate and enforces no limit. You work out the cost.
-- The cost of a configuration is known only once it has run. Configurations on other models can cost very
-  different amounts for the same task.
-- A batch runs `--trials` trials (3 by default) of each configuration in `--configs` on each task. Each
-  trial that runs is also graded: its judge cost is in `grades.judgeCostUsd`.
+- Judge calls are often the larger part of a batch, in count and in cost. Their number per trial comes
+  from the judge criteria, put in groups: criteria with the same evidence and the same derived evidence
+  script are one group; each `isolate` criterion, each other set of evidence, each other derived script and
+  the judged gates make more groups. Each group gets one judge call per vote. With 3 votes (the default,
+  `--votes`), 2 calls run first and the third only when they disagree. That is the range: the low end when
+  every first pair agrees, the high end when none does.
+- Outside the range: a judge answer that does not parse is tried once more, and the plan says that this
+  call is not counted. A trial with no output, or with a derived evidence script that fails, gets fewer calls.
+- Dollar figures are a second fact, and they are API prices, not what a subscription is charged. The cost
+  of a configuration is known only once it has run, and configurations on other models can cost very
+  different amounts for the same task. A trial's judge cost is in `grades.judgeCostUsd`.
 - Claude Code and Pi report their own cost (`costSource: reported`). Codex and Antigravity costs are
   estimated from tokens and the engine's price list (`estimated`). A model that the price list does not have
   gives no cost (`unknown`, shown as `?` in the `run` output). A project can have its own `pricing.yaml`.
 - Cost that is not in the harness's own report, such as image generation by an outside service, is not
-  counted.
-- With several configurations, you can run one trial of each one first (real numbers for each, more spend
-  before the batch), or scale from what you have. Say which numbers are measured and which are a guess.
+  counted. Static Fire enforces no spending limit.
 
-**Safe default.** The user sees the batch cost before the batch, with what is measured and what is guessed
-made clear, and approves it. Nothing runs before that approval.
+**Safe default.** The user sees the plan in runs before the batch, with the dollar figures you have as API
+prices, what is measured made clear from what is scaled up, and approves it. Nothing runs before that
+approval. A batch that starts prints the same plan first.
 
 **Do.** Run the batch, for example `static-fire run --suites <suite> --configs <a>,<b> --trials 3`, or
 `--configs all` for every configuration. It runs 4 trials at a time by default (`--concurrency`). A batch can
 take a long time, so start it in the background and follow it with `static-fire list` or on the dashboard's
 Runs page. When trials fail for a reason outside the task (a usage limit, a login),
-`static-fire rerun <batch>` runs the failed ones again in the same batch.
+`static-fire rerun <batch>` runs the failed ones again in the same batch, after the user approves it.
 
 ### 7. Open the dashboard
 
@@ -219,14 +288,17 @@ not exit. It prints `Static Fire → http://localhost:4477 (records: <path>)`. G
 `--port <n>` picks another port. You can start it at any time; trials show up as they are graded.
 
 **What the user sees.** The Results page shows the score of each configuration against its cost per task, as
-a chart or a table, for the Workflow Index (all suites), one suite, or one task. With one task in one suite,
-each of the three shows the same score. Each point is the mean of the valid trials with its 95% confidence
-interval. Opening a trial shows each verdict with its reason. The Runs page lists each batch and its trials,
+a chart or a table, for the Agent Performance Index or one task. The index covers the suites the user turns on with
+the suite buttons. With none picked, it covers the best set the data supports, and one line says which suites it
+covers and why each other suite is left out. `static-fire show` uses the same set and prints the same line. With one
+task in one suite, the index and the task show the same score. Each point is the mean of the valid trials with its 95% confidence
+interval. Opening a trial shows its output, with each verdict and its reason in the Verdicts tab. The Runs page lists each batch and its trials,
 and can start a new run. The Compare page shows what changed between two grading versions of a task.
 
-Before any trial is graded, the Results page says "No trials in this view". A task that is in no suite has no
-button on the Results page. The address `http://localhost:4477/?scope=task:<task>` still shows it, and so
-does `static-fire show --tasks <task>`, as text.
+Before any trial is graded, the Results page says "No trials in this view". The Task section of the Filter menu
+lists every task of the project, also a task that is in no suite; `static-fire show --tasks <task>` shows one
+task as text. The Tasks page shows each suite and task: its description, what the agent gets and how it is
+graded.
 
 ## Reference
 
@@ -402,12 +474,16 @@ The contract for a script:
 - **Input.** Argument 1 and `$OUTPUT` are the output file. `$TASK_DIR` is the task folder, so the script can
   read other task files. `$WORKSPACE` is the working folder, and the script runs in it. The time limit is 60
   seconds.
+- **At the same time.** Gradings run at the same time: up to `--concurrency` (4 by default) in a batch or a
+  regrade, each in its own trial folder. In one grading, every script it needs runs at once. So a script
+  writes only to its argument 2 path and keeps nothing in a fixed path that another run could share.
 - **Output.** Write one PNG to the path in argument 2 (also `$DERIVED`). A non-zero exit, a time-out, no file
   or a file that is not a PNG makes the verdict an error, so the grading is invalid.
 - **Where it stays.** `runs/trials/<trial-id>/gradings/<grading-id>/_derived/<script>.png`, with the
   script's output next to it as `.log`.
 - **Sharing.** Criteria that name the same script share one image, and one judge call when their other
-  evidence is the same. A new script name or new script content gives a new grading version.
+  evidence is the same. A new script name or new script content gives a new grading version: test it before
+  the first trial (step 3).
 
 An example script, for an image output:
 
@@ -484,7 +560,7 @@ tasks:
 | Field | What |
 |---|---|
 | `title` | Shown on the dashboard ("Thumbnails Index") |
-| `description` | Optional. Shown with the suite's charts |
+| `description` | Optional. Shown on the Tasks page |
 | `tasks` | The task ids |
 | `weights` | Optional. Task id to weight; 1 by default |
 
@@ -511,16 +587,17 @@ records the list it ran with.
 
 | Command | What |
 |---|---|
-| `static-fire init [--json]` | Make a project in the current folder, or list the outside tools again in a project |
+| `static-fire init [<folder>] [--json]` | Make the home (`~/static-fire`, or `<folder>`), or report the home or the current project; with a home, `<folder>` makes a second project. Lists the harnesses and outside tools |
 | `static-fire guide` | This guide |
+| `static-fire home [<folder>]` | Show the home, the project a command uses outside any project, or save a project folder as the home |
 | `static-fire new task <id>` | A new task folder with a commented `task.yaml` |
 | `static-fire tasks` | Each task, with its errors and warnings |
 | `static-fire configs` | Each configuration and its label |
 | `static-fire judges` | Each judge; the default is marked |
 | `static-fire suites` | Each suite and its tasks |
-| `static-fire run` | One batch. `--tasks <a,b>` or `--suites <a,b>`, `--configs <a,b>` or `all` (all by default), `--trials <n>` (3), `--judge <id>`, `--votes <n>` (3), `--concurrency <n>` (4) |
+| `static-fire run` | One batch. `--tasks <a,b>` or `--suites <a,b>`, `--configs <a,b>` or `all` (all by default), `--trials <n>` (3), `--judge <id>`, `--votes <n>` (3), `--concurrency <n>` (4). `--plan` prints the plan and starts nothing; `--plan --json` for JSON |
 | `static-fire rerun <batch>` | The failed trials of a batch again, as new trials. `--concurrency <n>` |
-| `static-fire regrade` | Grade stored outputs again under the current grading, with judge calls only. `--tasks`, `--configs`, `--batch <id>`, `--judge`, `--votes`, `--only-invalid`, `--concurrency` |
+| `static-fire regrade` | Grade stored outputs again under the current grading, with judge calls only. `--tasks`, `--configs`, `--batch <id>`, `--judge`, `--votes`, `--only-invalid`, `--concurrency`, `--plan` (with `--json`) |
 | `static-fire list` | Each batch, newest first, with its trial counts |
 | `static-fire show` | The scores as text. `--tasks` or `--suites`, `--judge`, `--grading <hash>`, `--since <date>`, `--until <date>`, `--json` |
 | `static-fire serve [--port <n>]` | The dashboard, at port 4477 by default |
@@ -543,5 +620,5 @@ A flag that a command does not take is an error. A list is comma-separated: `--c
   questions.
 - **The copy check reads text only.** An image output that copies a published original is not caught.
 - **Cost** is reported by Claude Code and Pi, estimated for Codex and Antigravity, and unknown for a model
-  that has no price. Judge cost is not part of a configuration's cost. There is no estimate before a batch
-  and no spending limit.
+  that has no price. Judge cost is not part of a configuration's cost. Before a batch there is the plan in
+  runs (`--plan`), not a dollar estimate. There is no spending limit.

package/src/server.ts

@@ -7,14 +7,18 @@ import type { ProjectPaths } from "./core/project";
 import { projectCatalog } from "./core/view";
 import { PAGES } from "../web/pages";
 
-/** Serves the dashboard of `project`: its store, and its configs, judges, suites and tasks. */
-export function serve(port: number, project: ProjectPaths) {
+/**
+ * Serves the dashboard of `project`: its store, and its configs, judges, suites and tasks. The page is bundled
+ * once, minified and cached, unless `development` asks for Bun's development mode (bundled on request, with hot
+ * reload), which is for work on the engine (`static-fire serve --dev`, `bun run dev`).
+ */
+export function serve(port: number, project: ProjectPaths, { development = false }: { development?: boolean } = {}) {
   const { root, dbPath, cache } = project;
   const store = openStore(dbPath);
   const api = apiRoutes({ store, dbPath, catalog: () => projectCatalog(root), root, cacheDir: cache });
   const server = Bun.serve({
     port,
-    development: process.env.NODE_ENV !== "production",
+    development,
     idleTimeout: 0,
     // Every page path serves the one page; it shows the page its path names (web/pages.ts).
     routes: { ...Object.fromEntries(PAGES.map((p) => [p.path, index])), ...api.routes },