@giamme/forge 0.8.0 → 0.9.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (61) hide show
  1. package/CHANGELOG.md +24 -0
  2. package/README.md +106 -1766
  3. package/SKILL.md +3 -2
  4. package/VERSION +1 -1
  5. package/docs/README.md +26 -0
  6. package/docs/add-ons.md +33 -0
  7. package/docs/artifacts.md +69 -0
  8. package/docs/commands.md +315 -0
  9. package/docs/dataset.md +20 -0
  10. package/docs/decomposed-runs.md +268 -0
  11. package/docs/development.md +95 -0
  12. package/docs/forge-ui.md +92 -0
  13. package/docs/fractal.md +181 -0
  14. package/docs/install.md +160 -0
  15. package/docs/memory.md +156 -0
  16. package/docs/models.md +118 -0
  17. package/docs/reviews.md +231 -0
  18. package/docs/safety.md +55 -0
  19. package/docs/troubleshooting.md +54 -0
  20. package/forge +5 -0
  21. package/package.json +3 -2
  22. package/references/decompose.md +1 -1
  23. package/references/fractal.md +1 -1
  24. package/references/harnesses.md +18 -6
  25. package/references/solo.md +1 -1
  26. package/scripts/forge-dispatch.sh +7 -1
  27. package/scripts/forge-runs.py +121 -0
  28. package/scripts/forge-runtime.py +59 -6
  29. package/scripts/forge-version.py +54 -12
  30. package/scripts/forge_dataset/harvest.py +1 -1
  31. package/scripts/forge_dataset/importer.py +1 -1
  32. package/scripts/forge_fractal/__init__.py +2 -1
  33. package/ui/README.md +80 -20
  34. package/ui/THIRD_PARTY_NOTICES.md +108 -0
  35. package/ui/docs/api.md +99 -17
  36. package/ui/forge_ui/export.py +16 -6
  37. package/ui/forge_ui/fractal.py +1 -2
  38. package/ui/forge_ui/reader.py +95 -7
  39. package/ui/forge_ui/server.py +10 -1
  40. package/ui/forge_ui/static/D-DINCondensed-Bold.woff2 +0 -0
  41. package/ui/forge_ui/static/app.js +279 -124
  42. package/ui/forge_ui/static/cloud3d.js +3 -2
  43. package/ui/forge_ui/static/cloudui.js +8 -5
  44. package/ui/forge_ui/static/convo.js +295 -0
  45. package/ui/forge_ui/static/datatable.js +232 -0
  46. package/ui/forge_ui/static/diffview.js +29 -10
  47. package/ui/forge_ui/static/findings3d.js +134 -0
  48. package/ui/forge_ui/static/graph.js +3 -3
  49. package/ui/forge_ui/static/icons.js +94 -0
  50. package/ui/forge_ui/static/index.html +25 -13
  51. package/ui/forge_ui/static/iso.js +70 -19
  52. package/ui/forge_ui/static/leaderboard.js +44 -48
  53. package/ui/forge_ui/static/panel.js +68 -32
  54. package/ui/forge_ui/static/runmodel.js +17 -1
  55. package/ui/forge_ui/static/scene3d.js +518 -43
  56. package/ui/forge_ui/static/search.js +2 -1
  57. package/ui/forge_ui/static/shell.js +137 -0
  58. package/ui/forge_ui/static/stagectl.js +150 -0
  59. package/ui/forge_ui/static/style.css +924 -469
  60. package/ui/forge_ui/static/theme.js +25 -0
  61. package/ui/forge_ui/transcripts.py +608 -0
@@ -0,0 +1,160 @@
1
+ # Install and update
2
+
3
+ [← README](../README.md) · [All docs](README.md)
4
+
5
+ Install forge from npm or a git checkout, check which agent CLIs it can reach, and keep every harness installation current.
6
+
7
+ ## Requirements
8
+
9
+ forge runs on macOS and Linux. It needs bash, git and Python 3.9 or later, plus the agent
10
+ CLIs you route models through; the doctor below shows which ones it finds. Installing from
11
+ npm brings none of these with it.
12
+
13
+ ## From npm
14
+
15
+ ```bash
16
+ npm install -g @giamme/forge
17
+ forge install
18
+ ```
19
+
20
+ `forge install` links the installed package into each harness, as the installer does for a
21
+ checkout below. It is a step of its own rather than an npm install script, so it works where
22
+ scripts are turned off (`--ignore-scripts`). To update:
23
+
24
+ ```bash
25
+ npm install -g @giamme/forge@latest && forge install
26
+ ```
27
+
28
+ The links point into npm's global package directory, so they keep working across updates;
29
+ `forge install` refreshes antigravity's copied plugin. Under nvm, that directory belongs to
30
+ one Node version: after switching Node, install forge on the new one and run `forge install`
31
+ again.
32
+
33
+ > **Foundry users:** Foundry's `forge` has the same name, and whichever directory comes first
34
+ > on your `PATH` wins; npm's global bin usually comes first. Runs start from your harness
35
+ > (`/forge`), so this command is only for `forge install` and the optional tools. If you keep
36
+ > Foundry's `forge` first, run ours as `"$(npm root -g)/@giamme/forge/forge"`.
37
+
38
+ ## From a git checkout
39
+
40
+ ```bash
41
+ git clone https://github.com/Giamme/forge.git ~/.claude/skills/forge
42
+ bash ~/.claude/skills/forge/scripts/forge-install.sh
43
+ ```
44
+
45
+ Clone anywhere you like — the installer works out its own location and links from there.
46
+ `~/.claude/skills/forge` is just the tidiest home: claude reads that path directly, and
47
+ opencode and grok scan it on their own.
48
+
49
+ The installer also offers optional Fractal support in an interactive terminal. Declining
50
+ or a Fractal installation failure does not prevent ordinary Forge setup. See
51
+ [Fractal installation](fractal.md#install-the-optional-runtime) for prerequisites and a separate install.
52
+
53
+ ## Check what forge can reach
54
+
55
+ `<forge>` is your checkout, or `"$(npm root -g)/@giamme/forge"` for an npm install:
56
+
57
+ ```bash
58
+ bash <forge>/scripts/forge-dispatch.sh doctor
59
+ ```
60
+
61
+ ```
62
+ HARNESS STATUS DETAIL
63
+ codex found /Users/you/.nvm/versions/node/v24.18.0/bin/codex
64
+ claude found /Users/you/.local/bin/claude
65
+ openclaude found /Users/you/.nvm/versions/node/v24.18.0/bin/openclaude
66
+ opencode found /Users/you/.opencode/bin/opencode
67
+ antigravity MISSING agy not on PATH
68
+ grok found /Users/you/.grok/bin/grok
69
+ ```
70
+
71
+ A `MISSING` harness is fine — forge only needs the ones you actually route models through.
72
+
73
+ **Binary presence is not reachability.** A harness can be installed and still fail on auth or
74
+ billing (opencode in particular fails this way), and that failure only surfaces mid-dispatch.
75
+ To prove a pairing end to end before spending a real run:
76
+
77
+ ```bash
78
+ bash <forge>/scripts/forge-dispatch.sh dwarf <spec> --prompt-file /dev/stdin --repo <scratch dir> \
79
+ <<< 'Reply with exactly: FORGE_OK'
80
+ ```
81
+
82
+ ## What the installer does per harness
83
+
84
+ Each CLI discovers skills differently, and the differences are not guessable — so verify
85
+ with the listed check rather than trusting file placement.
86
+
87
+ | Harness | Mechanism | Verify with |
88
+ |---|---|---|
89
+ | claude | `~/.claude/skills/forge/` | `/forge` appears in the skill list |
90
+ | openclaude | symlink into `~/.openclaude/skills/` | `openclaude skills list --json` |
91
+ | codex | symlink into **`~/.codex/skills/`** | `codex exec "do you have a skill named forge?"` |
92
+ | opencode | **nothing** — it scans `~/.claude/` itself | `opencode debug skill` |
93
+ | antigravity | `agy plugin install` of a generated wrapper | `agy --print='/forge'` |
94
+
95
+ > **The one ongoing caveat:** `agy plugin install` *copies* the files instead of referencing
96
+ > them. The other four track the source directory live, so an edit reaches them instantly —
97
+ > antigravity keeps running whatever it copied. **Re-run `forge-install.sh` after any edit**,
98
+ > or agy silently runs a stale version.
99
+
100
+ `forge-install.sh --dry-run` prints what it would do and touches nothing. It never clobbers
101
+ a real directory that isn't a symlink.
102
+
103
+ ## Updating all harness installations
104
+
105
+ Run the updater from your Forge checkout (or through one of its skill symlinks):
106
+
107
+ ```bash
108
+ ./forge update # same as: bash ~/.claude/skills/forge/scripts/forge-update.sh
109
+ ./forge version # VERSION, checkout commit, tracked upstream, last known release
110
+ ./forge version --check # ask the upstream for its newest release now
111
+ ```
112
+
113
+ It fetches the current branch's configured upstream, fast-forwards the checkout, and
114
+ runs Forge's installer to refresh Claude, OpenClaude and Codex links, OpenCode discovery,
115
+ and Antigravity's copied plugin. Harnesses that are not present are skipped. Restart
116
+ active harness sessions afterward to load the updated skill.
117
+
118
+ ```bash
119
+ # Preview without fetching or changing files:
120
+ bash <forge-checkout>/scripts/forge-update.sh --dry-run
121
+
122
+ # Refresh harness installations after local edits, without fetching:
123
+ bash <forge-checkout>/scripts/forge-update.sh --local
124
+
125
+ # Point a copied installation's updater at the original checkout:
126
+ bash <path>/scripts/forge-update.sh --source <forge-checkout>
127
+ ```
128
+
129
+ When a solo run, `plan` or `run` finishes, forge checks the checkout's upstream for a newer
130
+ release tag at most once a day (`git ls-remote`, 5-second timeout, cached in
131
+ `${XDG_STATE_HOME:-~/.local/state}/forge/update-check.json`) and, if there is one, prints a
132
+ single line on stderr: `forge 0.8 is available (you have 0.7.1): forge update`. The check is
133
+ silent offline, in a copied installation without Git, in CI (`CI` set) and with
134
+ `FORGE_UPDATE_CHECK=off`, and it never changes a run's outcome or exit code.
135
+
136
+ An npm installation has no Git upstream, so there the same daily check asks the npm registry
137
+ for the package's `latest` (`npm_config_registry` when you set one, as npm does), and the line
138
+ names npm's command instead: `forge 0.9.0 is available (you have 0.8): npm install -g
139
+ @giamme/forge@latest && forge install`. `forge version` shows the same command.
140
+
141
+ The default update requires a clean Git checkout and a branch with an upstream. Local
142
+ changes, detached HEAD, fetch failures and diverged branches stop the update without
143
+ resetting or stashing your work. `--local` deliberately allows current local edits.
144
+ Real skill directories are preserved; a conflicting directory or plugin installation
145
+ failure produces a nonzero exit and an explanation. After fixing an installation
146
+ failure, use `--local` to retry from the updated source. Updating Forge does not prompt
147
+ for optional Ripwire installation.
148
+
149
+ An npm installation is not a Git checkout, so the updater stops there and prints the npm
150
+ command [above](#from-npm) instead. `--local` still refreshes its harness installations.
151
+
152
+ ## `scripts/forge-install.sh`
153
+
154
+ ```
155
+ forge-install.sh [--dry-run]
156
+ ```
157
+
158
+ Links this directory into every installed harness. Skips harnesses that aren't present,
159
+ skips a destination that exists and isn't a symlink (it will never clobber a directory you
160
+ may have edited in place), and reports opencode as auto-detected.
package/docs/memory.md ADDED
@@ -0,0 +1,156 @@
1
+ # Project memory
2
+
3
+ [← README](../README.md) · [All docs](README.md)
4
+
5
+ What forge learns about a repository, where it keeps it, what each role is told, and what forge has spent there.
6
+
7
+ Every forge dispatch is a clean slate, so without help the tenth run in a repo rediscovers
8
+ what the first one learned. forge keeps that in the repo, in two files:
9
+
10
+ ```
11
+ .forge/memory.md small, capped, injected into every prompt
12
+ .forge/ledger.tsv append-only, one row per dispatch, never injected
13
+ ```
14
+
15
+ The split is the design. Injected context is paid for on every dispatch of every future run,
16
+ so "remember everything" and "stay small" pull against each other — the ledger remembers at
17
+ zero context cost and is read only when asked, while `memory.md` is capped at 40 lines / 4 KB.
18
+
19
+ ```markdown
20
+ ## Verify
21
+ - pytest -q runs the suite; make test also lints and is 4x slower
22
+
23
+ ## Known traps
24
+ - tests/test_api.py::test_timeout is flaky under -n auto [tests/test_api.py]
25
+
26
+ ## Recurring QA findings
27
+ - 3x: route added without updating openapi.yaml [src/routes]
28
+ ```
29
+
30
+ ## How a fact gets in
31
+
32
+ The role that just did the work writes it down, in its final message, next to the verdict:
33
+
34
+ ```
35
+ FORGE_LEARNING: trap | src/proto/*_pb2.py is generated, never hand-edit [src/proto]
36
+ ```
37
+
38
+ No extra dispatch and no separate model guessing what mattered — the one that hit the
39
+ landmine is the one that records it. Everything after that is deterministic bash.
40
+
41
+ **What stops it becoming a junk drawer:**
42
+
43
+ | Guard | Effect |
44
+ |---|---|
45
+ | recurrence threshold | a `finding` needs **two distinct runs** before it is injected. One occurrence is an incident, not a pattern |
46
+ | stale anchors | an entry ending in `[path]` is dropped once that path stops existing, so memory stays true as code moves |
47
+ | hard cap | over 40 lines / 4 KB, lowest-count and least-recent facts are trimmed; one entry is clamped to 180 chars |
48
+ | bias to silence | the prompt says most runs teach nothing durable and emitting nothing is correct. A false fact costs every future run; a missing one costs a single rediscovery |
49
+
50
+ Only `verify` and `trap` are kept on first sighting — those are facts, true the first time.
51
+
52
+ ## Slices, and what forge writes
53
+
54
+ | Role | Gets |
55
+ |---|---|
56
+ | planner, dwarf | Verify + Known traps + Recurring findings |
57
+ | qa | Known traps + Recurring findings |
58
+
59
+ QA is not told the build command; it is not building anything. It *is* told the recurring
60
+ findings, so it can notice this is the fourth time the same bug shipped.
61
+
62
+ `.forge/` is the **only** thing forge writes into your working tree rather than onto a branch
63
+ of its own. It is excluded from every captured diff, so a reviewer never sees it as the
64
+ dwarf's work — but it *will* show up as untracked in `git status` after your first run.
65
+
66
+ ## What to do with `.forge/` when it appears
67
+
68
+ forge never commits it. That decision is yours, and there are three sane answers:
69
+
70
+ | You want | Do this |
71
+ |---|---|
72
+ | memory shared with your team, and with future you | **commit `.forge/`** — the intended case; every teammate's runs then build on the same facts |
73
+ | memory kept to your machine | `echo '.forge/' >> .gitignore` |
74
+ | no memory at all | `--no-memory`, or `FORGE_MEMORY=off` in your environment |
75
+
76
+ Two things that are easy to get wrong:
77
+
78
+ **Gitignoring is not the same as turning it off.** An ignored `.forge/` is still written and
79
+ still injected into every prompt — it just stops travelling. Only `--no-memory` /
80
+ `FORGE_MEMORY=off` actually disables the feature.
81
+
82
+ **Never commit one file and ignore the other.** `memory.md` is *rebuilt from* `ledger.tsv` on
83
+ every write, so a clone that has the memory but not the ledger loses everything the moment
84
+ anyone runs forge there — the rebuild regenerates `memory.md` from that machine's own short
85
+ ledger and the shared facts are gone, silently. Verified behaviour, not a theoretical risk:
86
+
87
+ ```
88
+ teammate A commits memory.md only, ledger.tsv gitignored
89
+ memory.md: - pytest -q runs the suite
90
+ - src/a.py is generated [src/a.py]
91
+
92
+ teammate B clones, runs forge once, learns one unrelated thing
93
+ memory.md: - b learned something new ← A's two facts destroyed
94
+ ```
95
+
96
+ Commit both, or neither. If you commit them, expect `ledger.tsv` to gain a line per dispatch
97
+ — small, but it will appear in your diffs; that growth is the price of the recurrence
98
+ counting that keeps `memory.md` honest.
99
+
100
+ If you contribute to someone else's repo, gitignore it. A pull request containing forge's
101
+ notes to itself is noise to a reviewer who does not use forge.
102
+
103
+ ## `scripts/forge-memory.sh`
104
+
105
+ ```
106
+ forge-memory.sh inject <repo> <planner|dwarf|qa> # the block to prepend to a prompt
107
+ forge-memory.sh note <planner|dwarf|qa> # the instruction to append
108
+ forge-memory.sh record <repo> --last <file> --role <r> [--run-id X] [--task T]
109
+ [--model M] [--verdict V] [--duration S]
110
+ forge-memory.sh show <repo>
111
+ forge-memory.sh prune <repo> # re-apply staleness and the cap now
112
+ forge-memory.sh spend <repo> [--run <run-id>] # what forge has cost this repo
113
+ ```
114
+
115
+ `record` appends to the ledger and then rebuilds `memory.md` from the whole ledger — the
116
+ rebuild is idempotent, so a crashed run can leave the ledger a row short but can never leave
117
+ memory half-written. Editing `memory.md` by hand therefore does not stick; remove the ledger
118
+ rows instead.
119
+
120
+ ```bash
121
+ # what would a dwarf actually be told about this repo?
122
+ bash scripts/forge-memory.sh inject ~/dev/api dwarf
123
+
124
+ # start over
125
+ rm -rf ~/dev/api/.forge
126
+ ```
127
+
128
+ ## Spend
129
+
130
+ The ledger already holds a row per dispatch and is never injected into a prompt, so
131
+ accounting from it costs nothing:
132
+
133
+ ```bash
134
+ bash scripts/forge-memory.sh spend ~/dev/api
135
+ ```
136
+ ```
137
+ 14 dispatch(es) across 3 run(s), 1h12m of model time
138
+ Durations sum concurrent work, so they exceed the wall-clock of a parallel run.
139
+
140
+ by model disp total mean
141
+ sol:xhigh 4 38m20s 9m35s
142
+ opus 7 22m10s 3m10s
143
+ gemini:low:antigravity 3 11m30s 3m50s
144
+
145
+ by role disp total
146
+ dwarf 7 49m50s
147
+ qa 7 22m10s
148
+ ```
149
+
150
+ **Time and dispatch counts, never tokens or money.** The six CLIs expose usage differently
151
+ or not at all, and a number forge cannot actually measure would be worse than no number.
152
+
153
+ `duration_s` is the ledger's tenth column, appended at the end on purpose: a ledger written
154
+ by an older forge — possibly already committed and shared — has nine fields, and awk yields
155
+ an empty string for the missing one instead of shifting every column. Those rows are counted
156
+ but not timed, and the output says so.
package/docs/models.md ADDED
@@ -0,0 +1,118 @@
1
+ # Models, efforts and harnesses
2
+
3
+ [← README](../README.md) · [All docs](README.md)
4
+
5
+ How forge names a model (`alias:effort:harness`), what each harness offers, how effort is validated and clamped, when to drop the sandbox, and how to teach forge a new model.
6
+
7
+ ## The spec: `alias:effort:harness`
8
+
9
+ Everywhere forge takes a model, it takes the same three-field spec. Each field after the
10
+ first is optional:
11
+
12
+ | Spec | Resolves to |
13
+ |---|---|
14
+ | `sol` | alias `sol`, default effort for the role, its default harness (codex) |
15
+ | `sol:xhigh` | `sol` at `xhigh` effort on its default harness |
16
+ | `sol:xhigh:openclaude` | same model and effort, run through the `openclaude` CLI |
17
+ | `opus::claude` | explicit harness, effort left to the role default — the middle field may be empty |
18
+ | `gpt-5.6-sol::codex` | not in the registry? passed through as a literal model id — but only with an explicit harness |
19
+
20
+ An alias with no registry row still works, provided you name the harness. forge won't guess
21
+ a default harness for a model it has never seen; that guess would silently spend quota on
22
+ the wrong provider.
23
+
24
+ **Role defaults when you leave things out:**
25
+
26
+ - `--dwarf` missing → forge **asks**. Every dispatch spends real quota on your account, so
27
+ picking for you isn't its call.
28
+ - `--qa` missing → `opus`, which runs locally through `claude` and adds no cost surprise.
29
+ - effort missing → dwarf builds at `medium`, qa reviews at `xhigh`. A reviewer thinking
30
+ *less* than the implementer tends to wave things through, which wastes the stage entirely.
31
+
32
+ ## Effort, ceilings and clamping
33
+
34
+ Every harness has its own effort ladder:
35
+
36
+ | Harness | Ladder |
37
+ |---|---|
38
+ | codex | `low medium high xhigh max ultra` |
39
+ | claude | `low medium high xhigh max` |
40
+ | openclaude | `low medium high xhigh max ultracode` |
41
+ | antigravity | `low medium high` |
42
+ | grok | `low medium high xhigh max` (grok-4.7 offers up to `xhigh`) |
43
+ | opencode | provider-specific; forwarded verbatim |
44
+
45
+ And support is more irregular than one ceiling per backend:
46
+
47
+ - **Ceilings** — `luna` tops out at `max` on codex, so `luna:ultra` clamps down.
48
+ - **Gaps** — `gemini-pro` offers `low` and `high` but *not* `medium`, so `gemini-pro:medium`
49
+ clamps to `low`.
50
+ - **None at all** — Claude models on antigravity take no effort flag; passing one errors, so
51
+ forge reports it and drops it.
52
+
53
+ When forge clamps, it says so (`clamped=ultra -> max`) and the orchestrator passes that on
54
+ to you. Getting less thinking than you paid for, silently, is exactly the failure worth ten
55
+ lines of validation to prevent — and it's a real one: **codex accepts an invalid effort
56
+ without complaint and quietly runs at some other level.** forge validates first.
57
+
58
+ A word that isn't a recognized effort at all is an error, not a clamp.
59
+
60
+ ## Yolo mode
61
+
62
+ `--yolo-dwarf` and `--yolo-qa` drop that role's sandbox and approval gates:
63
+
64
+ | Harness | Flag used |
65
+ |---|---|
66
+ | codex | `--dangerously-bypass-approvals-and-sandbox` |
67
+ | claude / openclaude / antigravity | `--dangerously-skip-permissions` |
68
+ | opencode | `--auto` |
69
+ | grok | `--always-approve` |
70
+
71
+ Each flag applies to **one role only**, so `--yolo-dwarf` alone is the normal combination:
72
+ the implementer gets a free hand, the reviewer stays fenced in and still can't edit the code
73
+ it's judging.
74
+
75
+ You mostly don't need it. On codex, claude and openclaude the sandboxed roles are already
76
+ useful — the dwarf edits files and runs tests, qa reads and runs commands but cannot write.
77
+ Reach for yolo when a role genuinely must escape the sandbox: installing dependencies,
78
+ writing outside the repo, touching the network.
79
+
80
+ > **Antigravity is the exception worth knowing.** Without yolo it cannot run *any* shell
81
+ > command, and a single denied command **aborts the whole run** rather than degrading — a
82
+ > dwarf that tries to run the test suite ends with an error and an empty diff. forge detects
83
+ > this and appends a note telling that role not to attempt commands, which is why an
84
+ > antigravity dwarf reports its work as unverified by execution. If you want a Gemini dwarf
85
+ > that can actually run the tests it writes, it needs `--yolo-dwarf`.
86
+
87
+ ## Model registry
88
+
89
+ `registry.tsv` is the single source of truth for alias resolution. Four tab-separated
90
+ columns, plus an optional fifth:
91
+
92
+ ```
93
+ # alias harness model effort [timeout]
94
+ sol codex gpt-6.1-sol <=ultra
95
+ sol openclaude gpt-6.1-sol <=max
96
+ gemini-pro antigravity gemini-3.1-pro low,high
97
+ opus antigravity claude-opus-4-6-thinking none
98
+ grok grok grok-4.7 <=xhigh
99
+ grok opencode github-copilot/grok-4.6 -
100
+ ```
101
+
102
+ | `effort` form | Means |
103
+ |---|---|
104
+ | `<=X` | any effort on that harness's ladder up to and including `X` |
105
+ | `a,b,c` | exactly these efforts — gaps are real, and this is how they're expressed |
106
+ | `none` | takes no effort flag at all; passing one errors |
107
+ | `-` | no validated ladder; forwarded verbatim |
108
+
109
+ The optional fifth column, `timeout`, is a per-model default dispatch limit in seconds (`-`
110
+ or absent = the built-in default). It applies only when neither `--timeout` nor
111
+ `FORGE_TIMEOUT` was given and is scaled by the resolved effort (see
112
+ [timeout precedence](commands.md#scriptsforge-dispatchsh)). Rows without it behave exactly as before.
113
+
114
+ **Row order is meaningful:** the *first* row for an alias is that alias's default harness.
115
+ `--dwarf sol` means sol on codex; `--dwarf sol:high:openclaude` moves the same model
116
+ elsewhere.
117
+
118
+ Adding a model is one line in this file. Nothing else needs to change.
@@ -0,0 +1,231 @@
1
+ # Reviews, retries and recovery
2
+
3
+ [← README](../README.md) · [All docs](README.md)
4
+
5
+ What a reviewer must answer, what forge does when it fails a task, how to retry, tolerate or accept findings, and the checks forge runs around the reviewer.
6
+
7
+ ## Verdicts and failure
8
+
9
+ Decompose-mode QA must end with a sentinel line:
10
+
11
+ ```
12
+ FORGE_VERDICT: PASS no confirmed correctness bug (style nits are not failures)
13
+ FORGE_VERDICT: FAIL at least one CONFIRMED correctness bug
14
+ ```
15
+
16
+ Only a standalone final line counts, and the parser is shared by every runner (solo,
17
+ decomposed, Fractal's QA checkpoint). The prompt shows each verdict with a dash and its
18
+ meaning, and some reviewers copy that whole line, so `FORGE_VERDICT: PASS — meaning` is
19
+ accepted, as are trailing whitespace and CRLF line endings. Prose or a quoted marker earlier
20
+ in the reply is never a verdict. A missing verdict is `UNKNOWN` and is treated exactly like a
21
+ failure — excluded from integration and flagged. Merging a diff whose reviewer never reached
22
+ a conclusion would defeat the point of reviewing it, so the ambiguous case fails safe.
23
+
24
+ A failing task is excluded and **its branch and worktree are preserved**; the rest of the run
25
+ proceeds. forge never loops a dwarf against its reviewer on its own: a retry happens when you
26
+ type `retry`, or when you opt in for a run with `--retry-failed N` ([below](#retrying-inside-a-run---retry-failed-n)).
27
+
28
+ An `UNKNOWN` task is different from a rejected one: the reviewer never judged the code, so
29
+ `retry` (or `--retry-failed`) asks **QA again on the unchanged work** instead of paying an
30
+ implementer to rewrite something nobody rejected. No attempt is spent and no dwarf runs. If
31
+ the worktree no longer matches what was reviewed, the full task runs instead.
32
+
33
+ ## Retrying a failed task
34
+
35
+ ```bash
36
+ # re-run one task with the findings that failed it, on a stronger model
37
+ bash scripts/forge-parallel.sh retry "$PLAN" middleware --dwarf sol:xhigh
38
+ ```
39
+
40
+ The findings are the point. `retry` prepends the previous reviewer's report to the dwarf's
41
+ prompt — "your last attempt was reviewed and these were the findings; fix them" — and reuses
42
+ the worktree, so the dwarf continues from its own code rather than starting over. If the
43
+ verdict comes back `PASS` the task merges like any other.
44
+
45
+ `--dwarf` is written back into `tasks.tsv`, so the table shows the model that will actually
46
+ be spent and the escalation survives a re-plan.
47
+
48
+ Two things it deliberately does:
49
+
50
+ - **Reviews the task's cumulative diff**, against the commit the task was originally branched
51
+ from — not just the fix. Reviewing the fix alone would let the first attempt's code through
52
+ unread.
53
+ - **Refuses to re-review identical code.** If the retry produces a byte-identical diff, the
54
+ task fails again without spending a QA dispatch.
55
+
56
+ `retry` is something you run. Forge never loops a dwarf against its own reviewer on its own
57
+ initiative — unless you ask a *run* to with `--retry-failed N`, see
58
+ [retrying inside a run](#retrying-inside-a-run---retry-failed-n). A task whose reviewer
59
+ never gave a verdict (`UNKNOWN`) or whose run stopped on an infrastructure failure (`INFRA`)
60
+ resumes at the review stage instead: QA runs again on the unchanged work, no dwarf runs, and
61
+ no attempt is spent.
62
+
63
+ ## Retrying inside a run: `--retry-failed N`
64
+
65
+ ```bash
66
+ forge-parallel.sh run <plan-dir> --retry-failed 2
67
+ ```
68
+
69
+ A task whose reviewer says `FAIL` (or `UNKNOWN`) is re-queued at once, **inside the same
70
+ run**, up to `N` more times — with the reviewer's findings carried into the next prompt exactly
71
+ as `retry` does. Other tasks keep running meanwhile; dependents still wait for the task to
72
+ *merge*, so nothing downstream starts early. Without this, a failed task waited for the whole
73
+ run (verification included) to finish before a human could retry it.
74
+
75
+ It is opt-in and per invocation: it is not remembered by the plan, and the orchestrator never
76
+ adds it unprompted. Only reviewer rejections are retried automatically — `TIMEOUT`, `ERROR`,
77
+ `INVALIDATED` and `CONFLICT` still need a human. A retry that changes nothing (empty diff, or
78
+ a diff byte-identical to the previous attempt) writes a `noretry` marker and is **not**
79
+ retried again, so a stuck task cannot burn its whole budget repeating itself.
80
+ `tasks/<id>/attempts.tsv` records every entry (epoch, attempt number, resulting status, infra
81
+ class).
82
+
83
+ ## Infrastructure failures are not task failures
84
+
85
+ A quota or usage limit, an auth problem, a rate limit, a network error, or a harness that
86
+ returns nothing at all says nothing about the work. forge classifies those from the dispatch
87
+ log (`forge-runtime.py classify`) and `forge-dispatch.sh` exits **8** instead of 4. A
88
+ decomposed task then ends as **`INFRA`**, not `ERROR`/`FAIL`, and:
89
+
90
+ - **no attempt is spent** — a dwarf-stage stop refunds the counter; a QA-stage stop keeps the
91
+ dwarf's committed work and re-runs **QA only** next time (`tasks/<id>/resume`, revalidated
92
+ against the branch tip and fingerprint before it is trusted);
93
+ - the scheduler stops dispatching, lets running tasks drain, leaves unstarted ones `PENDING`
94
+ (never `BLOCKED`) and `run` exits **8**: fix the cause, then run the same command again;
95
+ - `run --infra-retries M` waits instead — up to `M` times, using the provider's retry-after
96
+ hint when it gave one (capped at 6 hours) and otherwise 60 s doubling to 30 min
97
+ (`FORGE_INFRA_BACKOFF` sets the base) — and resumes in the same run;
98
+ - verification is skipped for a paused run (it is unfinished); the next `run` verifies at its end.
99
+
100
+ `.infra` (`dwarf.infra`/`qa.infra`) holds `class=`, `retry_after=`, `detail=`, `rc=`,
101
+ `role=`. Detection is deliberately strict: strong phrases only, and ordinary text that merely
102
+ mentions "rate limit" in a reply that otherwise succeeded is never classified. A false
103
+ positive costs a pause, not a spent attempt. Under Fractal a dwarf-side infrastructure stop
104
+ still collapses to Fractal's own "failed" handling; QA-side stops propagate as above.
105
+
106
+ ## Severity, thresholds and known issues
107
+
108
+ QA is asked to label every finding `- [P0..P3][CONFIRMED|PLAUSIBLE] file:line — what breaks`
109
+ (P0 data loss/security/crash on the main path, P1 wrong behaviour on realistic input, P2
110
+ edge-case bug, P3 minor). By default the verdict rule is unchanged — any CONFIRMED bug fails.
111
+
112
+ `--qa-threshold Pn` (on `run` or `retry`; remembered in the plan; `none` clears it) names the
113
+ **least severe level that still blocks**: `P2` means P0–P2 block and only P3 is tolerated.
114
+ A task-specific `tasks/<id>/qa_threshold` file beats the plan-level value, which is how a
115
+ dev-tooling or investigation task gets a looser bar than production code. With a threshold, a
116
+ reviewer `FAIL` made only of CONFIRMED findings that are all labelled and strictly milder than
117
+ the threshold is accepted as **PASS with known issues**: the task merges, `tasks/<id>/known_issues.md`
118
+ keeps the findings, `qa.gate` records the decision, and every run/retry/integrate summary
119
+ lists them. The literal status stays `PASS`, so nothing downstream needed a new vocabulary.
120
+ It fails safe: an unlabelled CONFIRMED finding, or a FAIL with no CONFIRMED finding at all,
121
+ is never tolerated, and without a threshold nothing changes.
122
+
123
+ ## Accepting a task despite its reviewer
124
+
125
+ ```bash
126
+ bash scripts/forge-parallel.sh accept "$PLAN" tooling \
127
+ --reason "only an unreachable option left; tracked in the backlog" --approved
128
+ ```
129
+
130
+ When a reviewer keeps finding new corners of something that should ship (one real
131
+ investigation harness got 12 findings over five rounds, and only a human decision ended it),
132
+ `accept` is that decision made explicit instead of done with git plumbing. It needs **both**
133
+ `--reason` and `--approved`, because it overrides a reviewer; an orchestrating model never runs
134
+ it on its own. It accepts only a `FAIL` or `UNKNOWN` task whose reviewed work is **intact** —
135
+ the branch is still at the reviewed commit and the worktree still has the saved fingerprint —
136
+ otherwise it exits `3` and says to `retry` for a fresh review (accepting would sign off on code
137
+ nobody reviewed). It writes `tasks/<id>/known_issues.md` (your reason, who and when, and the
138
+ reviewer's reply copied verbatim) and `accepted.by`, merges through the same checks as any other
139
+ task, and puts everything back if the merge conflicts (exit `6`). Dependents are not touched;
140
+ `run` continues them.
141
+
142
+ ## Whole-run review (opt-in)
143
+
144
+ Per-task QA only sees one slice. In one real run, a holistic review of the combined result still
145
+ found two confirmed P2 defects in code every per-task review had passed. It is opt-in because it
146
+ spends QA quota and `integrate` made no model calls before:
147
+
148
+ ```bash
149
+ bash scripts/forge-parallel.sh review "$PLAN" --qa opus # read-only look at the combined result
150
+ bash scripts/forge-parallel.sh integrate "$PLAN" --approved --final-review opus # and gate the merge on it
151
+ ```
152
+
153
+ The reviewer gets the overall goal, every task's complete requirements (bounded to about 80 KB,
154
+ truncated visibly), what forge observed that no per-task reviewer saw (accepted known issues,
155
+ scope drift, guard notes, the verification result including a `FLAKY` pass and coverage
156
+ warnings), the combined diff (inline up to 400 KB), and the same labelled-findings contract —
157
+ with an emphasis on cross-task interactions, interface mismatches and ordering races. `review`
158
+ reads `integration.base..integration tip` from a disposable checkout, writes
159
+ `<plan>/final-review-<epoch>-<pid>/` (`qa.input`, `qa.last`, `verdict`, `qa.gate`, …), keeps
160
+ `<plan>/final-review` and `final-review.verdict` pointing at the newest, and never touches your
161
+ branch or the integration branch. With `integrate --final-review`, it runs after verification
162
+ and before your branch can move: `PASS` (or a `FAIL` accepted by `--qa-threshold`) continues,
163
+ anything else stops with exit `5`; an infrastructure stop exits `8`. The QA spec is `--qa`, else
164
+ the one remembered in `final_qa`, else the single spec every task shares (otherwise it asks).
165
+ Tasks that are not merged are not in the result, and it says so. `integrate` also names unmerged
166
+ tasks even without a review.
167
+
168
+ ## Shadow reviews: how much of a verdict is the reviewer?
169
+
170
+ `plan --shadow-qa <spec>[,<spec>...] [--shadow-rate R]` (default R = 0.1) sends a sampled
171
+ share of reviewed attempts to a second reviewer, the first listed one of another model
172
+ family than the task's. Its verdict goes to the run dataset and nothing in the run reads it,
173
+ so it never changes a status, merge, memory or retry. `forge dataset status` then reports how
174
+ often the two reviewers agree, overall and per model pair. The table shows the extra reviews
175
+ before you approve. The shadow runs beside the main review and is stopped soon after that
176
+ finishes, so the most it costs is reviews and a little latency. See
177
+ [decomposition](../references/decompose.md#shadow-reviews-opt-in).
178
+
179
+ ## Implementer guards
180
+
181
+ The dwarf is told not to commit, switch branches, start background processes, or claim checks
182
+ it did not run. forge then **observes** whether it did, rather than trusting its report:
183
+
184
+ - **self-commit / branch switch** — HEAD or the checked-out ref moved during the dispatch;
185
+ - **orphan processes** — the harness runs in its own session; anything still alive in it after
186
+ the harness exits is listed in `<role>.orphans` and stopped (`FORGE_ORPHAN_GUARD=off`
187
+ disables this; it is also skipped under Fractal);
188
+ - **promised later work** — "running in the background", "will report back once it
189
+ finishes" and similar in the final message.
190
+
191
+ These are warnings recorded in `tasks/<id>/guard.txt`, shown in the QA prompt ("Implementer
192
+ guard notes") and in the run summary. They never change a status: the diff and the reviewer
193
+ still decide. In solo mode a self-commit prints the `git reset --soft <old HEAD>` that would
194
+ undo it; forge never resets anything itself.
195
+
196
+ ## Verification coverage and flaky reruns
197
+
198
+ `--verify` is whatever you hand it, and a gate that skips a suite the project defines lets a
199
+ regression there reach your branch silently. `forge-verify-coverage.py` reads the project's own
200
+ test configuration (`package.json` scripts, Makefile targets, pytest/tox, Cargo, Go, Ruby,
201
+ Elixir, `scripts/test*.sh`), follows aggregate scripts, and **warns** — before any model is
202
+ paid for, again in the run summary and at integrate — about suites the verify command never
203
+ runs, e.g. `suite 'test:contract' (npm run test:contract) is not run by the verify command`.
204
+ It is advisory: it never fails a run, and `verification.coverage.txt` keeps the lines. It is a
205
+ text heuristic (root of the repo only; a mentioned command counts as run).
206
+
207
+ A failing verify command is rerun **once** (`--verify-retries N`; `0` restores the single strict
208
+ run) when the source fingerprint is unchanged. If the rerun passes the status is `PASS` but
209
+ `verification.flaky` is written, `verification.log` keeps the first (failing) run,
210
+ `verification.retry.log` the second, and `run`/`integrate` print a loud `FLAKY` line — an
211
+ intermittent failure can be a real bug, so it is never passed off as a clean pass. A command
212
+ that changes the tree is never retried. This deterministic rerun comes before Jev's optional
213
+ model-assisted triage, which then does not run.
214
+
215
+ ## Native review
216
+
217
+ `--native-review` switches a codex reviewer to its purpose-built reviewer, for a diff too large
218
+ to inline. Know the trade: codex refuses a custom prompt alongside its scope flags, so the
219
+ reviewer never learns what the dwarf was *asked* to do. It can still judge whether the code is
220
+ correct, but not whether it's the right change, so its result is informational and the verdict
221
+ is `UNKNOWN`. See [solo runs](../references/solo.md) for how the baseline is passed.
222
+
223
+ ## Fail-first evidence (opt-in)
224
+
225
+ `--fail-first` (on a solo run or `plan`) checks that the tests a change adds or edits can
226
+ actually fail. Before each review, forge runs each new or edited test file against the change
227
+ and against the base without it, using the per-file command in `.forge/verify-subset`, and hands
228
+ the reviewer the result: `proven`, `passes_without_change`, `inconclusive` or `error`. The
229
+ reviewer is asked for a P2 finding for every test that passes without the change. It is advice
230
+ only: forge never changes a status or verdict from it. forge-ui shows the evidence under the
231
+ review findings. See [fail-first evidence](../references/decompose.md#fail-first-evidence-opt-in).