@giamme/forge 0.9.0 → 0.11.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +56 -0
- package/SKILL.md +22 -3
- package/VERSION +1 -1
- package/docs/artifacts.md +28 -2
- package/docs/commands.md +55 -0
- package/docs/dataset.md +22 -0
- package/docs/decomposed-runs.md +54 -2
- package/docs/forge-ui.md +48 -7
- package/docs/memory.md +4 -2
- package/docs/reviews.md +87 -2
- package/forge +31 -1
- package/package.json +1 -1
- package/references/dataset.md +103 -4
- package/references/decompose.md +41 -5
- package/references/harnesses.md +43 -1
- package/references/ui.md +31 -6
- package/scripts/forge-config.py +350 -0
- package/scripts/forge-control.py +649 -0
- package/scripts/forge-dispatch.sh +235 -121
- package/scripts/forge-issues.py +1134 -0
- package/scripts/forge-lib-plans.sh +6 -2
- package/scripts/forge-lib-review.sh +158 -3
- package/scripts/forge-lib-task.sh +358 -41
- package/scripts/forge-parallel-options.sh +3 -1
- package/scripts/forge-parallel.sh +99 -13
- package/scripts/forge-prompt.py +61 -4
- package/scripts/forge-route.py +245 -28
- package/scripts/forge-runs-rm.py +840 -0
- package/scripts/forge-runs.py +19 -1
- package/scripts/forge-runtime.py +47 -1
- package/scripts/forge-schedule.py +227 -11
- package/scripts/forge-ui.py +85 -19
- package/scripts/forge_dataset/__init__.py +84 -8
- package/scripts/forge_dataset/capture.py +5 -2
- package/scripts/forge_dataset/cli.py +44 -1
- package/scripts/forge_dataset/export.py +36 -3
- package/scripts/forge_dataset/forecast.py +355 -0
- package/scripts/forge_dataset/harvest.py +45 -2
- package/scripts/forge_dataset/importer.py +431 -17
- package/scripts/forge_dataset/reviewers.py +145 -0
- package/scripts/forge_dataset/routing.py +1 -0
- package/scripts/forge_jev/__init__.py +7 -3
- package/scripts/forge_registry.py +183 -26
- package/ui/README.md +41 -7
- package/ui/docs/api.md +259 -21
- package/ui/forge_ui/__main__.py +10 -2
- package/ui/forge_ui/control.py +587 -0
- package/ui/forge_ui/export.py +17 -3
- package/ui/forge_ui/reader.py +210 -12
- package/ui/forge_ui/server.py +125 -4
- package/ui/forge_ui/static/app.js +104 -18
- package/ui/forge_ui/static/cloud3d.js +4 -1
- package/ui/forge_ui/static/cloudmodel.js +1 -1
- package/ui/forge_ui/static/cloudui.js +13 -4
- package/ui/forge_ui/static/controls.js +619 -0
- package/ui/forge_ui/static/findingsview.js +438 -0
- package/ui/forge_ui/static/graph.js +6 -5
- package/ui/forge_ui/static/index.html +7 -1
- package/ui/forge_ui/static/leaderboard.js +18 -0
- package/ui/forge_ui/static/modelstats.js +19 -1
- package/ui/forge_ui/static/panel.js +76 -14
- package/ui/forge_ui/static/rundelete.js +388 -0
- package/ui/forge_ui/static/runmodel.js +128 -13
- package/ui/forge_ui/static/scene3d.js +32 -15
- package/ui/forge_ui/static/settings.js +399 -0
- package/ui/forge_ui/static/style.css +184 -5
- package/ui/forge_ui/static/uikit.js +161 -0
package/forge
CHANGED
|
@@ -21,8 +21,11 @@ usage() {
|
|
|
21
21
|
' fractal opt-in Fractal execution: install, doctor, runs, status, tree, logs, open, report, ...' \
|
|
22
22
|
' jev optional Jev advice: setup, status, doctor, enable, disable, backtest' \
|
|
23
23
|
' dataset every run'"'"'s inputs and outcomes: status, dump, import, export, outcomes, ...' \
|
|
24
|
+
' issues file the review findings a run left open as GitHub issues (plain gh)' \
|
|
24
25
|
' ui forge-ui, the run viewer: open [DIR], status, stop' \
|
|
25
26
|
' runs this repo'"'"'s recent runs in the terminal: state, verdicts and run dirs' \
|
|
27
|
+
' control a plan'"'"'s live controls: show, set (QA threshold, retries, timeouts), pin, unpin' \
|
|
28
|
+
' config every config source: show, and set/unset the per-repo .forge files and Jev on/off' \
|
|
26
29
|
' update fast-forward this checkout and refresh every harness installation' \
|
|
27
30
|
' version the installed version, and whether a newer release is out' \
|
|
28
31
|
'' \
|
|
@@ -30,6 +33,18 @@ usage() {
|
|
|
30
33
|
'Runs start from your agent harness (/forge "<goal>" --dwarf <model>); see README.md.'
|
|
31
34
|
}
|
|
32
35
|
|
|
36
|
+
# suggest WORD — the listed command WORD most likely meant (a unique prefix or a close
|
|
37
|
+
# spelling), read from usage so the two never drift apart; prints nothing when none is close
|
|
38
|
+
# or python3 is missing. -I keeps a stray module in the current directory out of it.
|
|
39
|
+
suggest() {
|
|
40
|
+
usage | python3 -I -c 'import difflib, re, sys
|
|
41
|
+
word = sys.argv[1].lower()
|
|
42
|
+
names = re.findall(r"^ ([a-z]+) ", sys.stdin.read(), re.M)
|
|
43
|
+
prefixed = [name for name in names if len(word) >= 2 and name.startswith(word)]
|
|
44
|
+
hit = prefixed if len(prefixed) == 1 else difflib.get_close_matches(word, names, 1, 0.6)
|
|
45
|
+
print(hit[0] if hit else "")' "$1" 2>/dev/null
|
|
46
|
+
}
|
|
47
|
+
|
|
33
48
|
case "${1:-}" in
|
|
34
49
|
install)
|
|
35
50
|
shift
|
|
@@ -47,6 +62,10 @@ case "${1:-}" in
|
|
|
47
62
|
shift
|
|
48
63
|
exec python3 "$ROOT/scripts/forge-dataset.py" "$@"
|
|
49
64
|
;;
|
|
65
|
+
issues)
|
|
66
|
+
shift
|
|
67
|
+
exec python3 "$ROOT/scripts/forge-issues.py" "$@"
|
|
68
|
+
;;
|
|
50
69
|
ui)
|
|
51
70
|
shift
|
|
52
71
|
exec python3 "$ROOT/scripts/forge-ui.py" "$@"
|
|
@@ -55,6 +74,14 @@ case "${1:-}" in
|
|
|
55
74
|
shift
|
|
56
75
|
exec python3 "$ROOT/scripts/forge-runs.py" "$@"
|
|
57
76
|
;;
|
|
77
|
+
control)
|
|
78
|
+
shift
|
|
79
|
+
exec python3 "$ROOT/scripts/forge-control.py" "$@"
|
|
80
|
+
;;
|
|
81
|
+
config)
|
|
82
|
+
shift
|
|
83
|
+
exec python3 "$ROOT/scripts/forge-config.py" "$@"
|
|
84
|
+
;;
|
|
58
85
|
update)
|
|
59
86
|
shift
|
|
60
87
|
exec /bin/bash "$ROOT/scripts/forge-update.sh" "$@"
|
|
@@ -72,7 +99,10 @@ case "${1:-}" in
|
|
|
72
99
|
exit 2
|
|
73
100
|
;;
|
|
74
101
|
*)
|
|
75
|
-
printf 'forge: unknown command: %s\n
|
|
102
|
+
printf 'forge: unknown command: %s\n' "$1" >&2
|
|
103
|
+
GUESS="$(suggest "$1")"
|
|
104
|
+
[ -n "$GUESS" ] && printf 'Did you mean `forge %s`?\n' "$GUESS" >&2
|
|
105
|
+
printf '\n' >&2
|
|
76
106
|
usage >&2
|
|
77
107
|
exit 2
|
|
78
108
|
;;
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@giamme/forge",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.11.0",
|
|
4
4
|
"description": "Multi-model coding with a reviewer that reads the code: plan, build in parallel worktrees, review every real diff, merge only what passes.",
|
|
5
5
|
"keywords": ["claude-code", "codex", "opencode", "agent-skill", "code-review", "multi-model"],
|
|
6
6
|
"license": "MIT",
|
package/references/dataset.md
CHANGED
|
@@ -14,6 +14,8 @@ ${FORGE_DATA_HOME:-${XDG_DATA_HOME:-~/.local/share}/forge}/dataset/
|
|
|
14
14
|
errors.log capture failures
|
|
15
15
|
```
|
|
16
16
|
|
|
17
|
+
An empty or relative `XDG_DATA_HOME` counts as unset, as the XDG spec says.
|
|
18
|
+
|
|
17
19
|
Capture is on by default and costs a few milliseconds per execution. `FORGE_DATASET=off`
|
|
18
20
|
turns it off for one invocation.
|
|
19
21
|
|
|
@@ -28,20 +30,36 @@ time order: an attempt that PASSed, then a merge that CONFLICTed, then a human `
|
|
|
28
30
|
| kind | one per | what it holds |
|
|
29
31
|
|---|---|---|
|
|
30
32
|
| `start` | execution | a receipt written by `begin()`. A start with no matching end is an execution that was killed or is still running |
|
|
31
|
-
| `dispatch` | model call | role, alias, harness, **resolved** model, effort after clamping, timeout and its source, delivery (custom or native review), phase timings, exit, timeout, infra class, tokens, parsed verdict and findings, `FORGE_LEARNING` lines
|
|
33
|
+
| `dispatch` | model call | role, alias, harness, **resolved** model, effort after clamping, timeout and its source, delivery (custom or native review), phase timings, exit, timeout, infra class, tokens, parsed verdict and findings, `FORGE_LEARNING` lines, `session_id`, `session_continue`. Blobs: the delivered prompt and the final message |
|
|
32
34
|
| `attempt` | task attempt or solo run | status, attempt number (with refund and resume flags), trigger (`initial`, `auto`, `human`), dwarf and qa spec, declared vs changed files, numstat, drift, guard notes, gate line, patch key, the files this execution wrote and the ones it only carried. Blobs: the diff, retry findings, memory slices, capsule, Jev task files |
|
|
33
35
|
| `verification` / `verify_run` | verify, and each command execution | command and its source (flag, `.forge/verify`, Jev), status, flaky, Jev triage. One `verify_run` per execution, each with its own log |
|
|
34
|
-
| `plan` | `plan` | goal, original request, decompose level, exploratory routing decisions (`tasks[id].route`, with their odds), author (orchestrator or planner), per-task declared vs executed difficulty and who set it, pools, waves, Jev routing and coupling. Blobs: both task tables, prompts, approaches, excerpts of the declared files at plan time, the file inventory, `memory.md` |
|
|
36
|
+
| `plan` | `plan` | goal, original request, decompose level, exploratory routing decisions (`tasks[id].route`, with their odds), author (orchestrator or planner), per-task declared vs executed difficulty and who set it, pools, waves, Jev routing and coupling, options (including `reviewer_session` when that file is present). Blobs: both task tables, prompts, approaches, excerpts of the declared files at plan time, the file inventory, `memory.md` |
|
|
35
37
|
| `run_end` | `run` | results per task, the accepted integration commit, `memory.md` |
|
|
36
38
|
| `invocation` | user-facing `forge-parallel.sh` subcommand | argv, exit, duration, plus final review, accept, integrate and split/combine payloads |
|
|
37
39
|
| `event` | point event | `merge` (MERGED, CONFLICT with the files, INVALIDATED and why), `integrate_advanced`, `integrate_conflict`, `combine_conflict`, `sched_terminal`, `subset_run` (every outcome), `memory_record`, `memory_prune` |
|
|
38
|
-
| `event` `escalation` |
|
|
40
|
+
| `event` `escalation` | retry that changes the model | the model a retry escalated from and to, `after_attempt`, and `source` (`policy`, `retry` for an explicit `--dwarf`, or `pin`), recorded at the rewrite, before the retried attempt begins |
|
|
41
|
+
| `event` `control` | `forge control` change | a human's live change: `at`, `actor` (`cli`/`ui`), `action` (`set`, `pin`, `unpin`), `key`, `value`, `previous`; task events carry the task id. Also read back from `control.log` by `import` |
|
|
42
|
+
| `event` `run_delete_started` / `run_deleted` | `forge runs rm` | written strictly around a run's deletion: `path`, `run_id`, `archive`, and for `run_deleted` the bytes freed |
|
|
39
43
|
| `event` `shadow_review` | shadow review | how an opt-in second review ended (`done`, `invalid`, `overtime`, `orphaned`, ...), with the reviewed commit and both specs. Its dispatch is recorded with role `qa_shadow` |
|
|
40
44
|
| `jev` | Jev request | site, answers, usage, latency, mode. Blob: the **full request** (state and questions). Its original sha is `jev.jsonl`'s `key` |
|
|
41
45
|
| `fractal_step` / `fractal_node` | Fractal step, node state | decomposition decision, iteration, resolved model, cost, tokens, candidate diff |
|
|
42
46
|
| `ledger` | ledger row | imported `.forge/ledger.tsv` rows |
|
|
47
|
+
| `forecast` | forecast shown by `run` | the plan forecast printed before dispatch (see [plan forecast](#plan-forecast)), once per run and content. Also in the plan as `forecast.json` |
|
|
43
48
|
| `outcome` | integrated task, per horizon | written by `forge dataset outcomes`, never by a run: how many of the lines a task added survive N days after `integrate`, and what reverted it |
|
|
44
49
|
|
|
50
|
+
A dispatch carries `session_id` and `session_continue`. `session_id` comes from the attempt
|
|
51
|
+
dir's `<role>.session`, and from `metrics.json` when that file is absent. `session_continue`
|
|
52
|
+
is null when the attempt dir has no `<role>.continue`. That file is written only when the
|
|
53
|
+
dispatch was asked to continue a reviewer session: `mode=continued` is `continued`, and
|
|
54
|
+
`mode=failed`, a missing mode or any other mode is `failed`. The file's presence means
|
|
55
|
+
continuing was requested, so a failed continue that still printed a verdict is tagged too.
|
|
56
|
+
The `attempt` event's `resumed` flag is a different fact: a QA-only re-run after an
|
|
57
|
+
infrastructure stop (the `tasks/<id>/resume` marker).
|
|
58
|
+
|
|
59
|
+
`plan --reviewer-session` writes `$PLAN/reviewer_session` (`on`). The plan event records
|
|
60
|
+
it in `options`. The file joins the plan id only when it is present, so every plan already
|
|
61
|
+
in a store keeps its id, and turning the experiment on stores a new plan event.
|
|
62
|
+
|
|
45
63
|
Each blob reference carries `sha` (the stored bytes), `original_sha`, `n` (the original
|
|
46
64
|
size), `kept`, and `cut`. A cut keeps the head and the tail, because a QA prompt ends with
|
|
47
65
|
the diff and a log ends with the failure. Caps: prompt 1 MiB, diff 2 MiB, message 256 KiB,
|
|
@@ -115,6 +133,8 @@ forge still running keeps writing.
|
|
|
115
133
|
./forge dataset export [--slice S]... [--limit N] [--seed N] [--out DIR] # labeled decision cases
|
|
116
134
|
./forge dataset outcomes [--horizon-days 14] [--repo PATH] [--dry-run] # what integrated work became
|
|
117
135
|
./forge dataset routing [--repo PATH] [--since DAYS] [--json] # pass rate and runtime per tier and model
|
|
136
|
+
./forge dataset reviewers [--horizon-days 14] [--repo PATH] [--json] # how each reviewer's PASSed work held up
|
|
137
|
+
./forge dataset forecast <plan> [--repo PATH] [--json] # likely cost and first-try PASS rate per task
|
|
118
138
|
```
|
|
119
139
|
|
|
120
140
|
`status` runs a different completeness check for each mode, because each mode
|
|
@@ -183,6 +203,18 @@ finished on an untouched snapshot and reviewed the same commit counts. The prima
|
|
|
183
203
|
labels never change, because `qa_shadow` dispatches are not reviews of record.
|
|
184
204
|
`./forge dataset status` prints the agreement rate, overall and per model pair.
|
|
185
205
|
|
|
206
|
+
A dispatch with `session_continue` set is left out of every slice built from QA rows
|
|
207
|
+
(review cases, citation cases) and out of the shadow agreement pairs. That covers a
|
|
208
|
+
continue that succeeded and one that failed after printing a verdict. A fresh fallback
|
|
209
|
+
dispatch in the same round has `session_continue` null and stays in. The slices have no
|
|
210
|
+
switch that puts the continued reviews back, so a group never mixes the two. Agreement
|
|
211
|
+
still selects the attempt's last verdict-bearing primary, and then drops the pair when
|
|
212
|
+
that primary continued a session: an earlier fresh verdict does not stand in for it.
|
|
213
|
+
`./forge dataset status` prints the number of continued reviews and the number of failed
|
|
214
|
+
continues on their own lines whenever either is non-zero, including in a store with no
|
|
215
|
+
shadow review. The experiment's comparison reads the `dispatch` events (`session_continue`,
|
|
216
|
+
tokens, verdicts, attempts), not these slices.
|
|
217
|
+
|
|
186
218
|
`meta.jsonl` records each case's slice, run, repo and label source. It also holds hosted
|
|
187
219
|
Jev's own answers where forge recorded them, as baselines that are never part of a case.
|
|
188
220
|
Routing and retry cases add:
|
|
@@ -278,6 +310,56 @@ exporter attaches it to that diff's review.
|
|
|
278
310
|
Each verification event also records the commit it verified (`verification.commit`, named
|
|
279
311
|
before any early return), so a FAIL or UNVERIFIED result can be joined to its code too.
|
|
280
312
|
|
|
313
|
+
## Reviewer report card
|
|
314
|
+
|
|
315
|
+
`forge dataset reviewers` joins the survival outcomes above to the reviews that let the work
|
|
316
|
+
through. For each reviewed commit with an outcome at the horizon (`--horizon-days`, default 14,
|
|
317
|
+
only outcomes recorded at exactly that horizon), every primary reviewer (role `qa`) that gave
|
|
318
|
+
that attempt a PASS is credited with one landed outcome. A row is keyed by the reviewer's alias,
|
|
319
|
+
or its model when no alias was recorded, as the forge-ui Models view keys it. A landed outcome
|
|
320
|
+
**did not hold** when a later commit reverted it, or when under 80% of its added lines were kept
|
|
321
|
+
(rewritten): the export's `survives` rule.
|
|
322
|
+
|
|
323
|
+
- The rate of work that did not hold has a Wilson 95% interval, shown only from 10 landed
|
|
324
|
+
outcomes on. Below that the row reads `not enough data (n/10)`, and the JSON has
|
|
325
|
+
`enough: false` with `rate` and `ci` null.
|
|
326
|
+
- A shadow review's PASS decided nothing, so it is not credited. Work that landed after a FAIL
|
|
327
|
+
(the severity gate or a human accepted it) and work with no recorded review are counted as
|
|
328
|
+
unattributed. `lost` and `no_additions` outcomes are skipped.
|
|
329
|
+
- Survival is only followed for parallel integrations, so solo runs and hand merges are not here.
|
|
330
|
+
- It only advises: nothing that routes a task or builds a prompt reads it.
|
|
331
|
+
|
|
332
|
+
forge-ui shows the rate as the Reviewers table's **Did not hold** column, through
|
|
333
|
+
`GET /api/models/reviewers`, which runs this command with `--json`.
|
|
334
|
+
|
|
335
|
+
## Plan forecast
|
|
336
|
+
|
|
337
|
+
`forge dataset forecast <plan>` reads the plan's `tasks.tsv` (the declared tier from
|
|
338
|
+
`tasks.declared.tsv` when there is one) and, for each task not yet MERGED, looks up past tasks
|
|
339
|
+
that ran at the same tier on the same resolved model. The effort must match too when the task's
|
|
340
|
+
spec names one. It uses the same task history as `forge dataset routing`, without the plan's own
|
|
341
|
+
run.
|
|
342
|
+
|
|
343
|
+
- Tokens are input plus output (cached input is not added again), summed over every implement
|
|
344
|
+
and review dispatch of the task, retries included. Time is the sum of those dispatches'
|
|
345
|
+
durations. Both show median and p90 (nearest rank).
|
|
346
|
+
- The first-try PASS rate has a Wilson 95% interval. Tasks below 50% are flagged as likely to
|
|
347
|
+
need a retry. Under 5 past tasks the row reads `not enough history (n/5)` and the JSON has
|
|
348
|
+
`enough: false` with no rate.
|
|
349
|
+
- Totals add up the per-task medians and p90s, the wall time by wave (the slowest task of each
|
|
350
|
+
wave), and the expected number of first-try passes.
|
|
351
|
+
- It never creates the store, and prints nothing when the store is missing or has no history.
|
|
352
|
+
|
|
353
|
+
`forge-parallel.sh run` prints the same block before it dispatches. It runs with a time budget
|
|
354
|
+
(`FORGE_FORECAST_BUDGET_S`, default 3 seconds), prints one line and moves on if reading the
|
|
355
|
+
history takes longer, never prints errors and never changes the exit code. `--no-forecast` or
|
|
356
|
+
`FORGE_FORECAST=off` turn it off. The block is written to the plan as `forecast.json` and
|
|
357
|
+
recorded once as a `forecast` event, so it can later be compared to what the run did.
|
|
358
|
+
|
|
359
|
+
The history is observational: harder tasks tend to go to stronger models, so a rate describes
|
|
360
|
+
what happened to similar tasks, not what a model would do on this one. It only advises: nothing
|
|
361
|
+
that routes a task or builds a prompt reads it.
|
|
362
|
+
|
|
281
363
|
## Importing old runs
|
|
282
364
|
|
|
283
365
|
`import` reads run dirs that predate live capture through the same readers. An execution
|
|
@@ -297,6 +379,22 @@ so the importer will not guess which execution wrote a mutable file:
|
|
|
297
379
|
- Archives are extracted to a temp dir and mapped back to their original path. The legacy
|
|
298
380
|
run id hashes that original path, so the archive and the original dir agree on it.
|
|
299
381
|
|
|
382
|
+
- **Import is strict when a caller must know.** `importer.import_run` is strict by default and
|
|
383
|
+
`forge runs rm` relies on it: it stores through `record_strict`, which raises (and the deletion
|
|
384
|
+
stops) if any event, blob included, was not stored, or any part of the run could not be read,
|
|
385
|
+
instead of treating an unreadable file as absent. Live capture stays fail-open. Interventions import by content, not by time of capture:
|
|
386
|
+
`tasks/<id>/escalations.tsv` (four or five columns; a missing `source` reads as `policy`) and
|
|
387
|
+
`control.log` produce the same event ids as live capture, so a re-import fills gaps without
|
|
388
|
+
duplicating. A line that does not parse is reported as failed.
|
|
389
|
+
- **Archives describe themselves.** `archive_run` writes `FORGE_ARCHIVE.json` first
|
|
390
|
+
(`format`, `original_path`, `run_id`, `dataset_uid`, `created`), then `FORGE_ARCHIVE_EXCLUDED.txt`, then
|
|
391
|
+
the run dir under its basename. Symlinks, files over 50 MB, `review-*` snapshots and what the
|
|
392
|
+
importer never reads are left out and listed in the excluded file. Importing such an archive maps
|
|
393
|
+
it back to `original_path`, so its run id and uid match the original dir's; an unreadable path
|
|
394
|
+
during archiving is an error, not a silent omission. The archive is written to a temp name, fsynced
|
|
395
|
+
and renamed. `<role>.session` and `<role>.continue` are plain files in the attempt dir, so a
|
|
396
|
+
run archive carries them and import tags the dispatch from those files.
|
|
397
|
+
|
|
300
398
|
`~/.local/share/forge/legacy/` holds archives of run dirs taken before live capture, with a
|
|
301
399
|
`MANIFEST.tsv`. Passing that dir to `import` reads every archive, plus the ledger copies
|
|
302
400
|
listed in `ledgers/REPOS.tsv`.
|
|
@@ -308,7 +406,8 @@ listed in `ledgers/REPOS.tsv`.
|
|
|
308
406
|
- **Never read by a prompt builder.** Memory inject, the capsule and Ripwire must not read
|
|
309
407
|
the store. A reviewer that could see earlier runs' verdicts would not be independent.
|
|
310
408
|
`tests/test_dataset.py` enforces that outside its own package, the dataset is only
|
|
311
|
-
imported for capture.
|
|
409
|
+
imported for capture. The one read is `run`'s [plan forecast](#plan-forecast), which goes to
|
|
410
|
+
the operator's terminal and `forecast.json`, never into a prompt.
|
|
312
411
|
- **Read-only in your repos.** Capture and the harvester run only object-store git queries
|
|
313
412
|
in a repo: `rev-parse`, `rev-list`, `log`, `show`, `ls-tree`, `cat-file`, `diff` between
|
|
314
413
|
commits and `blame` at a commit. All run with `GIT_OPTIONAL_LOCKS=0`. None takes the
|
package/references/decompose.md
CHANGED
|
@@ -266,8 +266,10 @@ So:
|
|
|
266
266
|
## The capsule
|
|
267
267
|
|
|
268
268
|
Every dispatch is a clean slate — forge never passes `--continue`, `--resume`, or
|
|
269
|
-
`codex exec resume
|
|
270
|
-
|
|
269
|
+
`codex exec resume`, except the opt-in `plan --reviewer-session` experiment (it continues
|
|
270
|
+
a reviewer's session across rounds on claude, openclaude and codex). That keeps one task's
|
|
271
|
+
confused turn from poisoning another and keeps each task's context small, but it also
|
|
272
|
+
leaves each dwarf blind to the wider run.
|
|
271
273
|
|
|
272
274
|
`tasks/<id>/capsule.md` records ownership and baseline context. `baseline.capsule` preserves
|
|
273
275
|
the implementation-time capsule for QA. MERGED means the reviewed commit is an ancestor of
|
|
@@ -275,7 +277,8 @@ this task's pinned baseline, not merely that another task finished while QA was
|
|
|
275
277
|
Complete requirements, approach, retry findings and distinct memory facts are assembled once
|
|
276
278
|
per role; implementation and review instructions are separate.
|
|
277
279
|
|
|
278
|
-
It exists to prevent two specific failures that clean slates create
|
|
280
|
+
It exists to prevent two specific failures that clean slates create, which the opt-in
|
|
281
|
+
`--reviewer-session` experiment does not change:
|
|
279
282
|
|
|
280
283
|
- a dwarf reimplementing a helper an earlier task already merged, because nothing told it
|
|
281
284
|
that task was done;
|
|
@@ -390,7 +393,7 @@ realistic input or a broken existing contract; P2 edge-case bug, unusual-timing
|
|
|
390
393
|
hardening; P3 minor. Severity is impact; CONFIRMED/PLAUSIBLE is confidence.
|
|
391
394
|
|
|
392
395
|
`run`/`retry --qa-threshold Pn` (remembered in `<plan>/qa_threshold`; `none` clears it) sets
|
|
393
|
-
the **least severe level that still blocks**. `tasks/<id>/qa_threshold` overrides it per task —
|
|
396
|
+
the **least severe level that still blocks**. `tasks/<id>/qa_threshold` overrides it per task (the value `none` exempts the task from a plan threshold; `forge control set ... qa-threshold` writes both and takes effect at the next attempt) —
|
|
394
397
|
the way to give a dev-tooling or investigation task a looser bar than production code,
|
|
395
398
|
which otherwise gets a new finding in a new corner of a non-default option every round and
|
|
396
399
|
only a human decision ends it. The orchestrator decides *deterministically* (`forge-contract.py gate`),
|
|
@@ -425,7 +428,8 @@ Two details that are load-bearing rather than incidental:
|
|
|
425
428
|
diff.
|
|
426
429
|
- **An identical retry is not re-reviewed.** If the diff comes back byte-identical to the
|
|
427
430
|
previous attempt, the task fails again without a QA dispatch. Paying a reviewer to read
|
|
428
|
-
the same code twice buys nothing.
|
|
431
|
+
the same code twice buys nothing. A retry that only rejects the findings, and leaves
|
|
432
|
+
that diff unchanged, is still this identical retry.
|
|
429
433
|
|
|
430
434
|
`retry` is a command a human types. Forge does not loop a dwarf against its own reviewer on
|
|
431
435
|
its own initiative; the one exception is the opt-in `run --retry-failed N` below.
|
|
@@ -437,6 +441,14 @@ work: no dwarf, no new commit, no attempt spent. A `tasks/<id>/resume` marker re
|
|
|
437
441
|
revalidated against the branch tip and the saved source fingerprint before it is trusted; if the
|
|
438
442
|
worktree changed, the full task runs. Findings are carried only from a real review.
|
|
439
443
|
|
|
444
|
+
On a retry whose diff changed, QA receives the previous round under **Findings from the
|
|
445
|
+
previous review round**, the diff since that round (`tasks/<id>/changes.since.diff`,
|
|
446
|
+
`changes.since.range` and `changes.since.stat`; omitted when the earlier commit is
|
|
447
|
+
missing or not an ancestor, or when nothing changed) and the implementer's reply. It is
|
|
448
|
+
told to restate only findings that are still open, as `- [Px][CONFIRMED|PLAUSIBLE]`
|
|
449
|
+
lines, because every labelled line is counted. That is the retry's review, not the
|
|
450
|
+
QA-only re-run above.
|
|
451
|
+
|
|
440
452
|
**In-run retries.** `run <plan> --retry-failed N` re-queues a task whose reviewer said
|
|
441
453
|
`FAIL`/`UNKNOWN` as soon as it ends, up to `N` more times in that run, preparing the retry exactly
|
|
442
454
|
as `retry` does (same findings carry-forward, same pinned base, same cumulative review). Other
|
|
@@ -466,10 +478,34 @@ turns it off):
|
|
|
466
478
|
QA-only resume, which spends no implementer attempt.
|
|
467
479
|
- **Reporting:** the policy's state is decided after every attempt, so a run whose retry budget ran out
|
|
468
480
|
still reports `its next retry goes to <spec>` or the split suggestion under the results.
|
|
481
|
+
A pending pin is reported ahead of both, as `its next retry goes to <spec> (pinned)`, or
|
|
482
|
+
`(pinned; after the QA-only re-run)` only when the next entry really re-runs QA (the
|
|
483
|
+
`resume` marker, or an unreviewed verdict, still matches the worktree). A marker the
|
|
484
|
+
next retry will discard is not that wait.
|
|
469
485
|
|
|
470
486
|
History cannot yet say whether escalating rescues a task (31% of the 29 later attempts on a
|
|
471
487
|
changed spec passed, mostly effort bumps); the attempts this policy makes are what will.
|
|
472
488
|
|
|
489
|
+
**Live controls.** `forge control show|set|pin|unpin <plan>` changes a run while it works:
|
|
490
|
+
`qa-threshold` (read once per attempt, so an attempt keeps the bar it started with; per task, `none`
|
|
491
|
+
exempts it and `inherit` drops the override), `timeout`, `escalate-after`, `verify-retries`, and,
|
|
492
|
+
only while a scheduler is live, `retry-budget` (plan or `--task`) and `infra-retries`. The budgets
|
|
493
|
+
keep the **per invocation** rule: the flags of `run` are not remembered, and a live scheduler only
|
|
494
|
+
mirrors them into `retry_budget`, `infra_budget` and `tasks/<id>/retry_budget` (re-read at every
|
|
495
|
+
decision, removed when it stops; `tasks/<id>/retries.used` stays), so a change dies with the
|
|
496
|
+
run. `pin <task> <spec>` sets the model the task's next full retry uses, once (checked with
|
|
497
|
+
`forge-dispatch.sh doctor`, no quota): `prepare_retry` claims `escalation.pin` as
|
|
498
|
+
`escalation.pin.applying`, rewrites `tasks.tsv`, appends the `escalations.tsv` row with
|
|
499
|
+
`source=pin`, clears `escalation.stop`/`split.suggested`, then consumes the pin. A symlinked
|
|
500
|
+
`escalations.tsv`, or a symlink in any directory from the plan down to it, is not written:
|
|
501
|
+
the pin stays and nothing is retried. A QA-only resume leaves it for the next full retry;
|
|
502
|
+
Fractal plans never apply pins; a MERGED task refuses one.
|
|
503
|
+
The last column of `escalations.tsv` says who decided: `policy`, `retry` (an explicit
|
|
504
|
+
`--dwarf`) or `pin`; four-column rows read as `policy`. Every change goes to `control.log` (JSON lines:
|
|
505
|
+
`at`, `actor` `cli|ui`, `action`, `task`, `key`, `value`, `previous`) and to the dataset as a
|
|
506
|
+
`control` event. Exit 2 invalid input, 3 refused (not live, MERGED, Fractal, failing doctor,
|
|
507
|
+
lock busy over 5 s). Report them; never set one yourself unless asked.
|
|
508
|
+
|
|
473
509
|
**Ordinary execution resume:** run `run` again. Tasks already `MERGED` are skipped. A saved PASS is fingerprint-checked and merged without
|
|
474
510
|
redispatch. PENDING, `INFRA` (paused by an infrastructure failure) and newly unblocked tasks run
|
|
475
511
|
when eligible. Failed or interrupted tasks
|
package/references/harnesses.md
CHANGED
|
@@ -22,6 +22,7 @@ provider test. See [Fractal execution](fractal.md).
|
|
|
22
22
|
- [opencode](#opencode)
|
|
23
23
|
- [antigravity (agy)](#antigravity-agy)
|
|
24
24
|
- [grok](#grok)
|
|
25
|
+
- [Continuing a reviewer session](#continuing-a-reviewer-session)
|
|
25
26
|
- [Adding a harness](#adding-a-harness)
|
|
26
27
|
|
|
27
28
|
## At a glance
|
|
@@ -265,7 +266,8 @@ passes 30m and exposes `--agy-timeout` to change it.
|
|
|
265
266
|
|
|
266
267
|
```
|
|
267
268
|
grok --model <model> --prompt-file <prompt> --output-format streaming-messages-json \
|
|
268
|
-
--reasoning-effort <effort>
|
|
269
|
+
--reasoning-effort <effort> --background-wait-timeout <limit> \
|
|
270
|
+
(--permission-mode acceptEdits --allow Bash [--deny Edit --deny Write] | --always-approve)
|
|
269
271
|
```
|
|
270
272
|
|
|
271
273
|
The xAI Grok Build CLI, signed in with `grok login` (or `XAI_API_KEY`); `grok models` lists
|
|
@@ -294,6 +296,15 @@ GitHub Copilot) as a second row.
|
|
|
294
296
|
- **Untested:** whether `--permission-mode dontAsk` (deny without cancelling) together with
|
|
295
297
|
explicit Edit/Write/Bash allows would avoid the cancel. Nothing changes until that is
|
|
296
298
|
proven live.
|
|
299
|
+
- **Background work waits as long as forge's limit allows.** After its last turn a headless run
|
|
300
|
+
waits for the background jobs it started (a test suite run in the background, say) and wakes
|
|
301
|
+
the model with each result. By default it waits only 600 s, then stops the work and exits 1:
|
|
302
|
+
a real dwarf whose full suite ran longer ended as an error. forge passes the role's own limit
|
|
303
|
+
as `--background-wait-timeout`, or a day when the limit is off (`--timeout 0`; grok refuses
|
|
304
|
+
0). grok counts from the last turn and forge from the start, so forge's watchdog always fires
|
|
305
|
+
first and the run ends as a timeout. The flag is missing from `grok --help`; preflight checks
|
|
306
|
+
it with `grok --background-wait-timeout 1 --help` instead, because grok rejects an unknown flag
|
|
307
|
+
that comes before `--help` (exit 2).
|
|
297
308
|
- The prompt goes by `--prompt-file`, never argv, and stdin is `/dev/null`.
|
|
298
309
|
|
|
299
310
|
Verified live on 2026-10-19 with a `FORGE_OK` dispatch: exit 0, `permissionMode` acceptEdits,
|
|
@@ -301,6 +312,37 @@ the answer extracted from the `result` line, tokens recorded, nothing left runni
|
|
|
301
312
|
day, grok-4.7 dwarfs implemented and passed review on real tasks (the forge-ui integration run).
|
|
302
313
|
A grok reviewer has not been proven live yet.
|
|
303
314
|
|
|
315
|
+
## Continuing a reviewer session
|
|
316
|
+
|
|
317
|
+
`forge-dispatch.sh qa <spec> ... --continue-session <id>` continues an earlier reviewer session instead of
|
|
318
|
+
starting a fresh one. `<id>` is the session or thread id a dispatch left in `<run-dir>/<role>.session`
|
|
319
|
+
(8 to 128 characters of `[A-Za-z0-9-]`, not starting with `-`). It is accepted for the `qa` role on claude,
|
|
320
|
+
openclaude and codex in custom-prompt mode only; any other role, harness, `--native-review` or a malformed id
|
|
321
|
+
is a usage error (exit 2). Nothing in forge asks for it by default.
|
|
322
|
+
|
|
323
|
+
Probed 2026-10-22 (claude 2.1.307, codex 0.162.0):
|
|
324
|
+
|
|
325
|
+
| Harness | Argv | Notes |
|
|
326
|
+
|---|---|---|
|
|
327
|
+
| claude, openclaude | the fresh QA argv plus `--resume <id>` | Continues from a different working directory and keeps the same session id. Permission flags and the read-only restriction stay; the prompt is still on stdin. |
|
|
328
|
+
| codex | `codex exec resume <id> --json -m MODEL --skip-git-repo-check -o LAST [-c model_reasoning_effort=…] -c 'sandbox_mode="read-only"' -` | Continues from a different working directory. `exec resume` has **no `-C/--cd` and no `-s/--sandbox`**, so the dispatch's own `cd "$REPO"` sets the directory and the sandbox is passed as a config override (the quotes are TOML). The first turn's sandbox was inherited when probed (read-only); the override makes it explicit. Under `--yolo` the bypass flag replaces the override. |
|
|
329
|
+
|
|
330
|
+
The prompt is preceded by a short fixed preamble (this continues your earlier review, the work has changed,
|
|
331
|
+
the working directory is now `<repo>`, re-read files, this round's instructions win). `<role>.prompt` includes
|
|
332
|
+
it. `<role>.continue` records the attempt as `requested=<id>`, then `mode=continued` or `mode=failed` with
|
|
333
|
+
`reason=<one line>`, and is archived with the attempt.
|
|
334
|
+
|
|
335
|
+
**Exit 9** means the session was not continued and the caller may dispatch fresh. Dispatch never retries fresh
|
|
336
|
+
itself, so each dispatch keeps its own honest record. It happens when:
|
|
337
|
+
|
|
338
|
+
- the harness does not advertise the flags continuing needs (`codex exec resume --help`, or `--resume` in
|
|
339
|
+
claude's help) but would pass preflight for a fresh QA dispatch: `reason=unsupported`, before any model call;
|
|
340
|
+
a harness that fails the fresh preflight too keeps exit 3;
|
|
341
|
+
- the harness ran and exited non-zero: `reason=` the log's last error line.
|
|
342
|
+
|
|
343
|
+
A timeout stays exit 7 and an infrastructure stop stays exit 8; both still write `mode=failed`. `doctor --spec`
|
|
344
|
+
without `--continue-session` is unchanged.
|
|
345
|
+
|
|
304
346
|
## Installing forge into a harness
|
|
305
347
|
|
|
306
348
|
Each harness discovers skills differently, and the differences are not guessable — verify
|
package/references/ui.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# forge-ui
|
|
2
2
|
|
|
3
|
-
forge-ui is a
|
|
3
|
+
forge-ui is a local web page that shows a forge run live and can change it: a decomposed plan or a
|
|
4
4
|
solo run. It is opt-in per run. The orchestrator asks "Open forge-ui to watch this run?
|
|
5
5
|
[y/N]" once at the start of every new interactive run and passes the answer as `--ui` or
|
|
6
6
|
`--no-ui`. Nothing about the run depends on it: a UI that cannot start prints a note and the
|
|
@@ -23,13 +23,25 @@ run goes on, with the same exit code.
|
|
|
23
23
|
review's combined diff, with every P0–P3 finding shown under the line it names.
|
|
24
24
|
- **Fractal child tasks:** the tree a Fractal task was split into, in the inspector and above
|
|
25
25
|
the opened task in 3D, each node with its decision, cost, output and candidate diff.
|
|
26
|
+
- **Controls:** change a live plan from its inspector: QA threshold, timeout, escalate-after,
|
|
27
|
+
verify retries, and (while a scheduler is live) the retry and infra-retry budgets; per task, a
|
|
28
|
+
threshold override (`None` exempts it), a retry budget and a one-shot model pin. Each is one
|
|
29
|
+
`forge control` call; the change is confirmed first and logged in `control.log`.
|
|
30
|
+
- **Open findings and issues:** the findings a run (or all of its repository's runs) left open,
|
|
31
|
+
filed as GitHub issues or dismissed through `forge issues`.
|
|
32
|
+
- **Delete a run:** a dry run, then `forge runs rm`: the dataset keeps the traces, an archive is
|
|
33
|
+
written unless declined, and clean worktrees and unchecked-out branches can go with it.
|
|
34
|
+
- **Settings:** `forge config show` in one view, with edits for the per-repo files and Jev.
|
|
26
35
|
- **Export a run:** one HTML file to share that opens without forge-ui; prompts, diffs, reports
|
|
27
36
|
and local paths stay out unless chosen.
|
|
28
37
|
- **Quick jump and notifications:** Ctrl K (⌘K) or `/` searches every run, task, project and
|
|
29
38
|
model; the bell tells you when a run finishes or fails, a task fails or a final review gives its
|
|
30
39
|
verdict, and can raise browser notifications while the tab is in the background.
|
|
31
40
|
- **Plan run:** tasks in their planned waves with dependency cables, then the finish line:
|
|
32
|
-
integration, verification and the whole-run final review, each a node with its status.
|
|
41
|
+
integration, verification and the whole-run final review, each a node with its status. Tasks
|
|
42
|
+
merged with `accept` show as accepted (hollow green), not failed. A final review superseded
|
|
43
|
+
by later task merges into the integration branch shows as OUTDATED (hollow unlit) instead of
|
|
44
|
+
as the run's verdict; changes on the user's own branch after an integrate are not tracked. The
|
|
33
45
|
*Execution* view shows the same run on wall-clock time, one block per attempt in the lane it
|
|
34
46
|
ran in, with projected blocks for what has not started and a strip of busy slots against
|
|
35
47
|
`--max-parallel`.
|
|
@@ -96,10 +108,23 @@ how the map shows all of a repo's runs. `forge-parallel.sh run` also records the
|
|
|
96
108
|
|
|
97
109
|
## Privacy
|
|
98
110
|
|
|
99
|
-
The server binds only `127.0.0.1
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
111
|
+
The server binds only `127.0.0.1`. **Reading needs no token**: any process on the machine,
|
|
112
|
+
under any user who can reach loopback, can read every run it knows, including prompts, diffs and
|
|
113
|
+
review text from private repos. A running server keeps serving them, including runs that were
|
|
114
|
+
later rerun with `--no-ui`.
|
|
115
|
+
|
|
116
|
+
**Changing needs one.** The server never writes a run file, a repo or a config itself: every
|
|
117
|
+
change is a call of forge's own CLI (`forge control`, `forge issues`, `forge runs rm`,
|
|
118
|
+
`forge config`), so validation, locks and the audit log are the CLI's. Each server start makes a
|
|
119
|
+
random token and puts it in the page it serves; every change must send it back, from a `Host` of
|
|
120
|
+
`127.0.0.1` or `localhost` and with an `Origin` equal to that host when the browser sends one, or
|
|
121
|
+
it is refused before any CLI runs. There are no CORS headers, so another site in your browser
|
|
122
|
+
can neither send nor read a change. Any process that can fetch the page can read the token, as
|
|
123
|
+
it can read the runs, so on a shared machine start the server **read-only**: `--read-only`
|
|
124
|
+
(`python3 -m forge_ui`) or `FORGE_UI_READONLY=1` (also for `forge ui open`). A read-only server
|
|
125
|
+
issues no token and answers every change with 405; its `/api/health` says `"controls": false`.
|
|
126
|
+
`forge ui open` never reuses a server in the other mode: it replaces it, as it does one running
|
|
127
|
+
old code, and `forge ui status` shows the mode.
|
|
103
128
|
|
|
104
129
|
Every run is registered, so a running server shows every run on the machine, not only those
|
|
105
130
|
opened with `--ui`. The 3D view fetches three.js from cdn.jsdelivr.net (no run data is sent;
|