forgedock 1.35.0 → 1.36.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +78 -324
- package/bin/engine/events.mjs +152 -0
- package/bin/engine/events.test.mjs +128 -0
- package/bin/engine/telemetry-seam.mjs +4 -3
- package/bin/engine-cli.mjs +113 -18
- package/bin/experiment.mjs +55 -0
- package/bin/forgedock.mjs +1 -1
- package/bin/hooks/reread-nudge.mjs +15 -4
- package/bin/lean/batch.mjs +42 -2
- package/bin/lean/batch.test.mjs +101 -1
- package/bin/lean/conformance.mjs +14 -3
- package/bin/lean/conformance.test.mjs +26 -0
- package/bin/lean/effects.mjs +3 -0
- package/bin/lean/execution.mjs +2 -0
- package/bin/lean/i18n.mjs +65 -0
- package/bin/lean/i18n.test.mjs +47 -0
- package/bin/lean/lane.mjs +5 -2
- package/bin/lean/lean.mjs +37 -4
- package/bin/lean/lean.test.mjs +124 -5
- package/bin/lean/orchestrate.mjs +49 -5
- package/bin/lean/postdeploy-check.mjs +149 -0
- package/bin/lean/postdeploy-check.test.mjs +146 -0
- package/bin/lean/postdeploy-monitor.mjs +87 -0
- package/bin/lean/postdeploy-monitor.test.mjs +121 -0
- package/bin/lean/precheck.mjs +6 -4
- package/bin/lean/precheck.test.mjs +17 -2
- package/bin/lean/prefilter.mjs +29 -4
- package/bin/lean/prefilter.test.mjs +13 -5
- package/bin/lean/probe.mjs +2 -1
- package/bin/lean/prompts.mjs +10 -2
- package/bin/lean/sandbox-provider.contract.test.mjs +28 -0
- package/bin/lean/sandbox-remote.mjs +7 -0
- package/bin/lean/schedd.mjs +24 -4
- package/bin/lean/schedd.test.mjs +54 -0
- package/bin/stats/aggregate.mjs +1 -1
- package/bin/stats/experiment.mjs +44 -0
- package/bin/stats/github-source.mjs +2 -1
- package/bin/stats/index.mjs +42 -2
- package/bin/stats/release.mjs +75 -0
- package/bin/stats/release.test.mjs +68 -0
- package/commands/orchestrate.md +8 -3
- package/docs/CONFIG.md +49 -0
- package/docs/EVENTS.md +68 -0
- package/docs/STATS.md +31 -0
- package/docs/TELEMETRY.md +3 -1
- package/docs/site/install.md +89 -0
- package/package.json +1 -1
- package/scripts/bench-lightlane.mjs +2 -1
- package/scripts/eval-gate-scorecard.mjs +7 -0
- package/scripts/file-deferred-followups.sh +1 -0
- package/scripts/file-deferred-followups.test.sh +8 -0
package/docs/CONFIG.md
CHANGED
|
@@ -594,6 +594,52 @@ engine:
|
|
|
594
594
|
|
|
595
595
|
---
|
|
596
596
|
|
|
597
|
+
## `post_deploy` (OPTIONAL)
|
|
598
|
+
|
|
599
|
+
Deferred follow-ups whose items are all post-deploy checks are parked (`awaiting-promotion`, no lane) until the parent's fix is on `branches.default`. Keys: `precondition_cmd` (exit 0 ready, 1 parked) and `min_age` (soak window, e.g. `24h`).
|
|
600
|
+
|
|
601
|
+
### Named checks (#4626)
|
|
602
|
+
|
|
603
|
+
A post-deploy requirement carries a check id (`[check:<id>]` in the follow-up item). The command is read from `.forge/post-deploy-checks.json` on the default branch (`git show origin/<default>:...`), never from an issue body or comment:
|
|
604
|
+
|
|
605
|
+
```json
|
|
606
|
+
{ "login-smoke": { "run": "scripts/smoke.sh", "pass": "exit 0", "timeout": 120 } }
|
|
607
|
+
```
|
|
608
|
+
|
|
609
|
+
`pass` is `exit 0` (default) or a `/regex/flags` over the output; `timeout` is seconds. When the gate reports the follow-up ready, the check runs with no model call: pass closes the follow-up with a `FORGE:PROBE` record, fail reopens the parent with the output. An item with no check id, or an id the default branch does not define, gets the `awaiting-measurement` label and never a lane. A contract's check id must be on the default branch or added by the same PR.
|
|
610
|
+
|
|
611
|
+
Closing an experiment follow-up (#4634): add a check that runs `forgedock stats --experiment <id> --require-met` (exit 0 only when every target of the forge.yaml `experiment:` is met; n/a below min n is not met), and put `[check:<check id>]` in the follow-up item:
|
|
612
|
+
|
|
613
|
+
```json
|
|
614
|
+
{ "exp-light-vs-full": { "run": "forgedock stats --experiment light-vs-full --require-met" } }
|
|
615
|
+
```
|
|
616
|
+
|
|
617
|
+
Met targets pass the check and close the follow-up; with `for`/`every` it is re-measured through the window. The probe env is minimal: list the GitHub token variable in the forge.yaml `acceptance_probe.env` allowlist so stats can read telemetry.
|
|
618
|
+
|
|
619
|
+
### Windowed monitoring (#4627)
|
|
620
|
+
|
|
621
|
+
A check may declare `for` (the window, `30m`/`24h`/`7d`/`2w`) and `every` (the cadence, default `6h`; needs `for`); an entry with an invalid duration, or `every` without `for`, is ignored:
|
|
622
|
+
|
|
623
|
+
```json
|
|
624
|
+
{ "login-smoke": { "run": "scripts/smoke.sh", "for": "7d", "every": "6h" } }
|
|
625
|
+
```
|
|
626
|
+
|
|
627
|
+
When every item's check is windowed, the first pass records the release (the `origin/<default>` sha) and the window end in a `FORGE:PROBE` record, labels the follow-up `post-deploy-monitoring` and leaves it open (no lane, no second measurement by the pre-filter). It re-runs, with the command read from the default branch each time, when `every` has elapsed or `origin/<default>` moved (a later release). At the window end it closes. A fail after a pass opens a regression issue (`FORGE:REGRESSION`) naming the fix commit, the last passing release and the breaking release, and closes the follow-up; the parent is not reopened. A fail on the first run behaves as above (the parent reopens). No model call.
|
|
628
|
+
|
|
629
|
+
Triggers: `forgedock probe --due [--dry-run] [--repo owner/repo]` runs the due ones once; the `forgedock schedd` daemon runs them on a timer while it is up (`FORGEDOCK_SCHEDD_DUE_MS`, default 15 minutes).
|
|
630
|
+
|
|
631
|
+
|
|
632
|
+
## `events` (OPTIONAL)
|
|
633
|
+
|
|
634
|
+
Outbound lifecycle events (#4630). The local JSONL sink is always on; the webhook needs both keys. See [EVENTS.md](EVENTS.md) for the schema and signature.
|
|
635
|
+
|
|
636
|
+
```yaml
|
|
637
|
+
events:
|
|
638
|
+
webhook_url: https://hooks.example.com/forgedock # https, or http on localhost; unset = no webhook
|
|
639
|
+
webhook_secret_env: FORGE_EVENTS_SECRET # NAME of the env var holding the HMAC secret (default shown); never put the secret here
|
|
640
|
+
dir: ~/.forge/events # local JSONL directory
|
|
641
|
+
```
|
|
642
|
+
|
|
597
643
|
## `lean` (OPTIONAL)
|
|
598
644
|
|
|
599
645
|
Settings of the core work-on (`bin/lean/`) that change its contract.
|
|
@@ -709,6 +755,8 @@ verify:
|
|
|
709
755
|
| `e2e_paths` | string | No | Comma-separated globs of e2e specs (same glob rules as `skip_paths`). Default `e2e/**,**/e2e/**,*.e2e.*,*.e2e-spec.*,cypress/**,playwright/**`. |
|
|
710
756
|
| `e2e_blocking` | string | No | `changed` (default): only failures of the lane's own specs block the merge; other e2e failures are reported as an anomaly. `all`: every e2e failure blocks. |
|
|
711
757
|
|
|
758
|
+
**English-copy check (forge#4596).** At hand-off and after each base-sync merge, keys the lane added to a non-`en` JSON locale catalog (`messages/ta.json`, `locales/ta/common.json`) whose value equals the same key in the `en` counterpart go back to the builder once to translate (placeholders kept, status entries inserted in place, never re-sorted); a copy left over is an anomaly. Always on, deterministic, no cost without catalogs. The repo's own `i18n:check` is not run by the engine: list it in `lean.precheck.commands` to run it before the PR.
|
|
759
|
+
|
|
712
760
|
**Flow** (`ci`/`hybrid`, `bin/lean/verify.mjs` `runCiVerify`):
|
|
713
761
|
|
|
714
762
|
1. After the build and before every conformance review, the engine pushes the lane branch (a plain push, never forced) and runs CI on the head. A base-sync that moves the head is verified again.
|
|
@@ -1438,6 +1486,7 @@ These variables are read from the operator's runner environment, not from `forge
|
|
|
1438
1486
|
| `FORGE_TRAIL_QG_SINCE` | `2026-10-07T03:40:12Z` | ISO-8601 UTC cutoff for the legacy `FORGE:QUALITY_GATE` grace. An issue whose trusted `FORGE:BUILDER:COMPLETE` comment was last updated before this time has `QUALITY_GATE` waived. Set to an empty string to disable the grace. A malformed or future value (including a runner clock earlier than this default) fails closed: the verifier exits 2, an infrastructure block rather than `MISSING`. |
|
|
1439
1487
|
| `FORGEDOCK_ALLOW_API_BACKEND` | unset | Set to `1` (host owner approval) together with forge.yaml `execution.agent_backend: api` to let `--backend auto` use the paid API backend when no `claude` CLI is available. Without both, `auto` never selects `api` (#4257). |
|
|
1440
1488
|
| `FORGE_TELEMETRY` | `on` | Set to `off` to disable gate telemetry. Gate scripts append one structured record per decision to a trusted `FORGE:TELEMETRY v1` comment; emission is best-effort and never changes a gate's output or exit code. See [TELEMETRY.md](TELEMETRY.md). |
|
|
1489
|
+
| `FORGE_EVENTS` | `on` | Set to `off` to disable the lifecycle event stream (local JSONL and webhook). `FORGE_EVENTS_DIR` overrides the JSONL directory; the webhook secret is read from the env var named by `events.webhook_secret_env` (default `FORGE_EVENTS_SECRET`). See [EVENTS.md](EVENTS.md). |
|
|
1441
1490
|
|
|
1442
1491
|
### Install root resolution
|
|
1443
1492
|
|
package/docs/EVENTS.md
ADDED
|
@@ -0,0 +1,68 @@
|
|
|
1
|
+
# Lifecycle Events
|
|
2
|
+
|
|
3
|
+
ForgeDock emits a small, versioned stream of lifecycle events for outside consumers (dashboards, chat bots, audit logs). It
|
|
4
|
+
is derived from the records the engine already produces (the run log and `docs/TELEMETRY.md` inputs): deterministic, no model
|
|
5
|
+
call, no consumer required. With no configuration the events go only to a local JSONL file.
|
|
6
|
+
|
|
7
|
+
## Envelope (v1)
|
|
8
|
+
|
|
9
|
+
```json
|
|
10
|
+
{"schema":"forgedock.event","v":1,"id":"1793012345-9f2c1ab04d7e","type":"pr.merged","ts":"2026-10-26T12:00:00Z","repo":"o/r","issue":9,"pr":40,"run":"core_9_1793012000","cost_usd":0.75,"wall_ms":90000,"data":{"milestone":"m1","code":"ok"}}
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
| Key | Meaning |
|
|
14
|
+
|-----|---------|
|
|
15
|
+
| `schema` | Always `forgedock.event`. |
|
|
16
|
+
| `v` | Schema version (integer). Adding an event type or an optional key keeps `v`; renaming or removing one, or changing a meaning, bumps it. |
|
|
17
|
+
| `id` | Unique per event; consumers de-duplicate on it. |
|
|
18
|
+
| `type` | One of the types below. |
|
|
19
|
+
| `ts` | UTC, second precision. |
|
|
20
|
+
| `repo`, `issue`, `pr`, `run` | `owner/name`, issue number, PR number, run id; `null` when not known. |
|
|
21
|
+
| `cost_usd`, `wall_ms` | Present on `pr.merged`: the lane's total model spend (sum of every phase attempt's `cost_usd_delta`, else `cost_usd`) and the engine process wall time. |
|
|
22
|
+
| `data` | The type's keys (below); a key with no value is omitted. Strings are capped at 200 characters. |
|
|
23
|
+
|
|
24
|
+
## Event types
|
|
25
|
+
|
|
26
|
+
| Type | When | `data` |
|
|
27
|
+
|------|------|--------|
|
|
28
|
+
| `lane.picked_up` | A core lane starts an issue. | `lane`, `path` |
|
|
29
|
+
| `contract.frozen` | The lane's contract is frozen. | `hash`, `requirements`, `deferred`, `derived` |
|
|
30
|
+
| `pr.opened` | The lane opens its PR (`pr` is set). | none |
|
|
31
|
+
| `pr.merged` | The run ends `merged` (`pr`, `cost_usd`, `wall_ms` set). | `milestone`, `code` |
|
|
32
|
+
| `probe.passed` / `probe.failed` | A declared acceptance probe is recorded (`forgedock probe`, batch sweep). | `detail`, `fix` |
|
|
33
|
+
| `release.promoted` | Reserved (schema only): a fix reached the default branch. | `fix`, `default_branch` |
|
|
34
|
+
| `regression.detected` | A windowed post-deploy check failed and filed a regression issue (`forgedock probe --due`, daemon timer; `issue` is the monitored follow-up). | `subject`, `metric`, `detail` |
|
|
35
|
+
| `experiment.verdict` | `forgedock stats --experiment <id> --emit-event` found targets `met` or `missed` (`inconclusive` is not emitted; plain `stats` never emits). | `experiment`, `verdict`, `detail` |
|
|
36
|
+
|
|
37
|
+
`release.promoted` is still reserved: the schema, docs and `promotedEvent` builder exist, and the part that detects promotion
|
|
38
|
+
calls `emitEvent(type, data, ctx)` from `bin/engine/events.mjs` when it lands. Nothing emits it yet.
|
|
39
|
+
|
|
40
|
+
## Local sink
|
|
41
|
+
|
|
42
|
+
One JSON object per line, appended to `~/.forge/events/<owner>_<repo>.jsonl` (directory override: forge.yaml `events.dir` or
|
|
43
|
+
`FORGE_EVENTS_DIR`). The file is a cache of what was emitted on this machine, not a record of truth: GitHub stays the register.
|
|
44
|
+
|
|
45
|
+
## Webhook (optional)
|
|
46
|
+
|
|
47
|
+
```yaml
|
|
48
|
+
events:
|
|
49
|
+
webhook_url: https://hooks.example.com/forgedock # https, or http on localhost
|
|
50
|
+
webhook_secret_env: FORGE_EVENTS_SECRET # NAME of the env var holding the secret (this is the default name)
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
Both the URL and the env var must be set, else nothing is sent. The secret is never read from forge.yaml. Each event is one `POST`
|
|
54
|
+
of the envelope JSON with a 5 second timeout, no redirects and no retry; a failed delivery is dropped and never affects the run.
|
|
55
|
+
|
|
56
|
+
Headers: `X-Forge-Event` (the type), `X-Forge-Timestamp` (unix seconds), `X-Forge-Signature` = `sha256=` + hex
|
|
57
|
+
`HMAC-SHA256(secret, "<timestamp>.<raw body>")`. Verify against the raw body and reject old timestamps:
|
|
58
|
+
|
|
59
|
+
```js
|
|
60
|
+
const expected = createHmac("sha256", secret).update(`${ts}.${rawBody}`).digest("hex");
|
|
61
|
+
const ok = timingSafeEqual(Buffer.from(sig.replace(/^sha256=/, ""), "hex"), Buffer.from(expected, "hex"));
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
## Guarantees
|
|
65
|
+
|
|
66
|
+
- Never fails or slows the run beyond the webhook timeout; `FORGE_EVENTS=off` disables the sink and the webhook.
|
|
67
|
+
- Ids, counts, codes and costs only: no issue text, diffs, file contents or secrets.
|
|
68
|
+
- Contract test: `bin/engine/events.test.mjs` pins the types, keys and the signature algorithm.
|
package/docs/STATS.md
CHANGED
|
@@ -168,3 +168,34 @@ Comments are read from the repo-wide `issues/comments?since=` endpoint (a few pa
|
|
|
168
168
|
telemetry comment in the window get their full comment list re-read (6 in parallel, backing off when the API rate limit is
|
|
169
169
|
low). A run with no `--since` and no cache still walks the whole history once. Telemetry comments are edited in place, which
|
|
170
170
|
the comment window catches; if a cached item still looks stale, run with `--refresh`.
|
|
171
|
+
|
|
172
|
+
## Per-release regression rule (`--by version`, forge#4628)
|
|
173
|
+
|
|
174
|
+
```bash
|
|
175
|
+
npx forgedock stats --by version [--post-metrics <pr>] [--min-n N] [--repo <owner/repo>] [--json]
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
Median cost per run, wall p50, first-pass rate and needs-human rate of each release against the previous known one.
|
|
179
|
+
A metric whose own sample is under `--min-n` (default 10) is null, and so is its verdict; verdicts are
|
|
180
|
+
`regressed|improved|flat|null`, never a pass. `--post-metrics <pr>` posts (or updates) a `<!-- FORGE:METRICS v1 -->`
|
|
181
|
+
comment on that release PR with the table and the snapshot as JSON; without it nothing is written to GitHub.
|
|
182
|
+
|
|
183
|
+
## Live A/B experiments (`--experiment <id>`, forge#4629)
|
|
184
|
+
|
|
185
|
+
Declare one experiment in forge.yaml (flat keys under `experiment:`):
|
|
186
|
+
|
|
187
|
+
```yaml
|
|
188
|
+
experiment:
|
|
189
|
+
id: light-vs-full
|
|
190
|
+
arms: light,full # at least two
|
|
191
|
+
ratio: 1,1 # integer weights, default equal
|
|
192
|
+
metrics: usd_median,wall_p50_ms,first_pass_rate,needs_human_rate # default all four
|
|
193
|
+
targets: light.usd_median<=1.2xfull,light.first_pass_rate>=0.8 # arm.metric<=|>=limit, optionally x<other arm>
|
|
194
|
+
min_n: 10
|
|
195
|
+
```
|
|
196
|
+
|
|
197
|
+
The lean lane assigns each issue an arm (sha1 of id and issue number over the ratio, so a re-run keeps its arm) and tags
|
|
198
|
+
`experiment` and `arm` on `phase_attempt` and `run_terminal`. `forgedock stats --experiment <id>` reports the finished
|
|
199
|
+
runs per arm with the metrics of `--by version`; a metric under min n is null and so is any target that needs it, never a
|
|
200
|
+
pass. `targets_met` is true only when every target is met. `--require-met` (with `--experiment`) prints the same report but exits 1 unless `targets_met` is true (0 otherwise; 2 without `--experiment`), so a post-deploy check can gate on it (#4634). The report, `scripts/bench-lightlane.mjs aggregate` and
|
|
201
|
+
`scripts/eval-gate-scorecard.mjs` share one record: `{schema: "forge.bench.v1", source, arms, targets}`.
|
package/docs/TELEMETRY.md
CHANGED
|
@@ -5,6 +5,8 @@ counts can be measured without re-reading comments with a model. GitHub is the s
|
|
|
5
5
|
by code (never by a model) into a `FORGE:TELEMETRY` comment, and everything can be rebuilt from the GitHub API alone.
|
|
6
6
|
Local files are only a cache.
|
|
7
7
|
|
|
8
|
+
For a stable, versioned stream aimed at outside consumers (JSONL sink, signed webhook) see [EVENTS.md](EVENTS.md).
|
|
9
|
+
|
|
8
10
|
## Where records live
|
|
9
11
|
|
|
10
12
|
- One `<!-- FORGE:TELEMETRY v1 -->` comment per issue, or per PR for review-side events (`review_panel`,
|
|
@@ -41,7 +43,7 @@ is dropped. Payload keys are `[A-Za-z0-9_]+` and cannot shadow the shared keys;
|
|
|
41
43
|
| `ci_local_check` | `scripts/ci-local.sh run`, one per scoped, widened or executed check (at most 20 per run) | `job`, `selector`, `selected`, `total`, `cache` = `hit` \| `miss` \| `dedup` \| `na`, `duration_ms`, `wait_ms` (job slot wait), `widen` (reason or `none`). A remote-CI failure at merge on a check recorded here as narrowed is a selector bug |
|
|
42
44
|
| `liveness` | `scripts/remediate-liveness.sh` | `decision` = `PROCEED` \| `DEFER` \| `EXHAUSTED`, `why` |
|
|
43
45
|
| `phase_attempt` | the engine (`bin/engine/telemetry-seam.mjs`, one per phase attempt; `phase` = the phase id) | `attempt`, `trigger` (#4362: why the attempt ran, one of `plan`, `build`, `block_correction`, `contract_correction`, `precheck`, `structure`, `docs`, `enumerating`, `sequence`, `ci_unavailable`, `ci_red`, `conformance`, `review`, `merge_ci`, `timeout_restart`, `recovery`; `restart_of` = the trigger a `timeout_restart` replaced), `model`, `backend` (`cli`/`api`), `turns`, `duration_ms`, `cost_usd` (the raw reported value: a running total when the attempt resumed a CLI session), `cost_usd_delta` (#4407: this attempt's cost; consumers sum it when present, else `cost_usd`), `cost_usd_session` (the CLI session's running total), `input_tokens`, `output_tokens`, `cache_read_tokens`, `cache_creation_tokens` (any value the runner did not report is omitted) |
|
|
44
|
-
| `run_terminal` | the engine (`terminate()`, one per run segment) | `reason` (terminal reason), `code` (see `docs/OUTCOME-CODES.md`), `wall_ms` (this engine process, start to terminal) |
|
|
46
|
+
| `run_terminal` | the engine (`terminate()`, one per run segment) | `experiment` and `arm` (#4629, only when forge.yaml declares an `experiment:`; also on `phase_attempt`), `reason` (terminal reason), `code` (see `docs/OUTCOME-CODES.md`), `wall_ms` (this engine process, start to terminal) |
|
|
45
47
|
|
|
46
48
|
The engine records carry `run` = `r_<issue>_<start epoch seconds>` (one per `forgedock run-issue` process) so `forgedock stats`
|
|
47
49
|
(`docs/STATS.md`) can group the attempts of a run and tell a finished run from an unfinished one. They are the GitHub-side
|
|
@@ -0,0 +1,89 @@
|
|
|
1
|
+
# Install
|
|
2
|
+
|
|
3
|
+
Everything you need to install ForgeDock and keep it up to date. For a first run, `npx forgedock demo` needs no setup beyond the requirements below.
|
|
4
|
+
|
|
5
|
+
**Requirements:** [Claude Code](https://docs.anthropic.com/en/docs/claude-code) or [OpenCode](https://opencode.ai/) · [GitHub CLI](https://cli.github.com/) (authenticated) · Node.js ≥ 18.
|
|
6
|
+
|
|
7
|
+
```bash
|
|
8
|
+
npx forgedock # checks your environment, installs commands into ~/.claude/commands/ (available in every Claude Code session on this machine), detects your repo, and hands you a reviewed forge.yaml
|
|
9
|
+
```
|
|
10
|
+
|
|
11
|
+
For OpenCode, install namespaced native commands without changing your provider
|
|
12
|
+
or user-owned settings. When migrating an older ForgeDock adapter, it removes
|
|
13
|
+
only exact ForgeDock-managed legacy entries from `opencode.json`:
|
|
14
|
+
|
|
15
|
+
```bash
|
|
16
|
+
npx forgedock opencode install
|
|
17
|
+
# restart OpenCode, then: /forge/work-on <issue>
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
See [OpenCode support](../OPENCODE.md) for architecture, token-loading rules,
|
|
21
|
+
current parity boundaries, and lifecycle commands.
|
|
22
|
+
|
|
23
|
+
**Install is always global**, to `~/.claude/commands/`. `--global` is still accepted on the command line for backward compatibility but has no effect — there's no other install location to opt out of.
|
|
24
|
+
|
|
25
|
+
> **Want engine-mode dispatch?** `npx forgedock` is transient — the `forgedock` binary isn't persisted in PATH after install. `/orchestrate` and `/autopilot` use agent dispatch mode by default, which is fully functional. To enable engine-mode dispatch (`forgedock run-issue`) with its durable phase table and fail-closed review gate, run `npm install -g forgedock` instead.
|
|
26
|
+
|
|
27
|
+
One command does everything: it checks your environment, installs the slash commands into Claude Code, detects your repo (owner, branches, paths), and hands you a single annotated `forge.yaml` to review — press Enter to accept. Run `npx forgedock init` any time afterward to re-generate the config only.
|
|
28
|
+
|
|
29
|
+
Installing also registers a SessionStart hook, so every Claude Code session
|
|
30
|
+
in a forge-managed directory starts already knowing ForgeDock runs it.
|
|
31
|
+
Per-directory control: `npx forgedock enable` / `disable` / `status`.
|
|
32
|
+
|
|
33
|
+
Then just open Claude Code and run `/work-on <issue>`.
|
|
34
|
+
|
|
35
|
+
> **Cost:** ForgeDock is free and open-source. It orchestrates sessions on **your** Claude account — no compute resold, no per-task markup. A typical `/work-on` run on a straightforward bug costs about what a 15–20 minute manual Claude Code session does.
|
|
36
|
+
|
|
37
|
+
<details>
|
|
38
|
+
<summary><strong>Other install options & commands</strong></summary>
|
|
39
|
+
|
|
40
|
+
**Claude Code plugin marketplace** (Claude Code v2.1.143+):
|
|
41
|
+
|
|
42
|
+
```
|
|
43
|
+
/plugin marketplace add RapierCraftStudios/ForgeDock
|
|
44
|
+
/plugin install forgedock@forgedock
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
Commands then appear as `/forgedock:work-on`, etc. You still run `npx forgedock init` to generate `forge.yaml`.
|
|
48
|
+
|
|
49
|
+
**Headless / CI:** the pipeline also runs outside Claude Code. `npx forgedock run work-on <issue> --dry-run` previews the assembled prompt and tool plan. `npx forgedock run` picks an execution backend automatically (`--backend auto`, the default): if the Claude Code CLI (`claude`) is installed and already authenticated (Pro/Max subscription or a CLI-managed key), it drives the command through that — **no separate `ANTHROPIC_API_KEY` required**. It never falls back to the paid Anthropic API implicitly: without a CLI it fails with `CLI_BACKEND_UNAVAILABLE` unless the owner opted in (forge.yaml `execution.agent_backend: api` plus `FORGEDOCK_ALLOW_API_BACKEND=1`; see [CONFIG.md](../CONFIG.md#execution-optional)). Force either path explicitly with `--backend cli` / `--backend api` (or `FORGEDOCK_BACKEND=cli|api`) — the API backend is what CI environments without an interactive `claude` login should use. `npx forgedock run-issue <issue>` executes the same command specs on the durable engine (event-sourced run log, leases, crash-safe resume).
|
|
50
|
+
|
|
51
|
+
**Brand-new project?** `/bootstrap "<idea>"` (or `npx forgedock bootstrap "<idea>" --dry-run` to preview) turns an idea into a validated, milestone-ordered GitHub backlog that `/orchestrate` can build. See [Start a New Project with ForgeDock](start-a-new-project.md).
|
|
52
|
+
|
|
53
|
+
**Explicit install command:**
|
|
54
|
+
|
|
55
|
+
```bash
|
|
56
|
+
npx forgedock install # installs into ~/.claude/commands/
|
|
57
|
+
npx forgedock install --global # same thing — --global is accepted but is a no-op
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
**Maintenance:**
|
|
61
|
+
|
|
62
|
+
```bash
|
|
63
|
+
npx forgedock update # relink commands + refresh the SessionStart hook
|
|
64
|
+
npx forgedock enable # turn ForgeDock on for this directory
|
|
65
|
+
npx forgedock disable # turn ForgeDock off for this directory
|
|
66
|
+
npx forgedock status # show ForgeDock's state for this directory
|
|
67
|
+
npx forgedock doctor # installation health check with fix hints
|
|
68
|
+
npx forgedock report # 30-day pipeline impact receipts (--md for Markdown, --json for scripting)
|
|
69
|
+
npx forgedock uninstall # remove commands, the hook, and tracked copies
|
|
70
|
+
npx forgedock help # show everything
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
> Running `npx forgedock` from *inside* this repo uses the local working tree. From your own project, use `npx forgedock@latest` to pin the published release.
|
|
74
|
+
|
|
75
|
+
</details>
|
|
76
|
+
|
|
77
|
+
<details>
|
|
78
|
+
<summary><strong>A note on install location</strong></summary>
|
|
79
|
+
|
|
80
|
+
ForgeDock briefly experimented with a project-scoped-by-default install mode. It was backed out after causing a "split-brain" bug (`detect` assumed project-scoped while the installer still wrote globally — [#1589](https://github.com/RapierCraftStudios/ForgeDock/issues/1589)), so every version you'd realistically install today only ever writes to `~/.claude/commands/`. `--global` is still accepted as a flag for old scripts/muscle memory, but it changes nothing.
|
|
81
|
+
|
|
82
|
+
If you're carrying an older `--global` habit in scripts or CI, it's harmless to leave it — `npx forgedock --global` and `npx forgedock` do exactly the same thing.
|
|
83
|
+
|
|
84
|
+
</details>
|
|
85
|
+
|
|
86
|
+
**After upgrading**: run `npx forgedock labels setup` once per repo. Repos created before the `workflow:remediating` state (forge#3541) lack that label; the pipeline now creates it on demand (best-effort), but the setup command is the supported way to sync all labels.
|
|
87
|
+
|
|
88
|
+
|
|
89
|
+
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "forgedock",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.36.0",
|
|
4
4
|
"description": "Autonomous development pipeline for coding agents. GitHub as institutional memory, with native Claude Code and OpenCode entrypoints.",
|
|
5
5
|
"bin": {
|
|
6
6
|
"forgedock": "bin/forgedock.mjs"
|
|
@@ -31,6 +31,7 @@ import { tmpdir } from "node:os";
|
|
|
31
31
|
import { join } from "node:path";
|
|
32
32
|
import { pathToFileURL } from "node:url";
|
|
33
33
|
import { detectApiShape } from "../bin/lean/apishape.mjs";
|
|
34
|
+
import { benchRecord } from "../bin/experiment.mjs";
|
|
34
35
|
|
|
35
36
|
export const ARMS = ["light", "full", "control"];
|
|
36
37
|
export const TARGETS = Object.freeze({ cost_vs_control: 1.2, wall_vs_model: 1.3 });
|
|
@@ -225,7 +226,7 @@ export function aggregate(rows) {
|
|
|
225
226
|
false_fix_le_control: t(L.n && C.n ? L.false_fix_rate <= C.false_fix_rate : null, L.false_fix_rate, C.false_fix_rate),
|
|
226
227
|
zero_undetected_api_shape: t(L.n ? L.undetected_api_shape === 0 : null, L.undetected_api_shape, 0),
|
|
227
228
|
};
|
|
228
|
-
return { arms, light_eligible: { ...le, control_cost_usd: ce.cost_usd.sum, ids: [...eligIds].sort() }, targets };
|
|
229
|
+
return { arms, light_eligible: { ...le, control_cost_usd: ce.cost_usd.sum, ids: [...eligIds].sort() }, targets, record: benchRecord({ source: "bench-lightlane", arms, targets }) }; // #4629
|
|
229
230
|
}
|
|
230
231
|
|
|
231
232
|
/** JSONL (or a JSON array) -> rows. */
|
|
@@ -30,6 +30,7 @@
|
|
|
30
30
|
import { readFileSync, writeFileSync } from "node:fs";
|
|
31
31
|
import { dirname, join } from "node:path";
|
|
32
32
|
import { fileURLToPath } from "node:url";
|
|
33
|
+
import { benchRecord } from "../bin/experiment.mjs";
|
|
33
34
|
|
|
34
35
|
/** Fields the fresh scorecard must have (matches eval-scorecard-aggregator.mjs's output). */
|
|
35
36
|
export const REQUIRED_FIELDS = [
|
|
@@ -137,6 +138,12 @@ export function evaluateGate(
|
|
|
137
138
|
fresh_run_id: fresh.run_id ?? null,
|
|
138
139
|
baseline_run_id: baseline.run_id ?? null,
|
|
139
140
|
reason,
|
|
141
|
+
// #4629: the shared result record; the target is null (never a pass) when the rate assertion was skipped
|
|
142
|
+
record: benchRecord({
|
|
143
|
+
source: "eval-gate",
|
|
144
|
+
arms: { baseline: { success_rate_pct: baseline.success_rate_pct }, fresh: { n: freshIssuesRun, success_rate_pct: fresh.success_rate_pct } },
|
|
145
|
+
targets: { success_rate_within_threshold: { pass: rateAssertionSkipped ? null : pass, value: dropPp, limit: thresholdPp } },
|
|
146
|
+
}),
|
|
140
147
|
};
|
|
141
148
|
}
|
|
142
149
|
|
|
@@ -10,6 +10,7 @@
|
|
|
10
10
|
#
|
|
11
11
|
# Usage: file-deferred-followups.sh --parent N --repo OWNER/REPO [--items FILE|-] [--folded N]
|
|
12
12
|
# items TSV, one per line: <path> TAB <why> TAB <known-flaw flag: known-flaw|1|yes|true, else empty>
|
|
13
|
+
# A post-deploy why starts `post-deploy: [check:<id>] ...` (#4626); the token is kept verbatim in the checklist line.
|
|
13
14
|
# Env: DRY_RUN (non-empty: no create/edit, issue number 0), FORGE_TRAIL_TRUSTED_ASSOCIATIONS (default
|
|
14
15
|
# OWNER,MEMBER,COLLABORATOR), FORGE_TRAIL_TRUSTED_LOGINS (default empty), FORGE_DEFERRED_RETRY_SLEEP (secs).
|
|
15
16
|
# Stdout: `path<TAB>issue` per item, then DEFERRED_TELEMETRY deferred_items=N folded=N batched_into=#M filed=N.
|
|
@@ -195,5 +195,13 @@ printf 'x\n' > "$ISSUE_BODY"; issue_json 77 closed OWNER; claim_json OWNER 77
|
|
|
195
195
|
bash "$S" --parent 10 --repo o/r --items "$T/items.tsv" >/dev/null 2>&1
|
|
196
196
|
check "dry-run: no comment/reopen/close/edit" "! grep -qE '^issue (comment|reopen|close|edit|create)' '$GH_LOG'"
|
|
197
197
|
|
|
198
|
+
# 18. #4626: a post-deploy why's [check:<id>] token is kept verbatim in the checklist line
|
|
199
|
+
reset
|
|
200
|
+
printf 'x\n' > "$ISSUE_BODY"
|
|
201
|
+
echo '{"items":[{"number":77,"author_association":"OWNER","user":{"type":"User"},"body":"<!-- FORGE:DEFERRED_FROM: #10 -->"}]}' > "$SEARCH_JSON"; issue_json 77 open OWNER
|
|
202
|
+
printf 'R3: soak\tpost-deploy: [check:login-smoke] after a day\t\n' > "$T/check.tsv"
|
|
203
|
+
bash "$S" --parent 10 --repo o/r --items "$T/check.tsv" >/dev/null 2>&1
|
|
204
|
+
check "check token kept in the checklist line" "grep -q 'post-deploy: \[check:login-smoke\] after a day' '$ISSUE_BODY.edited'"
|
|
205
|
+
|
|
198
206
|
echo "passed=$PASS failed=$FAIL"
|
|
199
207
|
[ "$FAIL" -eq 0 ]
|