forgedock 1.34.0 → 1.36.0

package/docs/EVENTS.md

@@ -0,0 +1,68 @@
+# Lifecycle Events
+
+ForgeDock emits a small, versioned stream of lifecycle events for outside consumers (dashboards, chat bots, audit logs). It
+is derived from the records the engine already produces (the run log and `docs/TELEMETRY.md` inputs): deterministic, no model
+call, no consumer required. With no configuration the events go only to a local JSONL file.
+
+## Envelope (v1)
+
+```json
+{"schema":"forgedock.event","v":1,"id":"1793012345-9f2c1ab04d7e","type":"pr.merged","ts":"2026-10-26T12:00:00Z","repo":"o/r","issue":9,"pr":40,"run":"core_9_1793012000","cost_usd":0.75,"wall_ms":90000,"data":{"milestone":"m1","code":"ok"}}
+```
+
+| Key | Meaning |
+|-----|---------|
+| `schema` | Always `forgedock.event`. |
+| `v` | Schema version (integer). Adding an event type or an optional key keeps `v`; renaming or removing one, or changing a meaning, bumps it. |
+| `id` | Unique per event; consumers de-duplicate on it. |
+| `type` | One of the types below. |
+| `ts` | UTC, second precision. |
+| `repo`, `issue`, `pr`, `run` | `owner/name`, issue number, PR number, run id; `null` when not known. |
+| `cost_usd`, `wall_ms` | Present on `pr.merged`: the lane's total model spend (sum of every phase attempt's `cost_usd_delta`, else `cost_usd`) and the engine process wall time. |
+| `data` | The type's keys (below); a key with no value is omitted. Strings are capped at 200 characters. |
+
+## Event types
+
+| Type | When | `data` |
+|------|------|--------|
+| `lane.picked_up` | A core lane starts an issue. | `lane`, `path` |
+| `contract.frozen` | The lane's contract is frozen. | `hash`, `requirements`, `deferred`, `derived` |
+| `pr.opened` | The lane opens its PR (`pr` is set). | none |
+| `pr.merged` | The run ends `merged` (`pr`, `cost_usd`, `wall_ms` set). | `milestone`, `code` |
+| `probe.passed` / `probe.failed` | A declared acceptance probe is recorded (`forgedock probe`, batch sweep). | `detail`, `fix` |
+| `release.promoted` | Reserved (schema only): a fix reached the default branch. | `fix`, `default_branch` |
+| `regression.detected` | A windowed post-deploy check failed and filed a regression issue (`forgedock probe --due`, daemon timer; `issue` is the monitored follow-up). | `subject`, `metric`, `detail` |
+| `experiment.verdict` | `forgedock stats --experiment <id> --emit-event` found targets `met` or `missed` (`inconclusive` is not emitted; plain `stats` never emits). | `experiment`, `verdict`, `detail` |
+
+`release.promoted` is still reserved: the schema, docs and `promotedEvent` builder exist, and the part that detects promotion
+calls `emitEvent(type, data, ctx)` from `bin/engine/events.mjs` when it lands. Nothing emits it yet.
+
+## Local sink
+
+One JSON object per line, appended to `~/.forge/events/<owner>_<repo>.jsonl` (directory override: forge.yaml `events.dir` or
+`FORGE_EVENTS_DIR`). The file is a cache of what was emitted on this machine, not a record of truth: GitHub stays the register.
+
+## Webhook (optional)
+
+```yaml
+events:
+  webhook_url: https://hooks.example.com/forgedock   # https, or http on localhost
+  webhook_secret_env: FORGE_EVENTS_SECRET            # NAME of the env var holding the secret (this is the default name)
+```
+
+Both the URL and the env var must be set, else nothing is sent. The secret is never read from forge.yaml. Each event is one `POST`
+of the envelope JSON with a 5 second timeout, no redirects and no retry; a failed delivery is dropped and never affects the run.
+
+Headers: `X-Forge-Event` (the type), `X-Forge-Timestamp` (unix seconds), `X-Forge-Signature` = `sha256=` + hex
+`HMAC-SHA256(secret, "<timestamp>.<raw body>")`. Verify against the raw body and reject old timestamps:
+
+```js
+const expected = createHmac("sha256", secret).update(`${ts}.${rawBody}`).digest("hex");
+const ok = timingSafeEqual(Buffer.from(sig.replace(/^sha256=/, ""), "hex"), Buffer.from(expected, "hex"));
+```
+
+## Guarantees
+
+- Never fails or slows the run beyond the webhook timeout; `FORGE_EVENTS=off` disables the sink and the webhook.
+- Ids, counts, codes and costs only: no issue text, diffs, file contents or secrets.
+- Contract test: `bin/engine/events.test.mjs` pins the types, keys and the signature algorithm.

package/docs/STATS.md

@@ -168,3 +168,34 @@ Comments are read from the repo-wide `issues/comments?since=` endpoint (a few pa
 telemetry comment in the window get their full comment list re-read (6 in parallel, backing off when the API rate limit is
 low). A run with no `--since` and no cache still walks the whole history once. Telemetry comments are edited in place, which
 the comment window catches; if a cached item still looks stale, run with `--refresh`.
+
+## Per-release regression rule (`--by version`, forge#4628)
+
+```bash
+npx forgedock stats --by version [--post-metrics <pr>] [--min-n N] [--repo <owner/repo>] [--json]
+```
+
+Median cost per run, wall p50, first-pass rate and needs-human rate of each release against the previous known one.
+A metric whose own sample is under `--min-n` (default 10) is null, and so is its verdict; verdicts are
+`regressed|improved|flat|null`, never a pass. `--post-metrics <pr>` posts (or updates) a `<!-- FORGE:METRICS v1 -->`
+comment on that release PR with the table and the snapshot as JSON; without it nothing is written to GitHub.
+
+## Live A/B experiments (`--experiment <id>`, forge#4629)
+
+Declare one experiment in forge.yaml (flat keys under `experiment:`):
+
+```yaml
+experiment:
+  id: light-vs-full
+  arms: light,full          # at least two
+  ratio: 1,1                # integer weights, default equal
+  metrics: usd_median,wall_p50_ms,first_pass_rate,needs_human_rate   # default all four
+  targets: light.usd_median<=1.2xfull,light.first_pass_rate>=0.8     # arm.metric<=|>=limit, optionally x<other arm>
+  min_n: 10
+```
+
+The lean lane assigns each issue an arm (sha1 of id and issue number over the ratio, so a re-run keeps its arm) and tags
+`experiment` and `arm` on `phase_attempt` and `run_terminal`. `forgedock stats --experiment <id>` reports the finished
+runs per arm with the metrics of `--by version`; a metric under min n is null and so is any target that needs it, never a
+pass. `targets_met` is true only when every target is met. `--require-met` (with `--experiment`) prints the same report but exits 1 unless `targets_met` is true (0 otherwise; 2 without `--experiment`), so a post-deploy check can gate on it (#4634). The report, `scripts/bench-lightlane.mjs aggregate` and
+`scripts/eval-gate-scorecard.mjs` share one record: `{schema: "forge.bench.v1", source, arms, targets}`.

package/docs/TELEMETRY.md

@@ -5,6 +5,8 @@ counts can be measured without re-reading comments with a model. GitHub is the s
 by code (never by a model) into a `FORGE:TELEMETRY` comment, and everything can be rebuilt from the GitHub API alone.
 Local files are only a cache.
 
+For a stable, versioned stream aimed at outside consumers (JSONL sink, signed webhook) see [EVENTS.md](EVENTS.md).
+
 ## Where records live
 
 - One `<!-- FORGE:TELEMETRY v1 -->` comment per issue, or per PR for review-side events (`review_panel`,
@@ -41,7 +43,7 @@ is dropped. Payload keys are `[A-Za-z0-9_]+` and cannot shadow the shared keys;
 | `ci_local_check` | `scripts/ci-local.sh run`, one per scoped, widened or executed check (at most 20 per run) | `job`, `selector`, `selected`, `total`, `cache` = `hit` \| `miss` \| `dedup` \| `na`, `duration_ms`, `wait_ms` (job slot wait), `widen` (reason or `none`). A remote-CI failure at merge on a check recorded here as narrowed is a selector bug |
 | `liveness` | `scripts/remediate-liveness.sh` | `decision` = `PROCEED` \| `DEFER` \| `EXHAUSTED`, `why` |
 | `phase_attempt` | the engine (`bin/engine/telemetry-seam.mjs`, one per phase attempt; `phase` = the phase id) | `attempt`, `trigger` (#4362: why the attempt ran, one of `plan`, `build`, `block_correction`, `contract_correction`, `precheck`, `structure`, `docs`, `enumerating`, `sequence`, `ci_unavailable`, `ci_red`, `conformance`, `review`, `merge_ci`, `timeout_restart`, `recovery`; `restart_of` = the trigger a `timeout_restart` replaced), `model`, `backend` (`cli`/`api`), `turns`, `duration_ms`, `cost_usd` (the raw reported value: a running total when the attempt resumed a CLI session), `cost_usd_delta` (#4407: this attempt's cost; consumers sum it when present, else `cost_usd`), `cost_usd_session` (the CLI session's running total), `input_tokens`, `output_tokens`, `cache_read_tokens`, `cache_creation_tokens` (any value the runner did not report is omitted) |
-| `run_terminal` | the engine (`terminate()`, one per run segment) | `reason` (terminal reason), `code` (see `docs/OUTCOME-CODES.md`), `wall_ms` (this engine process, start to terminal) |
+| `run_terminal` | the engine (`terminate()`, one per run segment) | `experiment` and `arm` (#4629, only when forge.yaml declares an `experiment:`; also on `phase_attempt`), `reason` (terminal reason), `code` (see `docs/OUTCOME-CODES.md`), `wall_ms` (this engine process, start to terminal) |
 
 The engine records carry `run` = `r_<issue>_<start epoch seconds>` (one per `forgedock run-issue` process) so `forgedock stats`
 (`docs/STATS.md`) can group the attempts of a run and tell a finished run from an unfinished one. They are the GitHub-side

package/docs/site/install.md

@@ -0,0 +1,89 @@
+# Install
+
+Everything you need to install ForgeDock and keep it up to date. For a first run, `npx forgedock demo` needs no setup beyond the requirements below.
+
+**Requirements:** [Claude Code](https://docs.anthropic.com/en/docs/claude-code) or [OpenCode](https://opencode.ai/) · [GitHub CLI](https://cli.github.com/) (authenticated) · Node.js ≥ 18.
+
+```bash
+npx forgedock # checks your environment, installs commands into ~/.claude/commands/ (available in every Claude Code session on this machine), detects your repo, and hands you a reviewed forge.yaml
+```
+
+For OpenCode, install namespaced native commands without changing your provider
+or user-owned settings. When migrating an older ForgeDock adapter, it removes
+only exact ForgeDock-managed legacy entries from `opencode.json`:
+
+```bash
+npx forgedock opencode install
+# restart OpenCode, then: /forge/work-on <issue>
+```
+
+See [OpenCode support](../OPENCODE.md) for architecture, token-loading rules,
+current parity boundaries, and lifecycle commands.
+
+**Install is always global**, to `~/.claude/commands/`. `--global` is still accepted on the command line for backward compatibility but has no effect — there's no other install location to opt out of.
+
+> **Want engine-mode dispatch?** `npx forgedock` is transient — the `forgedock` binary isn't persisted in PATH after install. `/orchestrate` and `/autopilot` use agent dispatch mode by default, which is fully functional. To enable engine-mode dispatch (`forgedock run-issue`) with its durable phase table and fail-closed review gate, run `npm install -g forgedock` instead.
+
+One command does everything: it checks your environment, installs the slash commands into Claude Code, detects your repo (owner, branches, paths), and hands you a single annotated `forge.yaml` to review — press Enter to accept. Run `npx forgedock init` any time afterward to re-generate the config only.
+
+Installing also registers a SessionStart hook, so every Claude Code session
+in a forge-managed directory starts already knowing ForgeDock runs it.
+Per-directory control: `npx forgedock enable` / `disable` / `status`.
+
+Then just open Claude Code and run `/work-on <issue>`.
+
+> **Cost:** ForgeDock is free and open-source. It orchestrates sessions on **your** Claude account — no compute resold, no per-task markup. A typical `/work-on` run on a straightforward bug costs about what a 15–20 minute manual Claude Code session does.
+
+<details>
+<summary><strong>Other install options & commands</strong></summary>
+
+**Claude Code plugin marketplace** (Claude Code v2.1.143+):
+
+```
+/plugin marketplace add RapierCraftStudios/ForgeDock
+/plugin install forgedock@forgedock
+```
+
+Commands then appear as `/forgedock:work-on`, etc. You still run `npx forgedock init` to generate `forge.yaml`.
+
+**Headless / CI:** the pipeline also runs outside Claude Code. `npx forgedock run work-on <issue> --dry-run` previews the assembled prompt and tool plan. `npx forgedock run` picks an execution backend automatically (`--backend auto`, the default): if the Claude Code CLI (`claude`) is installed and already authenticated (Pro/Max subscription or a CLI-managed key), it drives the command through that — **no separate `ANTHROPIC_API_KEY` required**. It never falls back to the paid Anthropic API implicitly: without a CLI it fails with `CLI_BACKEND_UNAVAILABLE` unless the owner opted in (forge.yaml `execution.agent_backend: api` plus `FORGEDOCK_ALLOW_API_BACKEND=1`; see [CONFIG.md](../CONFIG.md#execution-optional)). Force either path explicitly with `--backend cli` / `--backend api` (or `FORGEDOCK_BACKEND=cli|api`) — the API backend is what CI environments without an interactive `claude` login should use. `npx forgedock run-issue <issue>` executes the same command specs on the durable engine (event-sourced run log, leases, crash-safe resume).
+
+**Brand-new project?** `/bootstrap "<idea>"` (or `npx forgedock bootstrap "<idea>" --dry-run` to preview) turns an idea into a validated, milestone-ordered GitHub backlog that `/orchestrate` can build. See [Start a New Project with ForgeDock](start-a-new-project.md).
+
+**Explicit install command:**
+
+```bash
+npx forgedock install           # installs into ~/.claude/commands/
+npx forgedock install --global  # same thing — --global is accepted but is a no-op
+```
+
+**Maintenance:**
+
+```bash
+npx forgedock update      # relink commands + refresh the SessionStart hook
+npx forgedock enable      # turn ForgeDock on for this directory
+npx forgedock disable     # turn ForgeDock off for this directory
+npx forgedock status      # show ForgeDock's state for this directory
+npx forgedock doctor      # installation health check with fix hints
+npx forgedock report      # 30-day pipeline impact receipts (--md for Markdown, --json for scripting)
+npx forgedock uninstall   # remove commands, the hook, and tracked copies
+npx forgedock help        # show everything
+```
+
+> Running `npx forgedock` from *inside* this repo uses the local working tree. From your own project, use `npx forgedock@latest` to pin the published release.
+
+</details>
+
+<details>
+<summary><strong>A note on install location</strong></summary>
+
+ForgeDock briefly experimented with a project-scoped-by-default install mode. It was backed out after causing a "split-brain" bug (`detect` assumed project-scoped while the installer still wrote globally — [#1589](https://github.com/RapierCraftStudios/ForgeDock/issues/1589)), so every version you'd realistically install today only ever writes to `~/.claude/commands/`. `--global` is still accepted as a flag for old scripts/muscle memory, but it changes nothing.
+
+If you're carrying an older `--global` habit in scripts or CI, it's harmless to leave it — `npx forgedock --global` and `npx forgedock` do exactly the same thing.
+
+</details>
+
+**After upgrading**: run `npx forgedock labels setup` once per repo. Repos created before the `workflow:remediating` state (forge#3541) lack that label; the pipeline now creates it on demand (best-effort), but the setup command is the supported way to sync all labels.
+
+
+

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "forgedock",
-  "version": "1.34.0",
+  "version": "1.36.0",
   "description": "Autonomous development pipeline for coding agents. GitHub as institutional memory, with native Claude Code and OpenCode entrypoints.",
   "bin": {
     "forgedock": "bin/forgedock.mjs"

package/scripts/bench-lightlane.mjs

@@ -31,6 +31,7 @@ import { tmpdir } from "node:os";
 import { join } from "node:path";
 import { pathToFileURL } from "node:url";
 import { detectApiShape } from "../bin/lean/apishape.mjs";
+import { benchRecord } from "../bin/experiment.mjs";
 
 export const ARMS = ["light", "full", "control"];
 export const TARGETS = Object.freeze({ cost_vs_control: 1.2, wall_vs_model: 1.3 });
@@ -225,7 +226,7 @@ export function aggregate(rows) {
     false_fix_le_control: t(L.n && C.n ? L.false_fix_rate <= C.false_fix_rate : null, L.false_fix_rate, C.false_fix_rate),
     zero_undetected_api_shape: t(L.n ? L.undetected_api_shape === 0 : null, L.undetected_api_shape, 0),
   };
-  return { arms, light_eligible: { ...le, control_cost_usd: ce.cost_usd.sum, ids: [...eligIds].sort() }, targets };
+  return { arms, light_eligible: { ...le, control_cost_usd: ce.cost_usd.sum, ids: [...eligIds].sort() }, targets, record: benchRecord({ source: "bench-lightlane", arms, targets }) }; // #4629
 }
 
 /** JSONL (or a JSON array) -> rows. */

package/scripts/eval-gate-scorecard.mjs

@@ -30,6 +30,7 @@
 import { readFileSync, writeFileSync } from "node:fs";
 import { dirname, join } from "node:path";
 import { fileURLToPath } from "node:url";
+import { benchRecord } from "../bin/experiment.mjs";
 
 /** Fields the fresh scorecard must have (matches eval-scorecard-aggregator.mjs's output). */
 export const REQUIRED_FIELDS = [
@@ -137,6 +138,12 @@ export function evaluateGate(
     fresh_run_id: fresh.run_id ?? null,
     baseline_run_id: baseline.run_id ?? null,
     reason,
+    // #4629: the shared result record; the target is null (never a pass) when the rate assertion was skipped
+    record: benchRecord({
+      source: "eval-gate",
+      arms: { baseline: { success_rate_pct: baseline.success_rate_pct }, fresh: { n: freshIssuesRun, success_rate_pct: fresh.success_rate_pct } },
+      targets: { success_rate_within_threshold: { pass: rateAssertionSkipped ? null : pass, value: dropPp, limit: thresholdPp } },
+    }),
   };
 }
 

package/scripts/file-deferred-followups.sh

@@ -10,6 +10,7 @@
 #
 # Usage: file-deferred-followups.sh --parent N --repo OWNER/REPO [--items FILE|-] [--folded N]
 #   items TSV, one per line: <path> TAB <why> TAB <known-flaw flag: known-flaw|1|yes|true, else empty>
+#   A post-deploy why starts `post-deploy: [check:<id>] ...` (#4626); the token is kept verbatim in the checklist line.
 # Env: DRY_RUN (non-empty: no create/edit, issue number 0), FORGE_TRAIL_TRUSTED_ASSOCIATIONS (default
 #      OWNER,MEMBER,COLLABORATOR), FORGE_TRAIL_TRUSTED_LOGINS (default empty), FORGE_DEFERRED_RETRY_SLEEP (secs).
 # Stdout: `path<TAB>issue` per item, then DEFERRED_TELEMETRY deferred_items=N folded=N batched_into=#M filed=N.

package/scripts/file-deferred-followups.test.sh

@@ -195,5 +195,13 @@ printf 'x\n' > "$ISSUE_BODY"; issue_json 77 closed OWNER; claim_json OWNER 77
 bash "$S" --parent 10 --repo o/r --items "$T/items.tsv" >/dev/null 2>&1
 check "dry-run: no comment/reopen/close/edit" "! grep -qE '^issue (comment|reopen|close|edit|create)' '$GH_LOG'"
 
+# 18. #4626: a post-deploy why's [check:<id>] token is kept verbatim in the checklist line
+reset
+printf 'x\n' > "$ISSUE_BODY"
+echo '{"items":[{"number":77,"author_association":"OWNER","user":{"type":"User"},"body":"<!-- FORGE:DEFERRED_FROM: #10 -->"}]}' > "$SEARCH_JSON"; issue_json 77 open OWNER
+printf 'R3: soak\tpost-deploy: [check:login-smoke] after a day\t\n' > "$T/check.tsv"
+bash "$S" --parent 10 --repo o/r --items "$T/check.tsv" >/dev/null 2>&1
+check "check token kept in the checklist line" "grep -q 'post-deploy: \[check:login-smoke\] after a day' '$ISSUE_BODY.edited'"
+
 echo "passed=$PASS failed=$FAIL"
 [ "$FAIL" -eq 0 ]