@juzser/blacksmith 0.2.0 → 0.4.0

package/factory/specs/roadmap.md

@@ -116,27 +116,11 @@ all; write `- error_issues: off` by hand to opt a project's tracker out.
 - epics: [factory-error-log]
 - goal: When the factory fails while building a project, open a GitHub issue in that project's repo. Requested by the operator 2026-09-04, and scoped by their answer to the fork the request contained: the errors reported are **the factory's own build-time errors** — a gate that returns `blocked`, a task that reaches `task_status: failed`, and every `error-logged` event — not runtime errors of the shipped product. **No code is injected into a generated project**: a project this factory builds carries no reporter, no dependency and no trace of Blacksmith beyond the "Built by blacksmith" line in its README, which is the whole point of the factory contract, so the issue is opened by the factory's own side through `gh issue create` against the project's remote. **Default ON, with a per-project disable switch** — a project that does not want its issue tracker written to says so once and the mechanism stays silent for it. Deliberately not scoped here and left to the plan: which of the three sources deduplicates against an already-open issue, what the issue body may carry given that event payloads can quote logs and diffs (`docs/standards/guardrails.md` "Secrets, keys, tokens" governs, and nothing may reach an issue unredacted), and whether the daemon or the run itself is the producer. Runs as its own epic after `phase-10` closes, by operator decision — not folded into that phase's plan.
 
-## Claude status bar — audit wave 1 (signing, hardening, Keychain ACL)
-- id: csb-audit-1
-- status: completed
-- epics: [csb-audit-1]
-- project: claude-status-bar-macos
-- kind: product
-- goal: The first epic cut from the four-axis audit of 2026-09-07 (45 findings, 0 S1, 9 S2, 29 S3, 7 S4; axes fixed by the operator as performance, code quality, architecture, security). Scope chosen by the operator: the security and build cluster only, because it is the one cluster that is already doing damage rather than only risking it, and because it touches no UI and no `AppState`, so it cannot regress the menu bar. Four items. **SEC-2 (S2)** `scripts/ensure-signing-identity.sh:56` calls `security set-key-partition-list -S apple-tool:,apple:,codesign: -s "$LOGIN_KEYCHAIN"`; `-s` is a predicate that takes no operand, so the keychain path is swallowed as a positional and the command rewrites the partition list of *every* signing private key in the login keychain, destroying `teamid:` partitions irreversibly — triggered by an ordinary `make app` on any machine that does not yet have the identity — the call sits inside the `find-certificate` creation branch opened at line 23 — which is every fresh machine, every new user account, and every keychain reset. The fix is to bound it with `-l "$SIGNING_IDENTITY_NAME"`. **SEC-3 (S2)** `scripts/make-app.sh:80` signs with neither hardened runtime nor entitlements, verified on the shipped artifact rather than inferred: `flags=0x0(none)`, `TeamIdentifier=not set`, `(no entitlements)`, and `DYLD_INSERT_LIBRARIES` honoured — which is what makes the Keychain ACL below borrowable by any process that can inject. **SEC-1 (S2)** `LiveCredentialWriter.swift:121-141` grants a Keychain ACL to a path fed to `SecTrustedApplicationCreateFromPath` with no signature check; two of three static candidates sit in user- or admin-writable locations, the dynamic fallback resolves the binary through an interactive login shell, and `LiveCredentialSelfHeal.swift:51-54` re-asserts the ACL on every launch, wake and unlock, so a bad grant is durable rather than a one-off. **ARCH-10 (S3)** is the same `make-app.sh` line read as a release concern — deprecated `--deep`, no `--timestamp`, no reproducibility — and lands here because `--options runtime --timestamp` plus dropping `--deep` closes it and SEC-3 in one change. Deliberately not in this epic and recorded so it is not lost: the "the bar lies to you" cluster (ARCH-1 hook read-modify-write with no lock, ARCH-2 the absolute path baked into settings.json, CQ-1 freshness never consulted on the menu bar, CQ-2 the 900s staleness applied to in-flight tool states) and the `AppState` extraction that ARCH-3/5/6 and CQ-3/8/9/11 and PERF-1 all reduce to. Also deferred here by operator decision: PERF-4's cheap variant — dropping the `PostToolUse` registration in `HookSettingsMerger.swift:8-9`, which halves the hook's per-tool-call cost but changes what the bar shows between tools, and is therefore a UI-visible change belonging with the UI cluster rather than with signing.
-
-## Claude status bar — per-path-class signing policy (SEC-1 follow-up)
-- id: csb-signing-policy-1
-- status: completed
-- epics: [csb-signing-policy-1]
-- project: claude-status-bar-macos
-- kind: product
-- goal: The successor epic `csb-audit-1` named when it closed (ledger `csb-audit-1-2026-09-07#338`, `successor_epic`), carrying the one S2 that epic left standing. `csb-audit-1` task-2d added `TrustedApplicationGate` in front of the Keychain ACL grant, but its single `policyRequirement` string admits any Developer-ID-signed binary on the machine as a trusted application for the credential item — far weaker than "the `claude` binary or this app", which is what the grant is for. The operator confirmed the finding and chose to land the gate as it was (better than no gate: it still refuses ad-hoc and unsigned binaries) and to fix the policy here rather than amend that plan a seventh time, because criteria 6 and 12 of task-2d had just pinned the requirement string and a new one needs its own specimens and its own grading. Measured design direction: per-path-class policy — for the `claude` binary, `anchor apple generic and identifier "com.anthropic.claude-code"` plus `certificate leaf[subject.OU] = "Q6L2SF6YDW"`; for the app's own self-signed bundle, the running process's own designated requirement (a pinned leaf-certificate hash is the fallback if a self-signed DR does not validate without a trusted anchor); a path in neither class is refused. The same epic closes the empty-ACL hole at both `SecAccessCreate` sites (a rejected write must leave the old ACL alone, never write an item that names zero trusted applications) and fixes the signing test harness so it never deletes the operator's `dist/` and escapes what it prints. Scoped like its parent: no UI, no `AppState`, no self-heal or locator files, and every agent bound by the same prohibition on mutating `/usr/bin/security` subcommands — the blast radius is the operator's real login keychain.
-
 ## Continuous loop — self-improve and on-command audit
 - id: continuous-loop
 - status: planned
 - epics: []
-- goal: Remove the operator from the middle of a run without removing them from the decisions. Requested 2026-09-07 and scoped in `docs/specs/continuous-loop-scope.md`, which splits it the way the request did. **Mechanism A — the factory's own self-improvement**, unattended, this checkout only, budget unbounded by operator decision: a `smith loop` process, separate from `smith daemon` and leaving that daemon's "never dispatches" refusal intact, which reads what `smith scheduler admit` classifies `auto` and spawns a headless `claude -p` session to carry each one through `/bs run`. The headless session is the missing link and the only new idea here — dispatch is a playbook needing the `Agent` tool, `smith` is a Node CLI that cannot call it, and a print-mode session is a real dispatching node that can (probed 2026-09-07: `bs=yes Agent=yes`). Unbounded budget lands as a per-epic override of `epic.cap_tokens`, the follow-up `budgets.yml` already asks for in its own comment, so one loop's decision does not widen every epic the factory will ever run. Local-only is enforced by a gitignored `state/loop/enabled` marker rather than promised in prose, and the self-improve path is excluded from `plugin-port` by name. **Mechanism B — `/bs audit <project>` and `/bs maintain <project>`** for projects outside the factory, on four axes fixed by the operator (performance, code quality, architecture, security), consolidating across axes before ranking because convergence is the strongest signal an audit produces, then stopping to ask the operator which findings become an epic. Triggered by a command, never by a timer — the operator's own amendment to the option they chose, and it removes the unattended-timer risk at no cost to what they wanted. Three things stay where they were: a person merges the PR, growth-review scope never auto-classifies, and a security-keyword surface always waits. **"Self-improve" is read wide, by operator decision 2026-09-07**: mechanism A may run the audit on the factory itself, cut epics from what it finds, and run them, rather than being limited to epics a person planned. That makes the merge the only human checkpoint between noticing and changing, so the scope prices it with four bounds — a severity floor of S1/S2-or-cross-axis-convergence on what may be auto-cut, one epic per PR per human merge, growth-review still never auto, and a protected set of declarations (`scheduler.yml`, `autonomy.ts`, `budgets.yml`, `delegation.yml`, `.claude/settings.json`, the enable marker) that the loop may propose changes to but never auto-dispatch, because a process allowed to widen its own leash has no fixed point. Four forks stay open and none blocks the epic — the loop interval, what a failed child does, `--max-turns` for the headless session, and the format of the durable findings file that keeps a wide loop from re-cutting the same epic every run.
+- goal: Remove the operator from the middle of a run without removing them from the decisions. Requested 2026-09-07 and scoped in `docs/specs/continuous-loop-scope.md`, which splits it the way the request did. **Mechanism A — the factory's own self-improvement**, unattended, this checkout only, budget unbounded by operator decision: a `smith loop` process, separate from `smith daemon` and leaving that daemon's "never dispatches" refusal intact, which reads what `smith scheduler admit` classifies `auto` and spawns a headless `claude -p` session to carry each one through `/bs run`. The headless session is the missing link and the only new idea here — dispatch is a playbook needing the `Agent` tool, `smith` is a Node CLI that cannot call it, and a print-mode session is a real dispatching node that can (probed 2026-09-07: `bs=yes Agent=yes`). Unbounded budget lands as a per-epic override of `epic.cap_tokens`, the follow-up `budgets.yml` already asks for in its own comment, so one loop's decision does not widen every epic the factory will ever run. Local-only is enforced by a gitignored `state/loop/enabled` marker rather than promised in prose, and the self-improve path is excluded from `plugin-port` by name. **Mechanism B — `/bs audit <project>` and `/bs maintain <project>`** for projects outside the factory, on four axes fixed by the operator (performance, code quality, architecture, security), consolidating across axes before ranking because convergence is the strongest signal an audit produces, then stopping to ask the operator which findings become an epic. Triggered by a command, never by a timer — the operator's own amendment to the option they chose, and it removes the unattended-timer risk at no cost to what they wanted. Two things stay where they were: growth-review scope never auto-classifies, and a security-keyword surface always waits. The third moved on 2026-09-18 — the operator extended the standing merge authorisation to the unattended loop, so the loop may merge its own PR, and §3.5 now spends the rest of its length on what replaces the person: six conjunctive conditions, none of them enforced by code until this epic writes them. **"Self-improve" is read wide, by operator decision 2026-09-07**: mechanism A may run the audit on the factory itself, cut epics from what it finds, and run them, rather than being limited to epics a person planned. That made the merge the only human checkpoint between noticing and changing, and once the merge became conditional too the remaining bounds carry the whole argument — a severity floor of S1/S2-or-cross-axis-convergence on what may be auto-cut, one epic per PR per merge decision rather than a batch, growth-review still never auto, and a protected set of declarations (`scheduler.yml`, `autonomy.ts`, `budgets.yml`, `delegation.yml`, `.claude/settings.json`, the enable marker) that the loop may propose changes to but never auto-dispatch, because a process allowed to widen its own leash has no fixed point. All four forks the scope left open were closed 2026-09-10 and the epic is ready to plan. **No interval** — iterations run back-to-back, because the loop works a queue rather than sampling a condition, and a queue that is empty costs one `scheduler run --dry`. **A failed child reuses `budgets.yml`'s `escalation_ladder`** one tier up rather than inventing a second retry policy: retry once at the same tier, escalate with the reason logged, then stop and report. **`--max-turns 300`** guards the headless child against a runaway, and a cut must surface as an `error-logged` event — which the implementing epic has to pay for first, because `taxonomy.yml` has no class for a turn-cap cutoff today (`execution` covers test/tool/env failures, `economy` covers budget, diff-cap and context overruns), so the error the guard raises is currently unwritable. **Findings live in `<project>/.blacksmith/findings.jsonl`**, append-only, deduped by fingerprint, with `declined` expiring after 90 days so a decision made about one version of a file does not silence the auditor forever — and it is explicitly not a waiver store: it records what an audit saw, never permission to ship past a gate.
 
 ## Blacksmith as a Claude Code plugin
 - id: plugin-port
@@ -144,11 +128,74 @@ all; write `- error_issues: off` by hand to opt a project's tracker out.
 - epics: []
 - goal: Make Blacksmith installable without cloning it — a Claude Code plugin carrying `/bs`, the agent templates and the guard hook, driven by a `smith` CLI published to npm and invoked through `npx` at a pinned version. Requested by the operator 2026-09-07 and scoped, before any path is moved, in `docs/specs/plugin-port-scope.md`. Three decisions are already taken there: the CLI ships through npm rather than as a vendored build; writable state moves to `<project>/.blacksmith/` rather than to a per-slug directory under the user's home; and the spec is written and reviewed before code. The work the scope measures is mostly one change — `factory/orchestrator/src/paths.ts` anchors shipped assets, runtime state and operator declarations on a single `REPO_ROOT`, and a plugin has to tell those three apart across 32 call sites in 8 modules — plus its sharpest consequence, that `guardrails.yml`'s `write_roots` are repo-root-relative literals matched segment-wise by `policy.ts`, so relocating state removes every judge's permission to write its own verdict unless the policy, the matcher and the judge templates move together. Five forks stay open for the operator and are why no epic is cut yet: the npm name, one repo or two, whether this clone migrates its own state, where the deny-list lives once `.claude/settings.json` is no longer the delivery mechanism, and whether the dashboard is in the first release at all. Docker and exposing the factory as a service were asked for in the same conversation and answered no, with the argument recorded in that scope's "Out of scope": dispatch is a playbook an interactive session executes, not code a container could run.
 
-## vam — audit wave 1 (error formatting, poll back-pressure, adapter boundary)
-- id: vam-audit-1
+## Audit store — retiring a finding the code has outgrown
+- id: audit-finding-retire
+- status: planned
+- epics: []
+- goal: Give `/bs audit` a way to retire an `accepted` finding that the code has moved out from under, and stop the store from suppressing a defect it no longer describes. Found 2026-09-18 while running the performance axis of a third audit on a project whose tree had been reorganised between audits: three of the four `accepted` findings in that store named files that no longer exist, and a fourth was moot as literally written because the mechanism it criticised had been replaced wholesale. None of them could be retired. `smith audit decide` takes only `accept|decline|merge`, and `audit resolve` requires `--epic`, so `fixed` is reachable only as a side effect of an epic landing — which means a finding fixed by an unrelated refactor, by a deletion, or by a rewrite nobody cut an epic for stays `accepted` forever. That is not merely untidy: `accepted` suppresses re-raise, so a stale row is an auditor permanently blinded to the file it names, and the store's one guarantee — that `fixed` never suppresses, so a regression re-raises — is quietly void for every defect that was never closed through an epic. Four things stay open for the plan. **Who decides** — an operator verb (`audit retire`, with a required note, symmetric with `decide`), or a check the auditor itself performs when it is handed the dedupe context and finds the cited path absent. **What the terminal status is** — reusing `fixed` says a thing that may be false (nobody verified a fix; the file merely vanished), so a distinct status such as `stale` or `superseded`, which behaves like `fixed` for suppression and unlike it for provenance, is probably right. **Whether a moved file is retirement or re-anchoring** — a rename is not a fix, and a store that retires on `file_path` alone will drop live defects every time a project reorganises, so the cheap `git log --follow` answer belongs in the design rather than in the operator's head. **Whether `audit open` should report it** — the audit that surfaced this had to be told by hand; `open` already prints the live findings, and printing which of them name paths that no longer exist costs one stat per row and turns a silent hole into a visible one. Nothing here is a code fix yet; the mechanism is what needs designing, and the gap was recorded the day it was found so it is not rediscovered by the next audit that trips over it.
+
+## Coverage gate — telling an excluded file from an unmeasured one
+- id: coverage-excluded-status
+- status: planned
+- epics: []
+- goal: Teach the coverage gate the difference between a file nobody measured and a file the project decided on purpose not to measure, so a deliberate exclusion stops reading as a failure. `coverageSubjects()` in `factory/orchestrator/src/coverage.ts` returns exactly three statuses — `measured`, `unmeasured`, `not-instrumented` — and none of them is "excluded by configuration". It infers the middle one from directory neighbours: a claimed path with no row in the summary whose directory holds other instrumented rows is reported `unmeasured`, "the run did not measure the file the criterion names", which the gate treats as a block. A file named in a vitest config's `exclude` list while its siblings are instrumented is exactly that shape, so the classifier cannot tell an intentional exclusion from an omission, and the gate fails the task either way. Found 2026-09-18 by the wave-runner for `harness-codex-dispatch` task A: `factory/orchestrator/src/cli.ts` is excluded by name at the repo-root `vitest.config.ts:50`, a decision predating the task and documented at `factory/orchestrator/test/cli.test.ts:13-15` — thin argv-to-module wiring, excluded from the coverage floor like UI glue per `stack.md`, verified end-to-end as a built binary instead. The gate blocked on `coverage-evidence` regardless; three coder rounds could not clear it, and a diagnostic run with the exclusion lifted still reported 0%, because `cli.ts` exports nothing (`main()` self-invokes at module scope, line 3858) and only ever runs as a spawned child process, so vitest's v8 provider never instruments it in-process. The task was parked rather than waived: a waiver would record "we accepted missing coverage" where the truth is "coverage was never owed here", and the waiver store is the signal the factory reads to decide what to fix, so routing a tool defect through it buries the defect and corrupts the signal. Not a one-off — `ui/vitest.config.ts` excludes `ui/src/lib/api.ts` on the same reasoning, so the next task claiming that file meets the same wall. Three things stay open for the plan. **Where the exclusion list comes from** — parsing the vitest config is the obvious answer and the wrong shape (this repo has four of them, they are TypeScript rather than data, and a claim does not know which one governs it), so resolving it from the coverage run's own metadata, or declaring excluded paths once in a policy file the gate already reads, are the alternatives worth pricing. **Whether `excluded` passes or abstains** — a fourth status that silently passes turns every future exclusion into an invisible hole in the floor, so the gate probably wants to pass the criterion *and* print the exclusion's justification as the evidence, rather than say nothing. **Whether the real fix is upstream** — the cheapest version of this bug is one that never reaches the gate: `plan validate` or `claims impact` could refuse a coverage-evidence criterion naming a path the coverage config excludes, so the planner cannot write an unsatisfiable criterion in the first place, and the gate keeps its three statuses. **The first of those three is now answered by measurement rather than argument.** `createVitest('test', { watch: false, run: false })` from `vitest/node` returns the *resolved* coverage config as data: run against this repo on 2026-09-18 it printed an `exclude` list led by `factory/orchestrator/src/cli.ts` — the config's own entries merged with vitest's defaults — beside the `include` glob, once for every entry in `v.projects`, each carrying its own `root`. So the config never has to be parsed by hand: vitest resolves its own TypeScript and hands back a list. That also settles "which of the four governs this claim": the governing project is the one whose `root` and `include` glob the claimed path falls under, and a claim under none of them is the `not-instrumented` the classifier already reports correctly today. What that does *not* settle is where the call may run: `vitest` is a devDependency here and the published package ships six runtime dependencies and `dist`, so the shipped `smith` cannot import it, and a project this factory gates may not use vitest at all. The resolution therefore belongs on the project's side of the boundary — resolved from the gated worktree's own `node_modules` by an import that degrades to today's behaviour when absent, or emitted by the project's own coverage run as data the gate reads. Both are cheaper than a parser and both leave the factory's dependency graph alone, and the first of the two is measured as well: on 2026-09-18 `createRequire` rooted at a project's own manifest resolved `vitest/node` from a process outside that project, through pnpm's content-addressed store, for this repo's vitest 5 and for a sibling project's vitest 4 alike — both exporting `createVitest` — and raised a plain module-not-found for a directory with no vitest at all, which is the entire degradation path in one catch. So the mechanism works across major versions and fails legibly, and its price is measured too: asking a project's own vitest for its resolved exclusion list, from outside that project, cost 247ms on a cold process and 15ms once warm on 2026-09-18 — of which the import of the module is 180ms and the call itself 64ms — and returned nineteen entries, the config's own two merged with seventeen defaults. A gate whose own suite runs for four minutes does not need to argue about a quarter of a second, so question one is answered end to end: the list is data, the project's own tool hands it over, the absence case is one catch, and the cost is noise. The second question — whether an excluded path passes or abstains — stands unchanged. The third is now sharper, and points the other way from the wording above: `smith plan validate` takes one plan file and hands it to `validatePlan` with no options at all, and those options carry a specs directory, a taxonomy and a schema set, never a project, so refusing an unsatisfiable criterion there means widening that signature until it can reach a checkout. `smith claims impact --plan` already stands in one: it builds its symbol graph from `--repo`, defaulting to the repo root, precisely because a claim says what a task may write and only the tree says what those files import today. The same `createRequire` resolution therefore works there unchanged. So the upstream refusal is cheap at `claims impact` and expensive at `plan validate` — but cost is the smaller half of it, because the two verbs do not run at the same time. `plan validate` is the mandatory checkpoint: the plan playbook requires a valid verdict before anything else touches a plan, which is the one moment early enough to stop an unsatisfiable criterion from being written down. The wave calls `claims impact` per task at step 10, against a worktree, *after* the gate has already returned its outcome — too late to save the gate run it was meant to save — and the plan-wide form of it is documented as an operator read, not a required one. So the verb that could refuse cheaply runs too late and is optional, and the verb that runs at the right moment is the one with no checkout to resolve against. That is the shape of the third question, and it is not the cheap fix the wording above suggests.
+
+## Policy rules that test themselves
+- id: policy-self-test
+- status: planned
+- epics: []
+- goal: Make every rule in `factory/policies/*.yml` carry the examples it was written for, and fail at load when it stops matching them. Today a rule is prose plus a pattern, and nothing anywhere asserts that the pattern still catches the thing the prose describes — a rule narrowed by an unrelated edit goes on loading cleanly and denying nothing, which is the one failure mode a guard cannot report about itself. `smith policy check` exists but answers a different question: it probes one command the operator types (`--command <cmd>`, exit 1 on deny), so it tells you what the guard does now, never that what it does is still what it was written to do. Verified 2026-09-18: no file under `factory/policies/` carries a `match`/`not_match` example list. The shape borrowed from OpenAI Codex's `execpolicy`, where each rule ships the strings it must match and the strings it must not, and the policy is a test suite the loader runs. Three things stay open for the plan. **Where the examples live** — beside each rule in the same YAML is the version that cannot drift, but it inflates files that agents read on demand, so a sibling `*.examples.yml` is worth pricing against that. **What happens on conflict** — two rules matching one string need a declared winner, and strictest-wins is the only rule that fails safe, but it has to be written down before it can be relied on. **Whether the check is a gate or a test** — running at policy load makes every CLI invocation pay for it and makes a bad edit unmissable; running as a vitest file makes it free at runtime and skippable in exactly the situation where someone is hand-editing a policy. Smallest change on this list and the one that protects the others, because several of them add policy surface.
+
+## The denominator — proving the factory got better
+- id: agent-bench
 - status: planned
-- epics: [vam-audit-1]
-- project: vam
-- kind: product
-- goal: Cut from audit 20260914-5fe9088c of `vam` (3 accepted findings: 3 S3-minor). Each finding below is one acceptance criterion; the plan decides the tasks. **6acbe3a3 (S3-minor, performance)** `src/adapter/useCanvas.ts` — The 4-second poll loop fires the next overview+timeline fetch on a fixed timer with no in-flight guard, so a slow or degraded black-smith backend causes concurrent request piles that never drain, and an explicit refresh() (fired on every write) stacks more on top instead of coalescing.. **4de3c5cb (S3-minor, architecture)** `src/panels/ReviewQueue.tsx` — ReviewQueue.tsx and canvas/actions.ts import ApiFinding/ApiLesson wire-format types directly from adapter/api.ts instead of a domain-layer abstraction, breaking the project's own consistently-applied rule that no UI component learns which system (black-smith) it is looking at.. **7d3536a4 (S3-minor, code-quality)** `src/canvas/Canvas.tsx` — The same error-unwrapping ternary (SmithApiError -> code:message, else Error -> message, else String(cause)) is written out verbatim twice in one file instead of being factored into a shared helper..
+- epics: []
+- goal: Give the factory a way to answer "did that change make it better?", which it currently cannot answer at all. Every other improvement on this roadmap is a hypothesis without this, and the factory has been changing itself for months on the strength of arguments rather than measurements. Four pieces, in dependency order. **Machine-gradable acceptance criteria**: SWE-bench's `grading.py` splits a task's tests into `FAIL_TO_PASS` (must go green, or the work did nothing) and `PASS_TO_PASS` (must stay green, or the work broke something), and grades FULL/PARTIAL/NO mechanically. The factory's acceptance criteria are prose a judge reads, which is why grading them costs a model call and why two runs of the same task are not comparable; a criterion that names test ids is gradeable by a machine, for free, identically every time. **A resumable golden-task harness**: a fixed corpus of tasks the factory has already done, re-run on demand, with a log directory and completed-sample reuse so an interrupted bench resumes instead of restarting, and epochs so the same task runs K times — the shape of Inspect's `eval_set()`. **The statistics to read it honestly**: at the N a local factory can afford, most differences are noise, and "Adding Error Bars to Evals" (arXiv:2411.00640) gives the arithmetic — paired differences on the same tasks rather than independent means, K repeats to separate run-to-run variance from task variance, and the power calculation that says when a result is unclaimable. The discipline this buys is the refusal: a 5% improvement over 12 tasks is not an improvement, and the bench should say so rather than let an agent report it. **A version stamp on every dispatch**: verified 2026-09-18 that nothing in `factory/orchestrator/src/` or `factory/policies/` records a template, prompt or playbook version on a `dispatch_decision` — so two runs a month apart are not known to have used the same agent template, and no before/after comparison of the factory is sound even in principle. One hashed field on the dispatch event, the way Temporal's `patched()` marks which code version a replay ran, and the bench gains a dimension it cannot otherwise have. Two things stay open. **What the corpus is** — replaying closed epics is free and realistic but leaks its own answers into the lessons file; hand-cut tasks are clean and expensive. **What the metric is for a task with no tests** — most factory work is prose, policy and playbooks, and FAIL_TO_PASS does not reach it. Researched 2026-09-18 across SWE-bench, Inspect and Temporal.
 
+## Recovery when an agent is capped or dies
+- id: agent-resume
+- status: planned
+- epics: []
+- goal: Make a capped or dead agent a state the factory recovers from mechanically, rather than a forensic exercise for whoever notices. This is the most-hit gap in the factory's daily operation: `budgets.yml` caps every role in turns, the cap fires regularly, and what happens next today is that a human reads the branch, guesses what the agent had done, and hand-writes a continuation brief. The same day this entry was written, a wave-runner died mid-wave and the orchestrator that took over had nowhere to put its dispatches except the dead agent's own session log, which `smith delegation check` then correctly reported as five violations of one-session-one-author (recorded at `harness-codex-dispatch-w1-2026-09-18#83`). Four pieces. **A declared checkpoint the agent takes before the cap, not after**: Temporal's Continue-As-New ends a workflow deliberately and starts a fresh one with typed carry-over, and `continueAsNewSuggested` is the pressure signal telling it when — the equivalent here is a turn-budget signal in the agent's own context plus a declared handoff schema it fills in voluntarily, so the continuation is written by the agent that has the context rather than reconstructed by one that does not. **A progress cursor on the event log**: Temporal's Activity Heartbeats carry details that are handed to the *next* attempt, so a retry resumes rather than restarts; heartbeat events here would make a capped agent's state a read of the log instead of a grep of the branch. **A resume path**: render a lost session from the durable log with a head/tail truncation policy and an explicit resumed-conversation marker, so the recovered context is honest about being partial. **Condensation as a first-class event**: the OpenHands SDK folds context with a tombstone in the log rather than by dropping messages, with `keep_first`, a minimum-progress floor and atomic-unit snapping so a fold never splits a tool call from its result — which is the difference between a replayable task and a hand-patched one. A fifth, cheaper piece worth carrying in the same epic: **stuck detection** over the event stream — repeated identical actions, alternating edits, repeated identical errors — with a one-shot nudge before the budget gate catches the same futility as exhaustion. Two things stay open. **Whether a takeover is a new session or a new actor in the old one** — `session start --continues <session>#<n>` already exists and keeps the causal chain, and is probably the whole answer, but it is not written down anywhere an orchestrator will find it mid-incident. **Whether a turn-cap cutoff is writable at all** — `taxonomy.yml`'s `execution` group has no class for it, which the continuous-loop milestone already records as a debt it must pay first; adding one is an operator decision under AGENTS.md's human gates, so this epic proposes and does not apply. Researched 2026-09-18 across Temporal, the OpenHands SDK and DBOS.
+
+## Verification scoped to what the task claimed
+- id: claims-verification
+- status: planned
+- epics: []
+- goal: Make the gates measure the lines the task actually claimed, instead of measuring whole files and hoping. The factory already has the scoping signal every surveyed tool lacks — a task declares its `claims` before it starts — and spends it on almost nothing at verification time. Four pieces, cheapest first. **Patch coverage over claimed paths**: the tier below mutation testing and adoptable immediately — the question "were the lines this diff touched covered" is answerable from the coverage summary the gate already reads, and it is a better question than the current per-file floor because it cannot be satisfied by a well-covered file the task did not change. **Mutation score over claimed line ranges**: Stryker's `--mutate` takes exactly the range shape `claims` already carries, and `thresholds.break` turns it into a gate — this replaces "the file was measured" with "no mutant inside your claimed lines survived", which is the first coverage signal here that a green-but-assertionless test cannot pass. Expensive per run, which is why it belongs scoped to a diff rather than to a suite. **A committed API-surface report**: API Extractor's `.api.md` is a checked-in rendering of the public surface, so a contract change shows up as a diff in review rather than as a red at the end of a serial merge queue — and a change that stays entirely inside a task's claims is exactly the one no reviewer is looking for. **Scope enforced at the tool boundary**: compile a task's `claims` into a per-worktree deny rule so an edit outside them is refused when it is attempted, not reported after the fact by `smith claims check`, and make the refusal name the claim it violated so the agent can propose a plan amendment instead of guessing. This milestone is adjacent to `coverage-excluded-status` and should be planned after it or with it: that one teaches the gate to tell an exclusion from an omission, and this one narrows what the gate looks at in the first place — done in the wrong order, the exclusion problem simply moves. Two things stay open. **Whether mutation testing is affordable per task or only per epic** — the honest answer depends on suite runtime and nobody has measured it here. **What a claim on a non-code path means** — a task claiming a playbook or a policy file has no lines to mutate and no coverage to report, and silently passing those is how a gate becomes decorative. Researched 2026-09-18 across Stryker, Codecov, diff-cover and API Extractor.
+
+## Merge lanes — parallel where the claims are disjoint
+- id: merge-lanes
+- status: planned
+- epics: []
+- goal: Stop paying serial cost for work that does not overlap. Verified 2026-09-18 at `factory/orchestrator/src/cli.ts:2079-2092`: `queue run` iterates admitted tasks in a `for…of`, awaits `step` on each, and breaks on the first outcome that is not `merged` — so N tasks cost N full test runs end to end, and one red task stops every task behind it regardless of whether they touch the same code. For a wave of four independent tasks on a suite that takes four minutes, that is sixteen minutes of wall clock to learn what could have been learned in four. The fix the CI world converged on is speculative merging: Zuul tests each change against the assumed success of everything ahead of it and reparents the tail when one fails (NNFI — "nothing not fixed"), Mergify bisects a failed batch to find the culprit, and Aviator has changes declare which targets they affect. Blacksmith holds a stronger card than any of them: `claims` are declared before the task runs, so disjointness is known up front rather than inferred from a file list after the fact, and two tasks with non-overlapping claims can be tested in genuinely parallel lanes rather than speculatively stacked. The speculative machinery is then only needed for the overlapping tail. Three things stay open. **What disjointness means when claims are directory-shaped** — two tasks claiming the same directory with different files inside it are disjoint in fact and overlapping as written, and the answer decides how much parallelism this actually buys. **Where the lanes run** — a lane is a worktree and a test process, and a laptop running four vitest suites against one disk is not four times faster; the ceiling is worth measuring before the scheduler is built. **What a failed lane does to its siblings** — under disjoint claims the answer should be nothing at all, and that is precisely the claim the epic has to earn rather than assume, because a disjointness bug that lets a bad merge through is worse than the serial cost it saved. Researched 2026-09-18 across Zuul, Mergify and Aviator; bors-ng examined and dropped, its own README now redirecting to GitHub's merge queue.
+
+## Judging economics — spend the model call where it decides something
+- id: judge-economics
+- status: planned
+- epics: []
+- goal: Stop spending judge calls on outputs that a free check could have rejected, and stop grading a capped output as though it were a finished one. Four pieces, independent enough to land separately. **Free critics ahead of paid judges**: an empty diff, a missing structured completion, and claims touched without a single test added are all decidable without a model, and each one currently reaches a judge that pays full price to say so. The OpenHands SDK runs exactly this tier first and classifies what it rejects as behaviour, infrastructure or unclear-request — a taxonomy worth having because "the agent failed" and "the environment failed" are different bugs with different owners, and today they land in the same place. **Grade a budget-capped output as worth less**: SWE-agent's reviewer discards out-of-range verdicts and tie-breaks on cost; here the sharper version is that an output produced by an agent that hit its turn cap is a partial output, and a judge that does not know that is being asked to grade a fragment as a whole — which is how a capped run earns a passing verdict for the third of the task it finished. **Submit-gating and an honest forfeit**: show the worker its own diff before it may report done, with a checklist that names the mistakes workers actually make — test files touched, debug output left, claims exceeded — and give it a structured way to give up. A forfeit is strictly more useful than a confident wrong answer: it costs one round instead of a judge round plus a verifier round plus a rework round, and it is information the lessons pass can learn from. **Budget as an admission check**: `min_budget_for_dispatch` per role, so a role is never dispatched into a budget that cannot finish it, and a ring-fenced judging reserve so a capped epic still gets graded rather than spending its last tokens on more coding. Two things stay open. **Whether a free critic may block or only warn** — a false rejection that costs a whole round is worse than the judge call it saved, so the first critics should probably warn while their precision is measured against `agent-bench`. **What a forfeit does to the escalation ladder** — `budgets.yml` already declares one, and a forfeit is either its first rung or a separate path, but it must not become a cheap exit an agent prefers to the work. Researched 2026-09-18 across SWE-agent and the OpenHands SDK.
+
+## Self-audit 1 — event-log append cost, rename-blind claim diffs, one git entry point
+- id: bs-audit-1
+- status: planned
+- epics: [bs-audit-1]
+- project: black-smith
+- kind: factory
+- goal: Cut from audit 20260925-f336f5e4 of `blacksmith` (7 accepted findings: 3 S2-major, 3 S3-minor, 1 S4-nit). Each finding below is one acceptance criterion; the plan decides the tasks. **d7c644db (S2-major, performance)** `factory/orchestrator/src/events.ts` — appendEvent() re-reads and re-JSON.parses the entire session event log on every single append (readEventsAtPath inside appendEventLocked), while holding both an in-process queue and a cross-process poll lock, so writing to a session log is O(n) per event and O(n^2) for a session of n events.. **0f3f9c23 (S3-minor, performance)** `factory/orchestrator/src/worktree.ts` — listStale() spawns one synchronous `git merge-base --is-ancestor` subprocess per worktree entry in a loop instead of a single batched check, so merge-queue/worktree cleanup cost scales linearly with subprocess spawns rather than with one git invocation.. **ae8cb69e (S2-major, architecture)** `factory/orchestrator/src/queue.ts` — The queue module shells out to git directly with execFileSync instead of going through git.ts, the module the codebase documents as the one place this factory shells out to git, so queue git failures bypass credential redaction and the structured GitCommandError type.. **fd03e000 (S3-minor, architecture)** `factory/orchestrator/src/diffstat.ts` — diffstat.ts calls execFileSync('git', ...) directly for rev-parse/diff/branch instead of through git.ts, duplicating the single git-invocation boundary the codebase declares for itself and losing that module's credential redaction and GitCommandError typing.. **160cc3d7 (S3-minor, architecture)** `factory/orchestrator/src/commit.ts` — commit.ts, which certifies whether a task's work is safe to integrate, runs its own execFileSync('git', ...) calls rather than routing through git.ts, so the one module the codebase names as the sole git entry point has at least two siblings that re-implement direct git shell-outs.. **07c5d1b0 (S2-major, code-quality)** `factory/orchestrator/src/claims.ts` — collectCommittedChanges, the collector behind postRunCheck (the claim-confinement check for a task worktree), runs `git diff --name-only` without `--no-renames`, so a rename that crosses a claim boundary is reported only by its destination path.. **ad9ea9ba (S4-nit, code-quality)** `factory/orchestrator/src/waveNext.ts` — The one-line helper nameList (ids.join(', ')) is duplicated verbatim in waveNext.ts and waveSchedule.ts rather than shared, so the two wave-scheduling files carry two independent copies of the same id-list formatting rule..
+
+## Audit 2: bounded judges, parallel feedback, dashboard origin guard
+- id: bs-audit-2
+- status: planned
+- epics: [bs-audit-2]
+- project: black-smith
+- kind: factory
+- goal: Cut from audit 20260925-49bffa9b of `blacksmith` (5 accepted findings: 2 S2-major, 3 S3-minor). Each finding below is one acceptance criterion; the plan decides the tasks. **50acc356 (S3-minor, performance)** `factory/orchestrator/src/daemon.ts` — runTick() re-reads and re-JSON.parses every session's full .jsonl event log from disk on every daemon tick, with no incremental/fingerprint caching, so the daemon's per-tick cost grows linearly and unboundedly with the factory's total historical event volume.. **d50d4c1f (S3-minor, performance)** `ui/server/src/app.ts` — Every dashboard write route that omits an explicit causalParent (waivers apply-batch, lessons approve/reject/edit) triggers a full read-and-parse of that session's entire event log via resolveContext()/lessonContext() just to read the id of the last event.. **4c610e7c (S2-major, security)** `ui/server/src/app.ts` — The dashboard's write endpoints (waiver apply-batch, lesson approve/reject/edit) have no CSRF protection: there is no Origin/Referer check and Hono's `c.req.json()` parses the request body as JSON regardless of the Content-Type header, so a cross-origin `text/plain` form POST (a CORS-exempt 'simple request' that needs no preflight) is accepted and executed.. **1b8fe398 (S3-minor, architecture)** `.claude/skills/bs/wave.md` — The wave playbook's own loop (steps 2-10) never invokes the one instrument that measures whether a wave actually ran in parallel (auditWaveConcurrency/`smith wave check`, factory/orchestrator/src/waveConcurrency.ts), so the factory's central scaling claim goes unchecked for the entire time it could still be corrected.. **7ef3cd2f (S2-major, architecture)** `.claude/skills/bs/dispatch.md` — The one hard cap that bounds every subagent's work (`maxTurns`, enforced by the Claude Code harness itself) is transmitted into each dispatch prompt purely as prose the dispatching agent must copy correctly by hand from `.claude/agents/<role>.md` 'at dispatch time' — no module in factory/orchestrator/src validates that a dispatch's stated turn budget matches the template the harness will actually enforce, so the one enforcement path for this invariant is a human/LLM re-reading a file correctly every single time.. **Operator-added scope (2026-09-25, from GitHub research, ideas only)**: (6) a `SubagentStop` hook in the judge-class agent templates' frontmatter that blocks a judge from ending while its declared artifact file is absent — complements `agent-resume`, which handles the cap itself; (7) planning emits parallel width mechanically — at plan ingest, tasks with disjoint claims and no dependency edge are marked parallel and the plan's achievable width is reported, and a small epic takes a right-sized planning path with a bounded review focus. The per-task pipeline (no wave barrier) is deliberately out of scope until a pros/cons analysis is decided.
+
+## Self-audit 3: UI read boundary and fail-closed daemon state
+- id: bs-audit-3
+- status: planned
+- epics: [bs-audit-3]
+- project: black-smith
+- kind: factory
+- goal: Cut from audit 20260927-b61eaa7e of `black-smith` (2 accepted findings: 2 S3-minor). Each finding below is one acceptance criterion; the plan decides the tasks. **871ee8f7 (S3-minor, architecture)** `ui/server/src/app.ts` — lessonSession() builds its own drizzle query directly against the `lessons` schema table instead of going through db/queries.ts, the module the file's own header comment declares as the single 1:1 wrapper boundary between the UI and the projection.. **442cd7f2 (S3-minor, code-quality)** `factory/orchestrator/src/daemon.ts` — daemon.ts defines its own readJsonFile() that silently swallows any parse/read error and returns null, diverging from the fail-closed readJsonFile() in cli.ts that throws on malformed JSON, so a corrupted daemon lock/status/memory file is treated as absent rather than reported..

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@juzser/blacksmith",
-  "version": "0.2.0",
+  "version": "0.4.0",
   "type": "module",
   "description": "Autonomous agent factory — loop runner and worktree engine.",
   "keywords": [
@@ -80,22 +80,22 @@
     "yaml": "2.9.1"
   },
   "devDependencies": {
-    "@biomejs/biome": "2.5.13",
+    "@biomejs/biome": "2.5.14",
     "@hono/node-server": "2.1.1",
     "@playwright/test": "1.63.0",
     "@tailwindcss/vite": "4.3.3",
     "@types/better-sqlite3": "9.6.0",
-    "@types/node": "26.5.1",
-    "@vitejs/plugin-vue": "6.0.8",
-    "@vitest/coverage-v8": "5.0.0",
+    "@types/node": "26.6.1",
+    "@vitejs/plugin-vue": "6.0.9",
+    "@vitest/coverage-v8": "5.0.1",
     "@vue-flow/core": "1.48.2",
     "drizzle-kit": "0.31.10",
-    "hono": "4.13.7",
+    "hono": "4.13.8",
     "tailwindcss": "4.3.3",
     "typescript": "7.0.2",
     "vite": "8.3.0",
-    "vitest": "5.0.0",
-    "vue": "3.5.42",
+    "vitest": "5.0.1",
+    "vue": "3.5.43",
     "vue-router": "5.3.1"
   }
 }