@gr8ful/spf 0.12.0 → 0.13.0

package/assets/skill/references/config.md

@@ -159,10 +159,13 @@ Full mechanism: the main README's "`spf watch`" section. Field reference:
 | `concurrency` | int ≥1 | Max ISSUES claimed and run at once — not attempts; see `fanout.concurrency`. The build lane's own budget (independent of `refine.concurrency`). Default `2`. |
 | `chain_options` | map of string -> string | Options passed straight through to `chain` (and `refine.chain`) for every unattended dispatch — the same shape an interactive `spf <chain> --suite <name>` builds, e.g. `{suite: strict}` or `{agent: some-agent}`. Default `{}`. Only useful for a chain whose behavior actually reads the option (a step-derived chain's `--suite`; an imperative chain ignores an option it doesn't know about). |
 | `jira.base_url` / `jira.project_key` | string | Only consulted when `issue_provider: jira`. |
-| `jira.issue_types` | map: `epic`/`feature`/`story`/`bug`/`task` -> string | Only consulted when `issue_provider: jira` AND `refine.enabled`. What each `RefinedIssue.kind` creates as on Jira — defaults `epic`/`feature` → `Epic`, `story` → `Story`, `bug` → `Bug`, `task` → `Task`, overridable per kind. Validated against the real project by both `spf watch init` and `spf watch`'s own startup check. |
+| `jira.issue_types` | map: `epic`/`feature`/`story`/`bug`/`task`/`spec` -> string | Only consulted when `issue_provider: jira` AND `refine.enabled`. What each `RefinedIssue.kind` (plus `spec`, a standalone spec proposed by a split — see below) creates as on Jira — defaults `epic`/`feature` → `Epic`, `story`/`spec` → `Story`, `bug` → `Bug`, `task` → `Task`, overridable per kind. Validated against the real project by both `spf watch init` and `spf watch`'s own startup check. |
 | `refine.enabled` | bool | Turns on the second lane: decompose a `<prefix>:spec-ready` product spec into a feature/story-or-bug tree of real issues, instead of running `chain` against it directly (a spec isn't individually workable). Default `false` — off by default, so an existing `watch:` config is unaffected by upgrading. Needs `issue_provider: github` or `"jira"` — both implement issue authoring (create + link a hierarchy); any other value fails loudly at startup. |
 | `refine.chain` | string | Which registered chain runs per claimed spec. Default `refine`. |
 | `refine.concurrency` | int ≥1 | The refine lane's own budget, separate from `concurrency`. Default `1`. |
+| `refine.max_leaves` | int ≥1 | The decomposition budget: at most this many LEAVES per refinement — each one becomes its own worktree, chain run, and pull request once promoted, so this is a human-review budget, not a model-effort one. Default `4`. Enforced by `gates.refinementWellFormed`; exceeding it (with no human-approved split already in the thread) sends the refiner back for a correction, and a genuine overrun proposes a `split` into several specs instead of publishing (see below) rather than force-fitting an oversized tree. |
+| `refine.max_nodes` | int ≥1 | Total node ceiling (leaves + containers). Default `6`. Doesn't bind at the other two defaults — it exists so raising `max_leaves` or `max_depth` alone can't silently uncap the whole tree. |
+| `refine.max_depth` | int ≥1 | Containment-depth ceiling — `1` is a top-level leaf/container, `2` is a leaf directly under a top-level container. Default `2`. **Raising this above 2 is unsafe on Jira**: `epic`/`feature` both map to Jira's Epic type by default, and Jira has no Epic-under-Epic nesting, so a 3-level tree fails partway through a non-transactional publish. At the default, that shape is rejected before any issue is created. |
 | `fanout.n` | int, 1-8 | Best-of-N per claimed issue: run `n` sibling attempts of the same issue and let code pick a winner (`core/fanout.ts`'s `pickBest`, the same mechanism `spf fanout` uses standalone). Default `1` — single dispatch, byte-identical to `spf watch` before this key existed. `n > 1` requires `chain` to have a commit step (`spf watch` refuses to start otherwise — a chain with no commit step would have its N-1 losing attempts' uncommitted work destroyed by best-of-N's own cleanup). |
 | `fanout.concurrency` | int ≥1 | Attempts of ONE issue's fan-out IN FLIGHT at once — **not** `concurrency`, which counts issues. The two multiply: `concurrency: 2` × `fanout.concurrency: 2` is up to 4 chain runs in flight; `spf doctor`'s `watch.fanout` line prints the exact product. Does **not** bound worktrees on disk — a successful attempt's tree is kept until every sibling in its fan-out has settled, so disk peak is `concurrency × fanout.n`, not `concurrency × fanout.concurrency`. Default `2`. |
 
@@ -177,6 +180,9 @@ watch:
     enabled: true       # decompose spf:spec-ready specs into a feature/story tree
     chain: refine
     concurrency: 1
+    max_leaves: 4       # optional — shown are the defaults; see refine.max_leaves above
+    max_nodes: 6
+    max_depth: 2
   fanout:
     n: 1                 # attempts per claimed issue. 1 = single dispatch (default) — no-op for the daemon
     concurrency: 2        # attempts IN FLIGHT per issue — not watch.concurrency (issues in flight)
@@ -206,18 +212,31 @@ The `refine` chain grounds its decomposition with a `scout` phase before the
 refiner runs, so `scout` is a required agent for it — a roster that pruned
 it fails `spf watch` startup by name.
 
-The refine lane's own state machine has an extra loop beyond
-`spec-ready → refining → spec-in-progress → done`/`blocked`: when the
-refiner raises material ambiguity instead of a tree (see
-`assets/prompts/refiner/system.md`'s "Ask, don't decide"), the spec moves to
-`<prefix>:needs-feedback` with a comment naming its questions, instead of
-publishing anything. A human answers in the issue's comments and adds
-`<prefix>:continue-refinement`; `spf watch` claims that label back into
+The refine lane's own state machine has two extra loops beyond
+`spec-ready → refining → spec-in-progress → done`/`blocked`, for two
+different problems. When the refiner raises material ambiguity instead of a
+tree (see `assets/prompts/refiner/system.md`'s "Ask, don't decide"), the spec
+moves to `<prefix>:needs-feedback` with a comment naming its questions,
+instead of publishing anything. A human answers in the issue's comments and
+adds `<prefix>:continue-refinement`; `spf watch` claims that label back into
 `refining` and resumes the **same** `adw_id` — the comment thread (split
 into "answers to your open questions" and "earlier discussion") is folded
 into the resumed prompt, and the refiner's own coding-agent session
 continues rather than starting cold. This can loop any number of rounds.
 
+The other loop is for size, not ambiguity: when the spec honestly exceeds
+`refine.max_leaves` (see above), the refiner proposes a `split` into two or
+more standalone specs instead of a scope question, and the spec moves to
+`<prefix>:split-proposed` with a comment naming each proposed spec and why
+it's coherent on its own. A human either adds `<prefix>:split-approved`
+(executed deterministically — `core/refine.ts`'s `publishSpecs()`, no agent
+re-run: it creates exactly what the proposal comment showed, each new spec
+carrying `<prefix>:type:spec` + `<prefix>:spec-ready`, never
+`<prefix>:refined`) or revises the proposal the same way a question gets
+answered (`<prefix>:continue-refinement`). The original spec then tracks
+both child specs' own trees to completion transitively, the same way it
+tracks a published tree's leaves.
+
 Publishing a tree does NOT mean the spec is done: `<prefix>:spec-in-progress`
 is where a spec lands right after publish, and it stays there — with a
 summary comment listing every issue created — until **every one of those

package/assets/templates/ts.spf.config.yaml

@@ -79,14 +79,21 @@ agents:
 #   # feature/story-or-bug tree of real issues instead of building it
 #   # directly. Off by default; needs issue_provider: github OR jira — both
 #   # support issue authoring (create + link a hierarchy). On Jira, each
-#   # RefinedIssue.kind maps to a real Jira issue type via
-#   # watch.jira.issue_types (defaults: epic/feature -> Epic, story -> Story,
-#   # bug -> Bug, task -> Task) — spf watch init validates this against the
-#   # real project when refine is enabled.
+#   # RefinedIssue.kind (plus spec, from a split proposal) maps to a real
+#   # Jira issue type via watch.jira.issue_types (defaults: epic/feature ->
+#   # Epic, story/spec -> Story, bug -> Bug, task -> Task) — spf watch init
+#   # validates this against the real project when refine is enabled.
+#   # max_leaves/max_nodes/max_depth are the decomposition budget: every leaf
+#   # becomes its own pull request once promoted, so this bounds how many a
+#   # human reviews per spec. An honest overrun proposes splitting the spec
+#   # into several standalone ones instead of publishing an oversized tree.
 #   refine:
 #     enabled: true
 #     chain: refine
 #     concurrency: 1
+#     max_leaves: 4
+#     max_nodes: 6
+#     max_depth: 2
 
 # Optional: push notifications for unattended work — spf watch's daemon
 # lifecycle, and every chain run (including watch's own per-issue runs).

package/dist/chains/steps.d.ts

@@ -317,20 +317,20 @@ export declare function refine(opts?: {
     extraGates?: string[];
 }): Step;
 /**
- * `cli/commands/watch.ts`'s `runRefine()` reads BOTH `refine_publish.json`
- * and `refine_questions.json` back, unconditionally, after this chain
- * exits — it has no other way to learn what THIS run did, since a chain's
- * return value is just an exit code. A resumed spec (`continue-refinement`)
- * reruns this entire chain from `request` on, into the SAME deterministic
- * `context_handoff_dir` a PRIOR round already wrote into. Without clearing
- * the file this run is NOT about to write, a stale `refine_questions.json`
- * from an earlier escalation round survives a LATER round's successful
- * publish — `runRefine` then reports those old questions as if raised
- * again THIS round, so `runSpec` escalates a second time even though real
- * issues were already created on the tracker seconds earlier. Called
- * before EITHER branch writes, so exactly one of the two files reflects
- * this run when the phase returns, never a leftover from a previous one.
- * `force: true` — a first-ever run has neither file yet, which is fine.
+ * `cli/commands/watch.ts`'s `runRefine()` reads `refine_publish.json`,
+ * `refine_questions.json`, AND `refine_split.json` back, unconditionally,
+ * after this chain exits — it has no other way to learn what THIS run did,
+ * since a chain's return value is just an exit code. A resumed spec
+ * (`continue-refinement`) reruns this entire chain from `request` on, into
+ * the SAME deterministic `context_handoff_dir` a PRIOR round already wrote
+ * into. Without clearing the files this run is NOT about to write, a stale
+ * one from an earlier round survives a LATER round's successful publish —
+ * `runRefine` then reports that stale outcome as if it happened again THIS
+ * round, so `runSpec` escalates (or splits) a second time even though real
+ * issues were already created on the tracker seconds earlier. Called before
+ * ANY branch writes, so exactly one of the three files reflects this run
+ * when the phase returns, never a leftover from a previous one. `force:
+ * true` — a first-ever run has none of them yet, which is fine.
  */
 export declare function clearStaleRefineOutputFiles(contextHandoffDir: string): void;
 export declare function publishIssues(opts?: {

package/dist/chains/steps.js

@@ -664,19 +664,26 @@ export function refine(opts = {}) {
         output_type: RefineOutput,
         description: opts.description ?? "Decompose the spec into a feature/story tree of vertical slices",
         // refinementWellFormed is NOT in GATE_ALLOWLIST — it is meaningless on
-        // any other envelope type (it reads `issues`/`questions`), so there is
-        // nothing to gain by letting a definition name it, and it stays
+        // any other envelope type (it reads `issues`/`questions`/`split`), so
+        // there is nothing to gain by letting a definition name it, and it stays
         // non-removable here.
         //
-        // retries: 1, not 0 — this gate's own doc comment says a violation
-        // "re-prompts the SAME refiner session before publishIssues() ever
-        // runs." With 0 retries that was never true: agents.ts throws
-        // GateFailure on the first violation and the whole spec goes straight to
-        // spf:blocked with no correction round-trip. The mutual-exclusion rule
-        // between `issues` and `questions` (see refinementWellFormed) makes that
-        // mismatch more likely to bite in practice, not less.
+        // retries: 2, not 1. `retries: 1` bought exactly one correction round
+        // (`core/agents.ts`'s gate loop: attempts 1..retries+1), which is enough
+        // for a MECHANICAL violation — an unresolved parent, a mislabeled kind —
+        // because the fix is local and obvious. The budget checks in
+        // `gates.refinementWellFormed` are not mechanical: the first correction
+        // is "cut this down," and a refiner that genuinely cannot needs a SECOND
+        // round to reach for the other branch (publish nothing, propose a
+        // `split`). Without it, the honest-overrun case exits as a
+        // `GateFailure` — which does NOT return non-zero through
+        // `cli/commands/watch.ts`'s `runRefine`, but THROWS past it into
+        // `core/watch.ts`'s `runSpec` catch, moving the spec to
+        // `<prefix>:blocked` and paging a `watch_error` at level "error". A spec
+        // that is merely too big is not an spf failure, and it should not look
+        // like one.
         gates: withExtraGates([gates.refinementWellFormed], opts.extraGates),
-        retries: opts.retries ?? 1,
+        retries: opts.retries ?? 2,
     });
 }
 /**
@@ -724,23 +731,23 @@ function parsePriorityOption(raw) {
     return raw;
 }
 /**
- * `cli/commands/watch.ts`'s `runRefine()` reads BOTH `refine_publish.json`
- * and `refine_questions.json` back, unconditionally, after this chain
- * exits — it has no other way to learn what THIS run did, since a chain's
- * return value is just an exit code. A resumed spec (`continue-refinement`)
- * reruns this entire chain from `request` on, into the SAME deterministic
- * `context_handoff_dir` a PRIOR round already wrote into. Without clearing
- * the file this run is NOT about to write, a stale `refine_questions.json`
- * from an earlier escalation round survives a LATER round's successful
- * publish — `runRefine` then reports those old questions as if raised
- * again THIS round, so `runSpec` escalates a second time even though real
- * issues were already created on the tracker seconds earlier. Called
- * before EITHER branch writes, so exactly one of the two files reflects
- * this run when the phase returns, never a leftover from a previous one.
- * `force: true` — a first-ever run has neither file yet, which is fine.
+ * `cli/commands/watch.ts`'s `runRefine()` reads `refine_publish.json`,
+ * `refine_questions.json`, AND `refine_split.json` back, unconditionally,
+ * after this chain exits — it has no other way to learn what THIS run did,
+ * since a chain's return value is just an exit code. A resumed spec
+ * (`continue-refinement`) reruns this entire chain from `request` on, into
+ * the SAME deterministic `context_handoff_dir` a PRIOR round already wrote
+ * into. Without clearing the files this run is NOT about to write, a stale
+ * one from an earlier round survives a LATER round's successful publish —
+ * `runRefine` then reports that stale outcome as if it happened again THIS
+ * round, so `runSpec` escalates (or splits) a second time even though real
+ * issues were already created on the tracker seconds earlier. Called before
+ * ANY branch writes, so exactly one of the three files reflects this run
+ * when the phase returns, never a leftover from a previous one. `force:
+ * true` — a first-ever run has none of them yet, which is fine.
  */
 export function clearStaleRefineOutputFiles(contextHandoffDir) {
-    for (const name of ["refine_questions.json", "refine_publish.json"]) {
+    for (const name of ["refine_questions.json", "refine_publish.json", "refine_split.json"]) {
         rmSync(path.join(contextHandoffDir, name), { force: true });
     }
 }
@@ -752,6 +759,7 @@ export function publishIssues(opts = {}) {
             throw new Error("publishIssues() requires a preceding refine() step in the chain's step list");
         }
         const questions = envelope.questions ?? [];
+        const split = envelope.split ?? [];
         const priorityCeiling = parsePriorityOption(state.options["priority"]);
         await run.phase(makePhaseParams({
             name: "publish",
@@ -765,6 +773,17 @@ export function publishIssues(opts = {}) {
                 ph.log({ escalated: questions.length });
                 return;
             }
+            if (split.length > 0) {
+                // Recorded, never executed, here: creating the proposed specs is
+                // `core/watch.ts`'s `executeApprovedSplits`' job, gated on a human
+                // adding `<prefix>:split-approved` — this phase only writes the
+                // proposal down for `runRefine` to hand to `runSpec`'s own
+                // `proposeSpecSplit`, same division of labor as the `questions`
+                // branch above (this writes, `runSpec` posts/transitions).
+                writeFileSync(path.join(run.context_handoff_dir, "refine_split.json"), JSON.stringify(split, null, 2));
+                ph.log({ split_proposed: split.length });
+                return;
+            }
             const tracker = refineLib.resolveAuthoringProvider(run.cfg);
             const created = await refineLib.publish(tracker, envelope.issues, {
                 labelPrefix: run.cfg.watch.label_prefix,

package/dist/cli/commands/doctor.js

@@ -23,6 +23,7 @@ import { PROVIDER_ENV_KEYS } from "../../core/providers.js";
 import { probeServedOllamaTags, resolveTiering } from "../../core/tiering.js";
 import { isRepoAt } from "../../core/git_helper.js";
 import { allChains, findChain, hasCommitStep, repoChainProblems, resolveRequiredAgents, resolveRequiredSuites } from "../../chains/index.js";
+import { refineBudget } from "../../core/gates.js";
 import * as sandbox from "../../core/sandbox.js";
 import { loadOpenSandboxSdk } from "../../core/sandbox_opensandbox.js";
 import { isInteractive } from "../ask.js";
@@ -978,6 +979,11 @@ export async function doctorCommand(argv) {
                 : cfg.watch.issue_provider === "jira"
                     ? "jira supports issue authoring (createIssue/parent field) — run `spf watch init` to validate watch.jira.issue_types against the real project"
                     : `watch.issue_provider is ${JSON.stringify(cfg.watch.issue_provider)} — the refine lane needs "github" or "jira"`);
+            // Same resolution `gates.refinementWellFormed` itself uses
+            // (refineBudget), never a second reading of these three keys, so this
+            // line can never drift from what a refinement is actually held to.
+            const budget = refineBudget({ repo_root: anchor.repo_root, cfg });
+            check(report, "watch.refine decomposition budget", true, `at most ${budget.maxLeaves} leaf/leaves, ${budget.maxNodes} node(s) total, ${budget.maxDepth} level(s) deep — every leaf becomes one pull request a human reviews once promoted`, "info");
         }
     }
     // OTel span export: informational in every direction. It is off unless

package/dist/cli/commands/watch.js

@@ -17,6 +17,7 @@ import { GitHubProvider } from "../../core/issues/github_provider.js";
 import { JiraProvider } from "../../core/issues/jira_provider.js";
 import { BitbucketProvider } from "../../core/issues/bitbucket_provider.js";
 import { isAuthoringProvider } from "../../core/issues/provider.js";
+import * as refineLib from "../../core/refine.js";
 import { createWatchState, tick } from "../../core/watch.js";
 import { findChain, hasCommitStep, resolveRequiredAgents, runChain as runChainDef } from "../../chains/index.js";
 import { withRunScope } from "../../core/sandbox.js";
@@ -559,7 +560,7 @@ export async function watchCommand(argv) {
         const code = await withRunScope(opts.adwId, () => runChainDef(chainDef, ctx, opts.chainOptions));
         if (code !== 0) {
             const detail = detailFromFailedPhase(cfg, opts.cwd, opts.adwId, `Refine chain "${cfg.watch.refine.chain}" (adw_id ${opts.adwId}) did not complete successfully. Run \`spf phases ${opts.adwId} --cwd ${opts.cwd}\` for detail.`);
-            return { accepted: false, adwId: opts.adwId, detail, created: [], questions: [] };
+            return { accepted: false, adwId: opts.adwId, detail, created: [], questions: [], split: [] };
         }
         const wtAnchor = paths.resolveAnchor(opts.cwd);
         const wtDataPaths = paths.resolveDataPaths(wtAnchor, cfg.defaults.data_dir, cfg.observability.db);
@@ -578,19 +579,32 @@ export async function watchCommand(argv) {
         catch {
             // best-effort, same as above — no questions file means this run wasn't an escalation
         }
+        let split = [];
+        try {
+            split = JSON.parse(readFileSync(path.join(handoffDir, "refine_split.json"), "utf-8"));
+        }
+        catch {
+            // best-effort, same as above — no split file means this run didn't propose one
+        }
         // Defense in depth: `steps.publishIssues()` now clears whichever of
-        // these two files it's NOT about to write, so both should never be
-        // non-empty here — but if some future change (or an older worktree's
-        // leftover files, before that fix existed) ever produces both, a real
-        // completed publish must never be silently overridden by a stale
-        // question. `runSpec` checks `questions.length > 0` first, so without
-        // this it would re-escalate over issues that already landed on the
-        // tracker seconds earlier.
-        if (created.length > 0 && questions.length > 0) {
-            console.error(`watch: ${opts.adwId}: refine_publish.json AND refine_questions.json both had content — treating the ${created.length} published issue(s) as authoritative and discarding the stale question(s)`);
+        // these three files it's NOT about to write, so at most one should ever
+        // be non-empty here — but if some future change (or an older worktree's
+        // leftover files, before that fix existed) ever produces more than one, a
+        // real completed publish must never be silently overridden by a stale
+        // question or split proposal. `runSpec` checks `questions` before
+        // `split` before the publish path, so without this it would re-escalate
+        // (or re-propose a split) over issues that already landed on the tracker
+        // seconds earlier.
+        if (created.length > 0 && (questions.length > 0 || split.length > 0)) {
+            console.error(`watch: ${opts.adwId}: refine_publish.json had content alongside a stale refine_questions.json/refine_split.json — treating the ${created.length} published issue(s) as authoritative and discarding the stale escalation`);
             questions = [];
+            split = [];
+        }
+        else if (questions.length > 0 && split.length > 0) {
+            console.error(`watch: ${opts.adwId}: refine_questions.json AND refine_split.json both had content — treating the questions as authoritative and discarding the stale split proposal`);
+            split = [];
         }
-        return { accepted: true, adwId: opts.adwId, detail: "", created, questions };
+        return { accepted: true, adwId: opts.adwId, detail: "", created, questions, split };
     };
     // `IssueAuthoringProvider`'s read-back half — `isAuthoringProvider()` is a
     // structural check (see `provider.ts`), so both GitHub and Jira are
@@ -639,6 +653,14 @@ export async function watchCommand(argv) {
         dryRun: Boolean(flags["dry-run"]),
         runChain,
         listChildren: authoringProvider ? (parent) => authoringProvider.listChildren(parent) : undefined,
+        // See `WatchDeps.publishSpecs`'s own doc comment: `authoringProvider` is
+        // guaranteed non-null whenever `refine.enabled` is true (the startup
+        // check above already refuses any `issue_provider` besides github/jira
+        // in that case), so this is never `undefined` in the one state
+        // `executeApprovedSplits` actually reads it in.
+        publishSpecs: authoringProvider
+            ? (specs, opts) => refineLib.publishSpecs(authoringProvider, specs, { labelPrefix: cfg.watch.label_prefix, originalSpecId: opts.originalSpecId, priority: opts.priority })
+            : undefined,
         log: (message) => (dashboard ? dashboard.log(message) : console.log(message)),
         notify: (event) => {
             notifier?.send(event); // unaffected either way — see mountWatchDashboard's own doc comment