@gr8ful/spf 0.3.0 → 0.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (91) hide show
  1. package/README.md +177 -9
  2. package/assets/defaults/spf.config.yaml +22 -0
  3. package/assets/prompts/refiner/system.md +53 -0
  4. package/assets/prompts/refiner/user.md +70 -0
  5. package/assets/prompts/reviewer/system.md +1 -1
  6. package/assets/skill/SKILL.md +1 -0
  7. package/assets/skill/cookbooks/authoring_chains.md +90 -7
  8. package/assets/skill/cookbooks/ocr_reviewer.md +196 -0
  9. package/assets/skill/cookbooks/roster.md +15 -4
  10. package/assets/skill/cookbooks/spf_overview.md +1 -0
  11. package/assets/skill/references/config.md +104 -3
  12. package/assets/skill/references/observability.md +11 -2
  13. package/assets/templates/ts-cc.spf.config.yaml +3 -3
  14. package/assets/templates/ts-flue-ollama.spf.config.yaml +67 -0
  15. package/assets/templates/ts.spf.config.yaml +15 -2
  16. package/dist/chains/context.d.ts +39 -0
  17. package/dist/chains/index.d.ts +94 -10
  18. package/dist/chains/index.js +75 -5
  19. package/dist/chains/repo_chains.d.ts +139 -0
  20. package/dist/chains/repo_chains.js +428 -0
  21. package/dist/chains/simple_sdlc.d.ts +74 -1
  22. package/dist/chains/simple_sdlc.js +134 -4
  23. package/dist/chains/steps.d.ts +237 -18
  24. package/dist/chains/steps.js +477 -58
  25. package/dist/cli/ask.d.ts +14 -1
  26. package/dist/cli/ask.js +32 -2
  27. package/dist/cli/commands/doctor.d.ts +1 -1
  28. package/dist/cli/commands/doctor.js +324 -10
  29. package/dist/cli/commands/init.d.ts +12 -0
  30. package/dist/cli/commands/init.js +108 -4
  31. package/dist/cli/commands/install-skill.js +5 -2
  32. package/dist/cli/commands/list.js +43 -5
  33. package/dist/cli/commands/run.js +29 -2
  34. package/dist/cli/commands/watch.d.ts +18 -0
  35. package/dist/cli/commands/watch.js +214 -16
  36. package/dist/cli/index.js +63 -6
  37. package/dist/cli/interview.js +81 -9
  38. package/dist/core/agent_cc.d.ts +40 -1
  39. package/dist/core/agent_cc.js +51 -4
  40. package/dist/core/agent_flue.js +28 -4
  41. package/dist/core/agents.d.ts +8 -0
  42. package/dist/core/agents.js +43 -3
  43. package/dist/core/data_types.d.ts +182 -4
  44. package/dist/core/data_types.js +141 -2
  45. package/dist/core/gates.d.ts +13 -0
  46. package/dist/core/gates.js +103 -0
  47. package/dist/core/git_helper.d.ts +29 -0
  48. package/dist/core/git_helper.js +41 -1
  49. package/dist/core/issues/github_provider.d.ts +35 -9
  50. package/dist/core/issues/github_provider.js +76 -28
  51. package/dist/core/issues/jira_provider.d.ts +14 -1
  52. package/dist/core/issues/jira_provider.js +9 -7
  53. package/dist/core/issues/provider.d.ts +77 -15
  54. package/dist/core/issues/provider.js +7 -4
  55. package/dist/core/notify/channel.d.ts +1 -1
  56. package/dist/core/ollama_provider.d.ts +70 -0
  57. package/dist/core/ollama_provider.js +208 -0
  58. package/dist/core/otel.d.ts +352 -0
  59. package/dist/core/otel.js +793 -0
  60. package/dist/core/providers.js +4 -0
  61. package/dist/core/refine.d.ts +39 -0
  62. package/dist/core/refine.js +152 -0
  63. package/dist/core/session.js +39 -2
  64. package/dist/core/tracer.d.ts +31 -2
  65. package/dist/core/tracer.js +69 -11
  66. package/dist/core/watch.d.ts +67 -1
  67. package/dist/core/watch.js +217 -13
  68. package/dist/test/chains.test.js +9 -3
  69. package/dist/test/data_types.test.js +140 -2
  70. package/dist/test/git_helper.test.d.ts +1 -0
  71. package/dist/test/git_helper.test.js +59 -0
  72. package/dist/test/hermetic_git.d.ts +1 -0
  73. package/dist/test/hermetic_git.js +22 -0
  74. package/dist/test/init_command.test.d.ts +14 -1
  75. package/dist/test/init_command.test.js +71 -1
  76. package/dist/test/interview.test.d.ts +15 -1
  77. package/dist/test/interview.test.js +131 -3
  78. package/dist/test/ollama_provider.test.d.ts +1 -0
  79. package/dist/test/ollama_provider.test.js +103 -0
  80. package/dist/test/otel.test.d.ts +26 -0
  81. package/dist/test/otel.test.js +512 -0
  82. package/dist/test/refine.test.d.ts +1 -0
  83. package/dist/test/refine.test.js +189 -0
  84. package/dist/test/repo_chains.test.d.ts +21 -0
  85. package/dist/test/repo_chains.test.js +416 -0
  86. package/dist/test/signoff.test.d.ts +1 -0
  87. package/dist/test/signoff.test.js +329 -0
  88. package/dist/test/ui_server.test.d.ts +7 -1
  89. package/dist/test/ui_server.test.js +1 -0
  90. package/dist/test/watch.test.js +297 -6
  91. package/package.json +5 -5
package/README.md CHANGED
@@ -15,7 +15,7 @@ Everyone can get an agent to write code once. Almost nobody gets the same result
15
15
  npm i -g @gr8ful/spf
16
16
  ```
17
17
 
18
- No Bun, no separate coding-agent binary to install first, no template stamped into your repo. `spf` ships with a packaged default agent roster and default prompts — it runs against any repo with zero setup, and only writes files into that repo if you explicitly ask it to (`spf init`, `spf install-skill`).
18
+ No Bun, no separate coding-agent binary to install first, no template stamped into your repo. `spf` ships with a packaged default agent roster and default prompts — it runs against any repo with zero setup, and only writes files into that repo if you explicitly ask it to (`spf init`, which also installs the Claude Code skill unless you pass `--no-skills`; `spf install-skill` does just the skill by hand).
19
19
 
20
20
  ```bash
21
21
  cd your-repo
@@ -38,6 +38,8 @@ On a real terminal, `spf init` asks a short interview — which coding agent (`c
38
38
 
39
39
  Without an interview, `spf init` writes the same small starter `.spf/spf.config.yaml`, commented, that merges on top of the packaged built-ins field by field. `--template <name>` writes a real, filled-in config instead of the commented-out starter — every packaged template's name prints after `spf init` runs, and the same files live in [`assets/templates/`](assets/templates/) to browse directly. Nothing here needs to exist for `spf` to run; it's how you make one repo's roster diverge from the defaults.
40
40
 
41
+ Every `spf init` run — interview or not — also installs the repo-local Claude Code skill (`.claude/skills/spf`), the same work `spf install-skill` does by hand: pass `--no-skills` to skip it. It's idempotent (a no-op once the skill is already current, and writes a `.new` sibling instead of overwriting a file you've locally edited), so re-running `spf init` never clobbers anything there. `spf install-skill --user` (installing to `~/.claude/skills/spf` instead) is still its own separate invocation.
42
+
41
43
  ### Local development
42
44
 
43
45
  Working from a clone instead of the published package:
@@ -108,7 +110,7 @@ agents:
108
110
  writes: [specs/] # the plan is all it may leave in the repo
109
111
  ```
110
112
 
111
- Five starter agents ship in the box: `planner`, `builder`, `scout` (read-only recon), `reviewer`, and `documenter`. There is no tester, because running a suite is a known command and therefore code, not an agent's job.
113
+ Six starter agents ship in the box: `planner`, `builder`, `scout` (read-only recon), `refiner` (decomposes a product spec for `spf watch`'s refine lane — see "`spf watch`" below), `reviewer`, and `documenter`. There is no tester, because running a suite is a known command and therefore code, not an agent's job.
112
114
 
113
115
  Every agent gets its own model, thinking level, prompts, and tools. Give the planner a frontier model and the builder a cheap fast one. Give the reviewer no ability to write code at all.
114
116
 
@@ -118,7 +120,7 @@ Config defines who an agent **is**. The chain call site defines how it is **used
118
120
 
119
121
  ### A second backend: Claude Code
120
122
 
121
- Set `coding_agent: claude_code` on any agent (or in `defaults`) to run it on your own installed [Claude Code](https://claude.com/product/claude-code) CLI instead of Flue — `spf doctor` checks it's on `PATH`. Model names follow Claude Code's own vocabulary (a bare alias like `sonnet`, not `provider/model-id`); everything else — `tools`, `writes`, `thinking` — stays the same shape. Pointing a `claude_code` agent at a local or cloud [Ollama](https://ollama.com) server needs no config at all — just `ANTHROPIC_BASE_URL`/`ANTHROPIC_AUTH_TOKEN` set before you run `spf`, since Claude Code's CLI reads those itself.
123
+ Set `coding_agent: claude_code` on any agent (or in `defaults`) to run it on your own installed [Claude Code](https://claude.com/product/claude-code) CLI instead of Flue — `spf doctor` checks whatever `SPF_CLAUDE_CMD`'s first token resolves to on `PATH` (`claude` itself, by default). Model names follow Claude Code's own vocabulary (a bare alias like `sonnet`, not `provider/model-id`); everything else — `tools`, `writes`, `thinking` — stays the same shape. Pointing a `claude_code` agent at a local or cloud [Ollama](https://ollama.com) server needs no config at all — just `ANTHROPIC_BASE_URL`/`ANTHROPIC_AUTH_TOKEN` set before you run `spf`, since Claude Code's CLI reads those itself.
122
124
 
123
125
  #### Proxy or wrapper launchers
124
126
 
@@ -126,17 +128,52 @@ To route the `claude` command through a wrapper, proxy server, or launcher (e.g.
126
128
 
127
129
  ```bash
128
130
  # Route through Ollama's launcher
129
- export SPF_CLAUDE_CMD="ollama launch claude"
131
+ export SPF_CLAUDE_CMD="ollama launch claude --model granite4.1:8b"
130
132
  spf build "your prompt"
131
133
  ```
132
134
 
135
+ `ollama launch <cmd>` uses cobra flag parsing, which treats anything typed after it as its own flags unless a literal `--` says otherwise — without one, `claude`'s own flags (`-p`, `--json-schema`, ...) fail with `unknown shorthand flag: 'p' in -p` before `claude` is ever reached. `agent_cc.ts` detects exactly this `ollama launch ...` shape and inserts that `--` automatically, so you never add the separator by hand for this specific launcher.
136
+
137
+ That `--` alone is not enough to reach `claude`, though: `ollama launch` also needs its OWN `--model <tag>` flag (a tag from `ollama list`), typed BEFORE the auto-inserted `--`, whenever it runs headless — which it always does under SPF, since SPF spawns with piped stdio. Without it, `ollama launch` falls back to an interactive model picker that can never run, and fails one step later than the `--` problem, with `model selection requires an interactive terminal; use --model to run in headless mode`. A `--model` typed after the `--` doesn't help — at that point it belongs to `claude`, not to `ollama launch`. So `SPF_CLAUDE_CMD` must include `ollama launch`'s `--model` yourself, exactly as written above; `spf doctor` hard-fails if it's missing.
138
+
133
139
  ```bash
134
140
  # Or use a custom wrapper script
135
141
  export SPF_CLAUDE_CMD=/path/to/my-wrapper
136
142
  spf build "your prompt"
137
143
  ```
138
144
 
139
- The command/launcher must support the full Claude Code CLI interface: `-p` for prompt, `--json-schema`, `--model`, `--session-id`/`--resume`, `--output-format stream-json`, and all other flags `agent_cc` uses. When unset, `SPF_CLAUDE_CMD` defaults to `claude` (resolved from `PATH` normally).
145
+ The command/launcher must support the full Claude Code CLI interface: `-p` for prompt, `--json-schema`, `--model`, `--session-id`/`--resume`, `--output-format stream-json`, and all other flags `agent_cc` uses. When unset, `SPF_CLAUDE_CMD` defaults to `claude` (resolved from `PATH` normally). A cmdSpec that already contains its own literal `--` is left completely alone — `agent_cc.ts` never inserts a second one.
146
+
147
+ ### flue + local Ollama
148
+
149
+ Point the default `flue` backend at a local Ollama server the same way you'd pick any other Flue provider — the model string's own prefix, `ollama/<tag>` (whatever `ollama list` shows on your machine) instead of `openai/...`/`anthropic/...`:
150
+
151
+ ```yaml
152
+ defaults:
153
+ coding_agent: flue
154
+ model: ollama/qwen3.8:27b-mlx
155
+ ```
156
+
157
+ ```bash
158
+ export OLLAMA_BASE_URL=http://localhost:11434/v1 # default if unset
159
+ spf build "your prompt"
160
+ ```
161
+
162
+ That's the whole config change. `ollama` is a keyless provider — `spf doctor` and the `spf init` interview both know this: no API key is ever asked for or checked, and interview.ts asks for `OLLAMA_BASE_URL` instead when you pick `ollama` as your provider. A ready-to-run starting point ships at [`assets/templates/ts-flue-ollama.spf.config.yaml`](assets/templates/ts-flue-ollama.spf.config.yaml) (`spf init --template ts-flue-ollama`).
163
+
164
+ Two things worth knowing before pointing a full roster at local models: tool-calling — the injected `sf_report` contract every agent's structured output rides on — worked reliably in testing down to a 3B-parameter model, which isn't a "only frontier models get tools" situation. And context-window occupancy reporting is disabled for `ollama/*` models specifically, which turns off threshold-based compaction rather than reporting a number Ollama's OpenAI-compatible API doesn't actually provide per model.
165
+
166
+ ---
167
+
168
+ ## Isolation: post-hoc, not a sandbox
169
+
170
+ Agent isolation in SPF is enforced **after the fact**, not upfront. This matters because the `tools:` capability list is not a sandbox — a `bash` tool can run `git checkout` to discard changes, and a `write` tool can reach any path, regardless of the allowlist you write. So permission is verified the way every other claim in this system is: the working tree is fingerprinted before an agent runs, then compared after. If the agent touched anything outside its `writes:` allowlist (or a `protected_files:` path it wasn't given access to), the phase fails and anything the agent *introduced* outside its allowlist is rolled back. What it cannot undo, it names: a file that was already dirty before the agent ran is left alone rather than discarded, and uncommitted work the agent reverted cannot be restored. And because the check is post-hoc and tree-scoped, nothing outside the working tree is in scope at all — a push, a network call, a write above `repo_root`. When running an agent on the claude_code backend, SPF spawns it with `--permission-mode bypassPermissions --dangerously-skip-permissions` because the backend has no need to enforce — SPF will enforce in code instead, within those bounds.
171
+
172
+ For unattended work, `spf watch`'s PR-merge is the only human accountability checkpoint: the daemon opens a PR for each automated run, and a human approves the merge. For any other workflow, this enforcement is the backbone of letting a bounded agent proposal survive code's inspection without a human having to read it in between — but it is bounded, not a sandbox, and the scope above is what a reviewer should actually rely on. (Attended `simple-sdlc` runs get a second checkpoint of their own — see below.)
173
+
174
+ See `core/permissions.ts` for the implementation — the snapshot/compare logic, not the config terms.
175
+
176
+ `simple-sdlc`'s `commit_build` phase is the one place in this codebase where an AI reviewer's `approved` flag alone would otherwise gate a commit — everywhere else the reviewer either doesn't run (`plan-build-test`, ...) or nothing commits on its verdict (`build-review` has no commit step). Run attended, that phase asks before committing — default **no**, never `review.approved` itself, bounded by `review.signoff_timeout_seconds` (default 300s; expiry answers no) — and only an explicit, recorded "yes" ever earns a `Signed-off-by:` trailer, built from `git config user.name`/`user.email` at the repo, never an env var. Run unattended (`spf watch`, CI), there is nobody to ask: `spf watch`'s own human gate is the PR merge above, informed by the reviewer's digest in the PR body, and the phase either proceeds on the AI verdict alone with a loud warning (`review.require_human_signoff: false`, this release's default — fail-open until `spf watch` itself becomes signoff-aware) or fails closed and asks you to rerun attended (`require_human_signoff: true`). See `assets/skill/references/config.md`'s `review` section for the two knobs, and [`assets/skill/cookbooks/ocr_reviewer.md`](assets/skill/cookbooks/ocr_reviewer.md) if you want the reviewer to fold a third-party review tool's findings into its own verdict as one more piece of evidence.
140
177
 
141
178
  ---
142
179
 
@@ -212,7 +249,7 @@ Files stay the raw record (`envelope.json`, `agent_map.json`, Flue's own convers
212
249
 
213
250
  ```bash
214
251
  spf list # every chain, its phases, what it needs
215
- spf <chain> "<prompt or path/to/prompt.md>" [--config <path>] [--adw-id a1b2c3d4] [--cwd <dir>]
252
+ spf <chain> "<prompt or path/to/prompt.md>" [--config <path>] [--adw-id a1b2c3d4] [--cwd <dir>] [--suite <name>]
216
253
  ```
217
254
 
218
255
  | Chain | Phases | Reach for it when |
@@ -230,6 +267,8 @@ spf <chain> "<prompt or path/to/prompt.md>" [--config <path>] [--adw-id a1b2c3d4
230
267
  | `document` | code(git diff), documenter | write up what just shipped |
231
268
  | `simple-sdlc` | plan, build, test, review, document | the work is real and its shape is not obvious |
232
269
 
270
+ A chain with a compiled-in quality suite (`build-test`, `plan-build-test`, `plan-build-test-quality`, `quality`) accepts `--suite <name>` to override it for one run — `spf list` marks which chains this applies to with `(--suite overrides)`. `simple-sdlc`'s suite is not overridable this way; passing `--suite` to it is rejected rather than silently ignored.
271
+
233
272
  `--adw-id` is optional everywhere. Omit it and a fresh id is minted and printed. Supply it and the run joins that session — same session directory, each agent **resumes its existing context window** instead of starting cold. That's how you chain runs:
234
273
 
235
274
  ```bash
@@ -237,12 +276,44 @@ spf plan "add a /health endpoint" # prints adw_id a1b2c3d
237
276
  spf build-test "implement the plan" --adw-id a1b2c3d4
238
277
  ```
239
278
 
279
+ ### Repo-local chains
280
+
281
+ Every chain above is built in. A target repo can also compose its own out of
282
+ the same step factories, with zero forking and zero build step, by dropping a
283
+ file in `.spf/chains/*.yaml` — `spf init` scaffolds `.spf/chains/example.yaml`
284
+ (fully commented out) showing the shape:
285
+
286
+ ```yaml
287
+ # .spf/chains/ship-it.yaml — spf ship-it "<prompt>" / spf run ship-it "<prompt>"
288
+ name: ship-it
289
+ describe: plan, build, test, land — with our own reviewer in the loop
290
+ steps:
291
+ - step: request
292
+ - step: plan
293
+ owner: architect
294
+ - step: build
295
+ extraGates: [jsonParses] # additive only — a step's built-in gates can never be removed
296
+ - step: fixLoop
297
+ suite: test
298
+ - step: commit
299
+ onlyIfAccepted: true
300
+ ```
301
+
302
+ It shows up in `spf list`/`spf doctor` exactly like a built-in, and runs on
303
+ the identical `stepChain()` driver — there is no second interpreter. The full
304
+ step vocabulary, the gate rules, and the `spf watch` main-anchor divergence
305
+ (a chain edited on an issue branch is *not* what an in-flight `spf watch` run
306
+ uses) are documented in the installed skill's "Repo-local chains" section
307
+ (`.claude/skills/spf/cookbooks/authoring_chains.md`, after `spf init`).
308
+
240
309
  ---
241
310
 
242
311
  ## `spf watch`
243
312
 
244
313
  Polls an issue tracker for issues labeled `<prefix>:ready`, runs a configured chain against each in its own git worktree, opens a PR against a code host, and tracks it through to merged or blocked — driving the same chains above rather than reimplementing an SDLC. Labels are the whole state machine: `ready → working → review → done`/`blocked`.
245
314
 
315
+ A second, optional lane (`watch.refine`, off by default) decomposes a `<prefix>:spec-ready` product spec into a feature/story-or-bug tree of real issues instead — see "Refining specs" below.
316
+
246
317
  The tracker (`issue_provider`) and the code host (`code_host`) are independent config choices, not one bundled "provider" — a tracker and a host are independent choices in practice (Jira issues against a Bitbucket repo is a real setup). Supported today: `issue_provider: github | jira`, `code_host: github | bitbucket` — any combination works, including Jira+GitHub or GitHub-issues+Bitbucket.
247
318
 
248
319
  The easiest way into any of this is `spf init`'s interview: it asks whether to enable `spf watch`, which tracker and code host, and collects exactly the env vars that combination needs (below) straight into `.env` — no hand-editing YAML or hunting down which credential pair a given combination wants.
@@ -274,6 +345,29 @@ watch:
274
345
  project_key: PROJ
275
346
  ```
276
347
 
348
+ ```yaml
349
+ # .spf/spf.config.yaml — GitHub issues + Bitbucket PRs, in TWO DIFFERENT repos
350
+ watch:
351
+ issue_provider: github
352
+ code_host: bitbucket
353
+ repo: workspace/repo_slug # Bitbucket's repo — the CODE HOST always reads plain `repo`
354
+ issue_repo: owner/name # GitHub's repo — only needed here, where issue tracker != code host
355
+ label_prefix: spf # polls GitHub issues labeled spf:ready
356
+ chain: plan-build-test
357
+ base_branch: main
358
+ ```
359
+
360
+ `repo` always names the code host's own repo; `issue_repo` overrides it for
361
+ the issue-tracker side. This only matters for `issue_provider: github` +
362
+ `code_host: bitbucket` — the one combination where they're genuinely
363
+ different repos in different systems, not the same repo worn two ways
364
+ (`github`+`github` is one repo by construction; `issue_provider: jira` never
365
+ reads `repo` at all, so it's never ambiguous). Leave `issue_repo` unset
366
+ everywhere else. `spf doctor` flags this specific combination when
367
+ `issue_repo` is missing, since the silent failure mode — polling GitHub with
368
+ the Bitbucket identifier — finds no matching issues and looks exactly like
369
+ nothing being configured at all.
370
+
277
371
  ```bash
278
372
  spf watch init # idempotently seed tracker state (no-op for Jira — see below); run this first
279
373
  spf watch # foreground daemon; Ctrl-C drains in-flight claims first
@@ -283,15 +377,61 @@ spf watch --dry-run # log intended claims/transitions, mutate nothing
283
377
 
284
378
  No GitHub App, no webhook, no Jira/Bitbucket app install — it's a plain REST poll against whichever combination is configured, same philosophy as the trace db's own polling contract. See [`assets/templates/`](assets/templates/) for full worked configs (also usable directly via `spf init --template <name>`), and `spf install-skill`'s installed skill (`roster.md`, `references/config.md`) for the field-by-field reference.
285
379
 
380
+ ### Refining specs (`watch.refine`)
381
+
382
+ A product spec isn't individually workable — it needs to become a feature,
383
+ broken down into user stories and bugs, before the build lane above has
384
+ anything to claim. `watch.refine` is a second lane over the same poll loop
385
+ that does exactly that: it polls `<prefix>:spec-ready`, runs a decomposition
386
+ chain (`refine` by default) against the spec in its own worktree, and
387
+ publishes what it produces as real tracker issues — a feature/epic container
388
+ plus story/bug/task leaves, linked via GitHub's native sub-issue hierarchy.
389
+
390
+ ```yaml
391
+ watch:
392
+ issue_provider: github # required — issue authoring isn't implemented for Jira yet
393
+ repo: owner/name
394
+ refine:
395
+ enabled: true
396
+ chain: refine # any chain that ends in steps.publishIssues()
397
+ concurrency: 1 # this lane's own budget, independent of watch.concurrency
398
+ ```
399
+
400
+ Every generated issue carries a `<prefix>:type:epic|feature|story|bug|task`
401
+ label. A container (a feature/epic — something else names it as `parent`)
402
+ gets only that label; a leaf additionally gets `<prefix>:refined` — **never**
403
+ `<prefix>:ready`. Promoting a leaf to `<prefix>:ready` is a deliberate human
404
+ decision: the refine lane never auto-promotes anything, so a spec fanning out
405
+ into twenty stories doesn't turn into twenty unattended chain runs and twenty
406
+ PRs with nobody having looked at the breakdown first. Once you do promote a
407
+ leaf, the existing build lane picks it up completely unchanged.
408
+
409
+ The spec issue itself gets a `spec-ready → refining → done`/`blocked`
410
+ lifecycle, same shape as the build lane, and a summary comment listing every
411
+ issue it created. Try it by hand first, against a real spec, before turning
412
+ on the daemon:
413
+
414
+ ```bash
415
+ spf refine "<spec text or path/to/spec.md>" --issue 42 # --issue renders a "## Parent: #42" back-reference
416
+ ```
417
+
418
+ `spf watch init` seeds the type labels alongside the state ones. This lane's
419
+ prompt (`assets/prompts/refiner/`) is adapted from a "tracer-bullet ticket"
420
+ decomposition skill — vertical slices, a `blocked_by` dependency graph, and
421
+ an expand/migrate/contract sequence for wide mechanical refactors — with a
422
+ gate (`gates.refinementWellFormed`) added on top to enforce the
423
+ container/leaf shape that skill left as prose convention rather than a
424
+ checked rule.
425
+
286
426
  ### GitHub (`issue_provider: github` and/or `code_host: github`)
287
427
 
288
428
  ```bash
289
429
  export GITHUB_TOKEN=... # classic PAT; spf doctor checks it's set
290
430
  ```
291
431
 
292
- `spf watch init` seeds `<prefix>:ready`/`working`/`review`/`done`/`blocked` labels with a color and description each — safe to re-run any time (creates what's missing, corrects any that drifted, leaves the rest alone).
432
+ `spf watch init` seeds `<prefix>:ready`/`working`/`review`/`done`/`blocked`/`spec-ready`/`refining`/`refined` labels, plus `<prefix>:type:epic`/`feature`/`story`/`bug`/`task` (used by the refine lane whether or not it's enabled), each with a color and description — safe to re-run any time (creates what's missing, corrects any that drifted, leaves the rest alone).
293
433
 
294
- A **classic** PAT (fine-grained tokens use different permission names — not covered here), scoped to the minimum that covers every call `spf watch`/`spf watch init` makes on GitHub: creating/editing labels, reading and labeling issues, posting comments, opening PRs, and reading PR/check-run status.
434
+ A **classic** PAT (fine-grained tokens use different permission names — not covered here), scoped to the minimum that covers every call `spf watch`/`spf watch init` makes on GitHub: creating/editing labels, reading and labeling issues, posting comments, opening PRs, reading PR/check-run status, and — with `watch.refine.enabled` — creating issues and linking them via the sub-issues API. All of it is already covered by `repo`/`public_repo`; refine needs no additional scope.
295
435
 
296
436
  | Target repo | Scope | Covers |
297
437
  |---|---|---|
@@ -372,6 +512,34 @@ genuinely informative, just worth expecting.
372
512
  Full field reference: `spf install-skill`'s installed skill
373
513
  (`references/config.md`).
374
514
 
515
+ ## Observability
516
+
517
+ Every run produces a complete trace: all events, phases, agent calls, and tool invocations stream into SQLite as they happen. The local trace stays the source of truth — prompts, envelopes, tool arguments, and your source code never leave the machine. Token counts and costs ride alongside.
518
+
519
+ ```bash
520
+ spf ui # browser-based visualizer over the trace
521
+ spf events <adw_id> --follow # live event stream, tailable
522
+ ```
523
+
524
+ The default export is SQLite only (`.spf/data/spf.db`). Optionally, you can export **spans only** (phase/agent/tool timing and allowlisted metadata) to an OpenTelemetry collector for integration with a trace UI or observability platform:
525
+
526
+ ```yaml
527
+ observability:
528
+ db: .spf/data/spf.db
529
+ poll_ms: 500
530
+ otel:
531
+ endpoint: https://your-otel-collector/v1/traces
532
+ service_name: spf # optional; default "spf"
533
+ headers: # optional; e.g. auth headers
534
+ Authorization: Bearer ...
535
+ ```
536
+
537
+ OTEL export is **explicit config only** — an unrelated shell variable cannot become a data-egress switch. OTEL is strictly a spans-only export: each phase is a span with child spans for agent calls and tool calls, annotated with phase status, agent model, token/cost counts, and gate results. This export never blocks a run: if the collector is slow or unreachable, SPF continues normally and logs a single line per run when export fails (not one per batch), and reports the number of dropped spans on the final flush as `spf.otel.dropped_spans` for the backend to surface. The complete trace stays in SQLite regardless.
538
+
539
+ **Attribute allowlist**: only phase name/kind/owner/status, chain name, adw_id, agent name/model/coding_agent, gate name + passed + violation count, token counts and cost, and durations. Prompts, envelopes, tool arguments, and your source code never leave — that guarantee is enforced in code, not just in documentation.
540
+
541
+ See `assets/skill/references/config.md`'s `observability` section for the full field reference.
542
+
375
543
  ## What's in this repo
376
544
 
377
545
  ```
@@ -385,7 +553,7 @@ super-portable-software-factory/
385
553
  └── assets/
386
554
  ├── defaults/ # the packaged default roster
387
555
  ├── prompts/ # default system.md + user.md per starter agent
388
- └── skill/ # an optional Claude Code skill — `spf install-skill` to use it
556
+ └── skill/ # a Claude Code skill — `spf init` installs it by default (`--no-skills` to opt out), or `spf install-skill` by hand
389
557
  ```
390
558
 
391
559
  There's no template stamped into your repo. Everything above ships inside the installed npm package; a repo you run `spf` against only ever gains a `.spf/` directory, and only if you ask for one.
@@ -34,6 +34,12 @@ defaults:
34
34
  observability:
35
35
  db: .spf/data/spf.db # tracer writes here directly; the UI polls it
36
36
  poll_ms: 500 # visualizer live-poll cadence
37
+ # Optional OpenTelemetry spans export (explicit config only; no ambient env activation)
38
+ # otel:
39
+ # endpoint: https://your-otel-collector/v1/traces
40
+ # service_name: spf # optional; default "spf"
41
+ # headers: # optional; e.g. auth headers
42
+ # Authorization: Bearer ...
37
43
 
38
44
  # No default quality checks or suites are shipped — an unconfigured chain
39
45
  # that needs one fails loudly at agents.validate() time (before anything
@@ -101,6 +107,22 @@ agents:
101
107
  # No tester agent: running the suite is a known command, so it is a kind="code"
102
108
  # phase over core/quality.ts. See SKILL.md hard rule 8.
103
109
 
110
+ - name: refiner
111
+ thinking: high
112
+ color: "#34d399"
113
+ purpose: Decompose a product spec into a feature/story-or-bug tree of vertical slices; change nothing.
114
+ prompt_engineering:
115
+ system: refiner/system.md
116
+ user: refiner/user.md
117
+ writes: [] # read-only w.r.t. the repo; refine_plan.md lands in context_handoff_dir, not here
118
+ tools: # full recon; write only for refine_plan.md, no edit
119
+ - read
120
+ - grep
121
+ - find
122
+ - ls
123
+ - bash
124
+ - write
125
+
104
126
  - name: reviewer
105
127
  model: openai/gpt-5.6-terra
106
128
  thinking: high
@@ -0,0 +1,53 @@
1
+ # Refiner Agent
2
+
3
+ ## Purpose
4
+
5
+ Decompose a product spec into a feature/story-or-bug tree of vertical slices the factory can build one at a time. Change nothing.
6
+
7
+ ## Instructions
8
+
9
+ - Read-only: explore the codebase to ground the decomposition, never write to it.
10
+ - Use the project's domain glossary vocabulary in every title and description, if the repo has one. Respect existing ADRs in any area you're touching — a slice that would contradict one is a slice to reconsider, not to write down anyway.
11
+ - You inherit the operator's shell environment — their PATH, toolchains and credentials are already live. Call tools by bare name (`bun`, `uv`, `pytest`); never hunt for a binary or fall back to an absolute `/usr/bin/*` path.
12
+ - Judge any command you run by its exit status, never by scanning its output for words. `error` or `not found` inside passing output is text, not a failure.
13
+ - Write your working notes to `<context_handoff_dir>/refine_plan.md` before emitting your Report JSON.
14
+
15
+ ## The tree
16
+
17
+ Every node you produce is either a **container** (`epic` or `feature` — exists only to group other nodes; never itself a unit of work) or a **leaf** (`story`, `bug`, or `task` — the independently workable unit). A node is a container exactly when some other node names it as `parent`; everything else is a leaf. At least one leaf is required — a decomposition that is all containers has produced nothing to build.
18
+
19
+ Emit a **flat list**, not nested JSON: each node names its `parent` by another node's `key`, or leaves `parent` empty for a top-level feature/epic. A flat list with parent pointers is far more reliable to produce correctly than a recursive tree, and it's what lets `blocked_by` point at any other node — sibling or not.
20
+
21
+ ## Vertical-slice rules (for every leaf)
22
+
23
+ - Each slice cuts a narrow but COMPLETE path through every layer it touches (schema, API, UI, tests) — vertical, never a horizontal slice of one layer.
24
+ - A completed slice is demoable or verifiable on its own.
25
+ - Size each slice to fit in a single fresh context window for the builder that will implement it — when in doubt, split.
26
+ - Any prefactoring the spec implies should be its own slice, sequenced first via `blocked_by`, not folded into the first "real" slice.
27
+
28
+ ## Dependencies: a DAG, not a tree
29
+
30
+ Give every node its `blocked_by`: the other nodes' `key`s that must land first. A node with no blockers can start immediately — leave `blocked_by` empty rather than inventing an order where none is required. The factory works the **frontier**: any leaf whose blockers are all done. For a purely linear chain that means top to bottom; for anything wider, only real dependencies belong in `blocked_by` — an artificial one just stalls the frontier.
31
+
32
+ ## Wide refactors — the one exception to vertical slicing
33
+
34
+ A **wide refactor** is one mechanical change (rename a column, retype a shared symbol) whose blast radius fans across the codebase, so a single edit breaks thousands of call sites and no vertical slice can land green on its own. Sequence it as **expand -> migrate -> contract**, each stage its own leaf(s):
35
+
36
+ - **expand**: add the new form beside the old so nothing breaks yet.
37
+ - **migrate**: batch the call sites by blast radius (per package, per directory) — each batch its own leaf, `blocked_by` the expand leaf. CI stays green throughout, because the old form still exists alongside the new one.
38
+ - **contract**: delete the old form — `blocked_by` every migrate batch.
39
+
40
+ If even a batch can't stay green alone, keep the same three-stage sequence but let the migrate batches share an integration branch that all block a final integrate-and-verify leaf — green is promised only there, not at every batch.
41
+
42
+ ## Writing titles and bodies
43
+
44
+ - Title and body describe end-to-end behavior from the user's (or the next engineer's) perspective — not a layer-by-layer implementation list.
45
+ - Avoid specific file paths or code snippets; they go stale fast. Exception: if your exploration surfaced a snippet that encodes a decision more precisely than prose can (a state machine, a reducer, a schema, a type shape), inline it and note briefly where it came from. Trim to the decision-rich part, not a working demo.
46
+ - Do **not** write a "Blocked by" or "Parent" section into `body` yourself — the harness renders both from `blocked_by`/the source issue automatically, with real issue numbers once everything is created. Writing your own would go stale or duplicate the real one.
47
+ - `body` should read as `## What to build` followed by `## Acceptance criteria` (a checklist).
48
+
49
+ ## Subagents
50
+
51
+ `subagent_create` / `_continue` / `_list` / `_remove` fan out exploration — one per subsystem or open question — when the spec spans more than you can read cheaply. Give each a self-contained task, hold it to read-only work, and omit `model`.
52
+
53
+ They run in the background. **Wait for every one you spawned to report before writing `refine_plan.md` or your Report JSON.** Skip them when a few reads would do.
@@ -0,0 +1,70 @@
1
+ # Refine Task
2
+
3
+ ## Variables
4
+
5
+ ### prompt
6
+
7
+ {{prompt}}
8
+
9
+ ### previous_envelope
10
+
11
+ {{previous_envelope}}
12
+
13
+ ### context_handoff_dir
14
+
15
+ {{context_handoff_dir}}
16
+
17
+ ## Task
18
+
19
+ Decompose the product spec in `prompt` into a feature/story-or-bug tree, following every rule in your system instructions (vertical slices, the `blocked_by` DAG, expand/migrate/contract for wide refactors, no "Blocked by"/"Parent" text of your own).
20
+
21
+ 1. Explore the codebase only as far as you need to ground the decomposition in what actually exists.
22
+ 2. Write your working notes to `<context_handoff_dir>/refine_plan.md`.
23
+ 3. Emit your `Report` JSON, declaring that one file in `artifacts`.
24
+
25
+ ## Report
26
+
27
+ Respond with ONLY valid JSON matching `RefineOutput` — no prose before or after:
28
+
29
+ ```json
30
+ {
31
+ "status": "success",
32
+ "summary": "<one sentence: how many features and how many leaves, and the shape of the dependency chain>",
33
+ "artifacts": ["<context_handoff_dir>/refine_plan.md"],
34
+ "notes_for_next_agent": "<anything the publish step or a human reviewer should know>",
35
+ "issues": [
36
+ {
37
+ "key": "F1",
38
+ "kind": "feature",
39
+ "title": "<feature title>",
40
+ "body": "## What to build\n\n<end-to-end behavior this feature covers>\n\n## Acceptance criteria\n\n- [ ] <criterion>",
41
+ "parent": "",
42
+ "blocked_by": []
43
+ },
44
+ {
45
+ "key": "S1",
46
+ "kind": "story",
47
+ "title": "<leaf title>",
48
+ "body": "## What to build\n\n<end-to-end behavior from the user's perspective>\n\n## Acceptance criteria\n\n- [ ] <criterion>\n- [ ] <criterion>",
49
+ "parent": "F1",
50
+ "blocked_by": []
51
+ },
52
+ {
53
+ "key": "S2",
54
+ "kind": "bug",
55
+ "title": "<leaf title>",
56
+ "body": "## What to build\n\n<the fix, described end-to-end>\n\n## Acceptance criteria\n\n- [ ] <criterion>",
57
+ "parent": "F1",
58
+ "blocked_by": ["S1"]
59
+ }
60
+ ]
61
+ }
62
+ ```
63
+
64
+ Rules on the shape (enforced by a gate — a violation sends this back to you as a correction, not a silent acceptance):
65
+
66
+ - `key` is your own local id for this run, unique within `issues` — never a real tracker id.
67
+ - A node is a **container** (`kind: "epic"` or `"feature"`) exactly when some other node names it in `parent`; every other node is a **leaf** (`kind: "story"`, `"bug"`, or `"task"`) and must not be a container. At least one leaf is required.
68
+ - `parent` is another node's `key`, or `""` for a top-level feature/epic.
69
+ - `blocked_by` is a list of other nodes' `key`s — real dependencies only, and no cycles (through `parent` or `blocked_by`, or both together).
70
+ - `body` is `## What to build` then `## Acceptance criteria` only — no "Blocked by" or "Parent" section; those are rendered for you once every node has a real issue number.
@@ -13,4 +13,4 @@ Confirm that what was built is what was asked for. This is not testing.
13
13
  - Change nothing. Findings go back to the builder — that is the only repair path.
14
14
  - `approved` is true ONLY when every requirement is met and `blocking` is empty. Every blocking item names the specific gap, so the builder can fix it without guessing.
15
15
  - You inherit the operator's shell environment — their PATH, toolchains and credentials are already live. Call tools by bare name (`bun`, `uv`, `git`); never hunt for a binary or fall back to an absolute `/usr/bin/*` path.
16
- - Judge any command you run by its exit status, never by scanning its output for words. `error` or `not found` inside passing output is text, not a failure.
16
+ - Judge any command you run by its exit status, never by scanning its output for words — except tools whose output is the product, such as `git diff`, or a review tool you were asked to consult, which you read as evidence, never as a verdict. `error` or `not found` inside passing output is text, not a failure.
@@ -45,6 +45,7 @@ phases`/`spf events` exist so nobody has to guess the schema again).
45
45
  | "add/retune an agent or model" | `roster.md` |
46
46
  | "add a chain / a phase / an output type / a gate" | `authoring_chains.md` |
47
47
  | "what is SPF" | `spf_overview.md` |
48
+ | "fold a third-party review tool's findings into the reviewer" | `cookbooks/ocr_reviewer.md` |
48
49
  | envelope/gate/session contract | `references/handoff.md` |
49
50
  | trace schema, spend vs. context | `references/observability.md` |
50
51
  | config field reference | `references/config.md` |
@@ -1,13 +1,86 @@
1
1
  # Authoring Chains
2
2
 
3
3
  Composing a new chain, extending an existing one, and adding the engine
4
- primitives a chain needs (an output type, a gate, a step) are one skill with
5
- three doors. All three live in `src/` inside the SPF package itself — there
6
- is no per-repo copy to edit. If you need to change engine behavior for a
7
- specific target repo without forking the package, that's `spf eject` (prints
8
- the path to the installed package's `src/` for you to copy and load from your
9
- own `.spf/` engine-level changes are the one thing `.spf/` config can't
10
- express).
4
+ primitives a chain needs (an output type, a gate, a step) is one skill with
5
+ four doors. Engine primitives (a new gate, a new envelope type, a modified
6
+ phase primitive) live in `src/` inside the SPF package itself those do
7
+ require forking the package, because they change what every repo's chains
8
+ can mean. But COMPOSING a chain out of the existing step factories does not:
9
+ that's what a repo-local chain (below) is for. Read that section first if
10
+ you just want your own chain in your own repo — reach for the rest of this
11
+ document only once you've decided you need a new primitive.
12
+
13
+ ## Repo-local chains (`.spf/chains/*.yaml`)
14
+
15
+ The zero-fork door: a `.spf/chains/*.yaml` file in a target repo names
16
+ existing step factories from `spf`'s own `src/chains/steps.ts` and passes
17
+ them params — never imports or runs code from the target repo. `spf init`
18
+ scaffolds `.spf/chains/example.yaml`, fully commented out, showing the shape.
19
+
20
+ ```yaml
21
+ # .spf/chains/ship-it.yaml — spf ship-it "<prompt>" / spf run ship-it "<prompt>"
22
+ name: ship-it
23
+ describe: plan, build, test, land — with our own reviewer in the loop
24
+ steps:
25
+ - step: request # every chain opens with this
26
+ - step: plan
27
+ owner: architect # any agent named in spf.config.yaml
28
+ - step: build
29
+ retries: 2
30
+ extraGates: [jsonParses] # additive — see below
31
+ - step: fixLoop
32
+ suite: test
33
+ - step: commit
34
+ onlyIfAccepted: true
35
+ ```
36
+
37
+ **Shape.** One file, one chain — the filename is a handle a problem can point
38
+ at. `steps` is a flat list; each entry names a `step:` and its params sit as
39
+ FLAT SIBLINGS of `step:` — never nested under a `params:` key. Param names are
40
+ exactly the step factory's `opts` keys, camelCase (`extraGates`, `fixExtraGates`,
41
+ `onlyIfAccepted`, ...). An unknown param, an unknown step, or a param of the
42
+ wrong type is a load-time problem naming the step index and what's allowed —
43
+ never a silently-ignored typo and never a runtime surprise.
44
+
45
+ **Step vocabulary** (`repo_chains.STEP_NAMES`) — the same factories the table
46
+ in Step 2 below documents: `request`, `plan`, `build`, `scout`, `promptOnly`,
47
+ `qualityCheck`, `fixLoop`, `reviseLoop`, `commit`, `changes`, `document`,
48
+ `refine`, `publishIssues`.
49
+
50
+ **Gates are additive only, and per-step.** A step's built-in gates
51
+ (`diffMatchesClaims` on `build`, `verdictConsistent` on `reviseLoop`'s review
52
+ phase, ...) are non-removable — there is no `gates:` param that replaces
53
+ them, and there must never be one (see `GATE_ALLOWLIST`'s comment in
54
+ `steps.ts` for why: the first thing anyone deletes under deadline pressure is
55
+ the gate that keeps failing). `extraGates` can only ADD from an explicit
56
+ allowlist SCOPED to what that param's envelope actually supports:
57
+ `artifactsExist`/`filesNonEmpty`/`jsonParses` anywhere (they only read
58
+ `envelope.artifacts`, present on every envelope); `diffMatchesClaims` only on
59
+ a param whose phase produces a `BuildOutput` (`build.extraGates`,
60
+ `fixLoop.fixExtraGates`, `reviseLoop.reviseExtraGates`); `verdictConsistent`
61
+ only on `reviseLoop.extraGates` (the review phase, a `ReviewOutput`). Naming
62
+ a gate outside its param's list is a load problem, not a chain that loads
63
+ clean and then fails (or vacuously passes) its gate on every run.
64
+
65
+ **Names are guarded.** A chain name must be lowercase letters/digits/`._-`
66
+ (it's typed on the command line). It cannot collide with an `spf` subcommand
67
+ (`run`, `list`, `init`, `watch`, ...) — `spf watch` would always run the
68
+ daemon, never a same-named chain. Two repo-chain files can't claim the same
69
+ `name` either: the trace's `chain_name` column has to keep meaning one thing.
70
+
71
+ **The `spf watch` divergence.** `spf watch` registers `.spf/chains/` from the
72
+ MAIN repo anchor ONCE, at daemon start — not per-issue, not per-worktree. A
73
+ chain file edited on an issue branch (inside the worktree `spf watch` checks
74
+ that branch out into) is NOT what runs for that issue; the daemon keeps using
75
+ whatever `.spf/chains/` looked like when it started. This is deliberate: the
76
+ disposer stays the OPERATOR's, never the branch's, which is exactly what
77
+ keeps an agent from rewriting its own quality gate mid-run by editing a chain
78
+ file as part of the change it's making.
79
+
80
+ Everything below this section is about the OTHER three doors: designing a
81
+ brand-new built-in chain, adding a step to the vocabulary above, or adding an
82
+ engine primitive (gate/envelope type/phase primitive) — none of which a
83
+ target repo needs, or can do, on its own.
11
84
 
12
85
  ## Step 1 — design the chain before writing code
13
86
 
@@ -24,6 +97,16 @@ the step list itself — see Step 2 — so it can't drift from what actually run
24
97
  | test | code | quality | — | (suite pass/fail is the phase's own result) |
25
98
  | commit | code | git | — | — |
26
99
 
100
+ If you need to change engine behavior for a specific target repo without forking
101
+ the package, that's `spf eject`. It copies the installed engine's compiled
102
+ artifacts (`dist/core/` and `dist/chains/`) out to `.spf/engine/` for reference
103
+ or reading — but understand that these files are **not wired into any spf
104
+ command**: editing them changes nothing about how spf itself runs. Engine-level
105
+ changes (a new gate, a new envelope type, a modified phase primitive) have no
106
+ config surface by design, because the codebase is a single shared engine all
107
+ repos trust to sequence their work the same way. If you are genuinely changing
108
+ the engine semantics, you are forking the package itself, not ejecting a copy.
109
+
27
110
  **Ownership rules**, non-negotiable:
28
111
 
29
112
  - **`engineer`** phases capture input only — no agent, no code decision.