@pmelab/gtd 2.7.0 → 3.0.0

package/docs/design/work-packages.md

@@ -0,0 +1,194 @@
+# Design: the builder / work-packages flow as pure workflow configuration
+
+> Status: LANDED, then relocated (2026-07-22) — Option A shipped in the bundled
+> default workflow's `picking` state; the bundled default has since been
+> drastically simplified and no longer includes it. The full Option-A machine
+> now lives as a copy-paste example at
+> [docs/examples/advanced-workflow.md](../examples/advanced-workflow.md). The
+> rest of this document is kept as the decision record: why Option A was chosen
+> over B/C/D, and the process-per-task topology (§6) remains a documented,
+> unimplemented recipe.
+>
+> Context (as originally written): v2 had an engine-level package loop
+> (`hasPackages` / `packagesRemaining` guards, the `close-package` label,
+> `removePackageDir`). v3 deleted all of it; the bundled default's `building`
+> state used to implement the entire task set in ONE monolithic agent turn. This
+> doc plans how to get the incremental task-by-task loop back **without the
+> engine ever learning what a "package" is** — it stays a concern of the
+> workflow configuration.
+
+## 1. What the loop actually needs
+
+The per-task cycle is: _pick the next task → implement it → check → (fix loop) →
+close the task → if tasks remain, loop; else review_. Decomposed into v3
+vocabulary:
+
+| Need                              | Already expressible?                                                                                                                                                                                           |
+| --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Task queue                        | Yes — files under a glob (e.g. `.gtd/tasks/NN-name.md`), written by `decompose`.                                                                                                                               |
+| "Close a task"                    | Yes — the actor deletes the task file; the diff carries `D .gtd/tasks/…`.                                                                                                                                      |
+| Per-task check + fix loop         | Yes — the existing `checking`/`fixing` shape, unchanged.                                                                                                                                                       |
+| Feed ONE task into the prompt     | Mostly — a file naming the current task can be `it.read()` into the building prompt.                                                                                                                           |
+| **The exit branch: "none left?"** | **No.** Patterns match the pending **diff**; deleting the _last_ task file produces the same diff shape as any other (`D .gtd/tasks/…`). Emptiness of a glob is a **tree** property, invisible to the grammar. |
+
+So the whole design question is: who decides "the queue is empty", and how does
+that decision become a diff the patterns can see? Everything below is an answer
+to only that.
+
+## 2. Option A — scripted arbiter state (pure config, deterministic) — RECOMMENDED
+
+Apply the `checking` discipline to the queue itself: a `script` state whose only
+job is to inspect the tree (which patterns can't) and leave a file-op verdict
+(which patterns can). Mechanics in the script, semantics in the `on` map — no
+trust in an agent, no engine change.
+
+```yaml
+decompose:
+  actor: agent
+  prompt: |
+    … write ordered task files under .gtd/tasks/ …
+  on:
+    "* .gtd/tasks/**": picking
+
+picking: # the deterministic queue arbiter
+  actor: check
+  script: |
+    next=$(ls .gtd/tasks/*.md 2>/dev/null | head -n 1)
+    if [ -n "$next" ]; then
+      printf '%s' "$next" > .gtd/NEXT.md
+    else
+      rm -f .gtd/NEXT.md
+    fi
+  on:
+    "D .gtd/NEXT.md": reviewing # queue just emptied — MUST precede the `*` row
+    "* .gtd/NEXT.md": building # queue non-empty (A first time, M after)
+    "C": reviewing # queue empty and NEXT.md never existed
+
+building:
+  actor: agent
+  prompt: |
+    Implement exactly ONE task: <%~ it.read(".gtd/NEXT.md") %> — read that
+    file, do the work, delete the task file when done. Touch nothing else
+    under .gtd/.
+  on:
+    "* **": checking
+
+checking:
+  # unchanged, except green loops back to the arbiter:
+  on:
+    "A .gtd/FEEDBACK.md": fixing
+    "M .gtd/FEEDBACK.md": fixing
+    "C": picking
+```
+
+Properties:
+
+- **Deterministic.** The branch is a shell `ls`, not an agent's memory. Every
+  verdict lands in the commit history as an ordinary `gtd(check): <state>` turn.
+- **`NEXT.md` is both the verdict and the context** — the building prompt
+  interpolates it, so each building turn sees exactly one task.
+- The three `on` rows are total over the script's possible outcomes
+  (write/overwrite, delete, no-op) — no refusal path.
+- Cost: one extra state and one extra driver round-trip per task (same cost
+  `checking` already pays; the loop already executes scripts via `gtd run`).
+
+## 3. Option B — agent-encoded verdict (pure config, zero extra states)
+
+The v3 verdict discipline says branch outcomes are encoded by _which files the
+actor writes or deletes_ (`D FEEDBACK.md` = approve). Apply it here: `decompose`
+also writes a marker `.gtd/tasks/OPEN`; the building/closing prompt instructs:
+_implement the next task and delete its file; if it was the last one, also
+delete `.gtd/tasks/OPEN`_. Declaration order does the rest:
+
+```yaml
+building:
+  on:
+    "D .gtd/tasks/OPEN": reviewing # declared first — wins when present
+    "* **": checking
+```
+
+- Pro: no extra state, no script execution needed (works for drivers that only
+  handle prompt/message states).
+- Con: correctness rests on the agent honouring the marker protocol. The failure
+  mode is soft (a forgotten marker just means one more "none left, delete the
+  marker" turn), but the branch is not deterministic and never appears as a
+  machine-authored verdict.
+
+Documented as the zero-infrastructure variant; A is preferred wherever the
+driver can run scripts.
+
+## 4. Option C — generic engine helper: tree-predicate guards on patterns
+
+The one candidate engine change, if the arbiter round-trip ever proves too
+heavy. The grammar gains an optional **tree predicate** appended to a pattern:
+`<event> & N <glob>` ("and no file matching glob exists in the tree after the
+pending changes") / `& E <glob>` ("and at least one exists"):
+
+```yaml
+building:
+  on:
+    "D .gtd/tasks/** & N .gtd/tasks/**": reviewing # closed the LAST task
+    "D .gtd/tasks/**": picking # closed a task, more remain
+```
+
+- Still fully agnostic: the engine learns _glob emptiness_, never "packages".
+- δ-purity holds in the same sense as today: the decision is a function of (HEAD
+  tree + pending diff); the tree listing is already available at the edge
+  (`changedPaths` / `WorktreeReader`).
+- The `&` form keeps rows **event-triggered** (a bare `N …` row that could fire
+  on any unrelated diff would be a foot-gun); the predicate only guards.
+- Cost: pattern parser/matcher/validation growth, resolve needs a tree listing,
+  STATES.md §3 grows a second concept, property tests. Deliberately **deferred**
+  — Option A covers the need with zero engine surface; this only pays for itself
+  if arbiter states show up in every workflow and feel like boilerplate.
+
+## 5. Option D — template enumeration helper: `it.ls(glob)`
+
+Small, orthogonal, generic: add `ls(glob) → string[]` (sorted, working-tree)
+next to `read(path)` in the template context.
+
+- Lets Option A's script shrink to pure verdict-writing and lets prompts and
+  messages show the queue ("Remaining tasks:\n<%=
+  it.ls('.gtd/tasks/*.md').join('\n') %>") without shell gymnastics.
+- Lets Option B's prompt state the ground truth instead of trusting agent memory
+  ("these tasks remain: …; if this list has one entry, also delete the marker").
+- No new decision power (templates render content, they don't route), so it
+  cannot compromise the δ discipline.
+
+## 6. A topology variant worth knowing: process-per-task
+
+Nothing stops a workflow from making **each task its own process**: the per-task
+cycle ends in its own `commit:` state, squashing just that task's work into one
+commit; the squash boundary resolves back to the initial state, whose router (an
+Option-A-style arbiter at or right after `initial`) picks the next task or falls
+through to the review/idle path.
+
+Why it's attractive:
+
+- **Per-task commits** — history reads one commit per task instead of one per
+  cycle.
+- **Per-task budgets** — `retry` visit-counts are process-scoped, so today a
+  10-task cycle POOLS `fixing`'s `max: 3` across all tasks; task-sized processes
+  give each task its own cap. (This interaction exists regardless of which
+  option is chosen and should be called out in docs.)
+- Smaller diffs at review time.
+
+Why it's not the default recommendation: the initial state doubles as the human
+entry point, so the router shape is more intricate (the arbiter must distinguish
+"mid-flight, tasks remain" from "fresh repo, wait for a human"), and cycle-level
+review/squash semantics change meaningfully. A worked example belongs in
+`docs/configuration.md` once the basic loop lands.
+
+## 7. Recommendation
+
+1. **Now (config-only, no engine change):** re-author the bundled default's
+   builder segment as the Option-A loop
+   (`decompose → picking ⇄ building ⇄ checking/fixing → reviewing`), with the
+   `NEXT.md` arbiter script and the one-task building prompt. Document Option B
+   in `docs/configuration.md` as the script-less variant.
+2. **Now (one small generic helper):** add `it.ls(glob)` (Option D) — cheap,
+   broadly useful, and it makes both A and B materially better.
+3. **Deferred:** Option C (`& N/E <glob>` tree-predicate guards) — revisit only
+   if arbiter boilerplate becomes a demonstrated pain across real workflows.
+4. **Docs either way:** call out the retry-cap pooling interaction (§6) and the
+   process-per-task topology as a recipe.

package/docs/development.md

@@ -0,0 +1,101 @@
+# Development
+
+```bash
+npm install
+npm run dev          # run from source, no build (node dev/run.mjs)
+npm run build        # tsdown → dist/gtd.bundle.mjs
+npm test             # format:check, typecheck, lint, unit + e2e tests, fallow
+npm run test:unit    # vitest unit tests (the pure resolver) — --project unit
+npm run test:e2e     # gherkin e2e via vitest + quickpickle — --project e2e
+npm run test:mutation # StrykerJS mutation testing (manual only, ~2 min)
+npm run typecheck
+npm run lint
+```
+
+## Pre-commit hook
+
+A pre-commit hook is installed automatically via the `prepare` script when you
+run `npm install` on a fresh clone — no manual setup needed. The hook runs
+[lint-staged](https://github.com/lint-staged/lint-staged) with
+[oxfmt](https://oxc.rs/docs/guide/usage/formatter.html), formatting every staged
+file before each commit (`oxfmt --no-error-on-unmatched-pattern --write`),
+mirroring the `format:check` step enforced in CI (`oxfmt --check .`).
+
+## Templates
+
+Every state's content (`script`/`prompt`/`message`/`commit`) is an Eta template
+string, rendered by `src/PatternTemplates.ts`'s `renderStateTemplate` against
+the variable set documented in
+[Configuration](configuration.md#template-variables). There is no template-file
+registry to keep in sync: a workflow's content strings are either inline in its
+YAML/`.gtdrc` or a `./`-relative file reference auto-inlined at config load
+(`src/PatternConfig.ts`) — see
+[Configuration](configuration.md#content-values-inline-or-a-file-reference). The
+bundled default workflow (`src/workflows/default.yaml`) is imported as raw text
+via tsdown's `.yaml`-as-text loader (`tsdown.config.ts`) and compiled through
+the same path a `.gtdrc` `workflow:` key goes through
+(`src/workflows/default.ts`) — no privileged code path.
+
+`npm run dev` runs `src/main.ts` directly via Node's native TypeScript
+type-stripping (requires Node 22.6+). It registers `dev/hooks.mjs`, which
+resolves `./Foo.js` specifiers to the on-disk `./Foo.ts` and imports `*.yaml`
+files as raw text, mirroring tsdown's `.yaml`-as-text loader — this is what lets
+`src/workflows/default.yaml` resolve when running from source.
+
+The decision core is pure and IO-free: the pattern machine's shape (states,
+patterns, retry) is a plain-data `WorkflowDefinition` (`src/PatternMachine.ts`),
+and the same module's `resolveState`/`step` are the pure resolver/interpreter
+over it — so the whole engine is trivially unit-testable in isolation; all
+git/filesystem/template IO is confined to the edge (`src/Edge.ts`).
+
+`npm run build` produces `dist/gtd.bundle.mjs`, which npm exposes as the `gtd`
+binary via the `bin` field in `package.json`.
+
+## The LSP server
+
+`gtd lsp` (`src/Lsp.ts`) serves editor tooling over `.gtd/`'s two steering-file
+formats — document symbols and diagnostics for `.gtd/TODO.md`'s open questions
+and `.gtd/REVIEW.md`'s review chunks/hunks, plus check/uncheck code actions for
+the latter. It's keyed on file NAME, not workflow state: a v3 workflow declares
+no state→file mapping (a state names no file), so the server needs no
+git/`.gtdrc` dependency at all and serves whatever content the editor hands it
+over the LSP protocol, exactly like any other document.
+
+`src/OpenQuestions.ts` and `src/ReviewDoc.ts` are the pure parsers behind both
+the LSP and the bundled default workflow's own bash validators
+(`todo-validating`/`review-validating` in `src/workflows/default.yaml`) — see
+[docs/design/steering-file-loops.md](design/steering-file-loops.md) for the
+"executable spec ↔ bash validator" contract linking the two independent
+implementations, and each module's own doc comment for the format it defines.
+Their unit tests are that format's spec tests.
+
+## Mutation testing
+
+Run mutation testing on-demand with `npm run test:mutation` (StrykerJS) — never
+run it as part of routine development; it is a deliberate, manually-triggered
+check. `stryker.config.json`'s `mutate` list names the v3 pattern-machine module
+set (`src/PatternMachine.ts`, `src/PatternConfig.ts`, `src/PatternTemplates.ts`,
+`src/Edge.ts`, `src/Config.ts`, `src/Format.ts` — see
+[Architecture](../AGENTS.md#the-pattern-machine-module-map) for the v3 module
+map), plus the steering-file format parsers and the LSP built on them
+(`src/OpenQuestions.ts`, `src/ReviewDoc.ts`, `src/Lsp.ts` — see
+["The LSP server"](#the-lsp-server) below). `src/Lsp.ts`'s protocol-adapter tail
+(`startLspServer`) is exercised only by the `@live` e2e (excluded from the
+mutation run's Cucumber harness), so mutants confined to that tail may survive
+without it being a coverage gap in the ordinary test suite — the pure
+symbol/edit/diagnostic builders above it are unit-tested and mutation-checked
+normally.
+
+`src/Git.ts` is excluded: the Cucumber harness stubs git at the Effect boundary,
+so `Git.ts` mutants have zero in-memory coverage.
+
+The HTML report lands in `reports/mutation/mutation.html` (git-ignored).
+
+## Releasing
+
+Releases are automatic. Push releasable Conventional Commits (`fix:`, `feat:`,
+or breaking changes) to `main` and the Release workflow runs the tests, then
+`npx semantic-release`. Semantic-release computes the next version, writes it
+into `package.json`, builds the bundle, commits the bump back as
+`chore(release): X.Y.Z [skip ci]`, tags `vX.Y.Z`, and creates the GitHub release
+with `gtd.bundle.mjs` attached.