@multileaf/ai-workflow 0.11.0 → 0.13.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +25 -0
- package/dist/blocking-reasons.d.ts +1 -1
- package/dist/blocking-reasons.js +30 -0
- package/dist/digest.d.ts +48 -0
- package/dist/digest.js +106 -0
- package/dist/eval/artifact-evals.d.ts +42 -0
- package/dist/eval/artifact-evals.js +141 -0
- package/dist/generator.js +1 -1
- package/dist/human-approval.d.ts +13 -2
- package/dist/human-approval.js +43 -8
- package/dist/legacy-migration.d.ts +45 -0
- package/dist/legacy-migration.js +175 -0
- package/dist/orchestration.d.ts +38 -1
- package/dist/orchestration.js +82 -3
- package/dist/quality-gates.d.ts +1 -1
- package/dist/quality-gates.js +23 -0
- package/dist/risk-assess.d.ts +4 -0
- package/dist/risk-assess.js +5 -1
- package/dist/standards.d.ts +34 -0
- package/dist/standards.js +159 -0
- package/dist/types.d.ts +15 -1
- package/dist/types.js +24 -0
- package/dist/workflow.d.ts +7 -0
- package/dist/workflow.js +504 -59
- package/docs/backlog.md +13 -3
- package/docs/continuation-inventory.md +21 -16
- package/docs/opportunities/OPP-009-process-entry-points.md +56 -0
- package/docs/opportunities/OPP-010-orchestration-worktree-isolation.md +52 -0
- package/docs/opportunities/OPP-011-evals-redteam-generated-artifacts.md +56 -0
- package/docs/opportunities/OPP-012-standards-discovery.md +51 -0
- package/docs/opportunities/OPP-013-read-optimized-artifacts-digest-layer.md +52 -0
- package/docs/opportunities/OPP-014-paired-verification-contract.md +54 -0
- package/docs/opportunities/OPP-015-token-recovery-governance.md +52 -0
- package/docs/opportunities/OPP-016-behavioral-eval-scenarios.md +44 -0
- package/docs/opportunities/OPP-017-orchestration-depth.md +50 -0
- package/docs/opportunities/OPP-018-legacy-migration.md +59 -0
- package/docs/opportunities/README.md +32 -10
- package/docs/release-process.md +1 -1
- package/package.json +1 -1
- package/resources/skills/implementation-planning.md +1 -1
- package/resources/templates/brainstorm.md +0 -1
- package/resources/templates/decision-record.md +0 -1
- package/resources/templates/implementation-evidence.md +1 -1
- package/resources/templates/implementation-plan.md +2 -1
- package/resources/templates/migration-report.md +0 -1
- package/resources/templates/security-review.md +0 -1
- package/resources/templates/task.md +0 -1
- package/resources/templates/technical-design.md +0 -1
- package/resources/templates/test-plan.md +0 -1
- package/resources/templates/verification-report.md +2 -1
package/docs/backlog.md
CHANGED
|
@@ -116,11 +116,21 @@ Derived from the Gentle-AI ODD/RDD analysis; scope and rejected alternatives rec
|
|
|
116
116
|
- [x] **OPP-001** Add a light lane for small, understood work (proportional ceremony). _Acceptance:_ see [OPP-001](../opportunities/OPP-001-proportional-ceremony-light-lane.md); no durable spec/plan artifacts for light tasks, evidence and lane rationale still recorded, explicit escalation to the full lane on scope growth, full lane unchanged.
|
|
117
117
|
- [x] **OPP-002** Add `aiw risk assess` — read-only, content-based risk classification with an exact runnable next command. _Acceptance:_ see [OPP-002](../opportunities/OPP-002-risk-assess-command.md); deterministic and offline by default, no `.aiw/` mutation, escalation raises and never lowers tier, `next_command` executes as printed.
|
|
118
118
|
- [x] **OPP-003** Ship an optional `adversarial-review` package (blind judges + refuter with proof-carrying findings). _Acceptance:_ see [OPP-003](../opportunities/OPP-003-adversarial-review-package.md); severe findings without reproducible evidence degrade to advisory; refutation outcomes are corroborated/refuted/inconclusive with proof references; human acceptance still required.
|
|
119
|
-
- [
|
|
119
|
+
- [x] **OPP-004** Emit typed reason codes and exact continuation commands on every blocking exit. _Acceptance:_ see [OPP-004](../opportunities/OPP-004-exact-continuation-commands.md); every block carries a reason code and either a tested runnable `next_command` or an explicit terminal statement; continuation inventory stays complete in CI.
|
|
120
120
|
- [x] **OPP-005** Add a friction benchmark classifying blocks as in-band / out-of-band / dead-end over packaged-CLI journeys. _Acceptance:_ see [OPP-005](../opportunities/OPP-005-friction-benchmark.md); deterministic classification, continuations actually executed, CI fails on any dead end.
|
|
121
121
|
- [x] **OPP-006** Adopt the emission ≠ usability test policy: messages naming a command must have a test executing it. _Acceptance:_ see [OPP-006](../opportunities/OPP-006-emission-usability-test-policy.md); extraction helper covers emitted and generated command mentions; generated `ai-init` skills' commands are executed in e2e tests.
|
|
122
|
-
- [ ] **007
|
|
123
|
-
- [ ] **008
|
|
122
|
+
- [ ] **OPP-007** Decision model — findings-evidence triage and lane scope-watch (advisory-only). _Acceptance:_ see [OPP-007](../opportunities/OPP-007-decision-findings-triage-scope-watch.md); unconfigured = byte-identical offline behavior; advisory flags never block, never lower tiers, never choose lanes.
|
|
123
|
+
- [ ] **OPP-008** Embedding retrieval — covering-test detection and judge context scoping. _Acceptance:_ see [OPP-008](../opportunities/OPP-008-embedding-covering-tests-judge-context.md); retrieval only prevents false `untested` flags (absence fails safe toward high); scoped judge prompts labeled as context aids within token ceilings.
|
|
124
|
+
- [x] **OPP-009** Processes as entry points — bug-fixing and idea-assessment process registries alongside the feature SDD lane. _Acceptance:_ see [OPP-009](../opportunities/OPP-009-process-entry-points.md); per-process stage lists, diagnosis/assessment artifacts with gates, default feature process byte-identical, archived ledgers backward compatible.
|
|
125
|
+
- [x] **OPP-010** Worktree isolation for orchestrated execution (`orchestrate --isolate`). _Acceptance:_ see [OPP-010](../opportunities/OPP-010-orchestration-worktree-isolation.md); per-task worktrees, evidence-bound checkpoints, capability-declared adapter refusal, cleanup guarded by doctor; no flag = unchanged behavior.
|
|
126
|
+
- [x] **OPP-011** Evals + red-team layer for generated artifacts. _Acceptance:_ see [OPP-011](../opportunities/OPP-011-evals-redteam-generated-artifacts.md); per-adapter contract invariants in CI, deterministic injection fixtures, offline local run, ratchet for skill/template changes.
|
|
127
|
+
- [x] **OPP-012** Standards discovery — extract codebase conventions into confirmed, versioned standards. _Acceptance:_ see [OPP-012](../opportunities/OPP-012-standards-discovery.md); evidence-cited extraction with confirmation contract, deterministic offline floor, just-in-time injection into rules/skills, advisory drift reporting.
|
|
128
|
+
- [x] **OPP-013** Read-Optimized Artifacts — fingerprint-bound digest layer so downstream stages read digests instead of re-reading full artifacts. _Acceptance:_ see [OPP-013](../opportunities/OPP-013-read-optimized-artifacts-digest-layer.md); digests extracted deterministically at approval, stale-safe (fail toward full read), ≥60% reduction in recorded full-artifact reads, no model-generated summaries.
|
|
129
|
+
- [x] **OPP-014** Paired Verification Contract — plan declares `command → expected`, evidence records `command → observed`, gates compare pairs. _Acceptance:_ see [OPP-014](../opportunities/OPP-014-paired-verification-contract.md); mismatch/missing pairs fail with typed reasons and machine-readable task ids; gates never execute commands; legacy format accepted with advisory.
|
|
130
|
+
- [x] **OPP-015** Token & Recovery Governance — enforced stage budgets, context-recovery managed block, advisory lane suggestion, regression-pinning declaration. _Acceptance:_ see [OPP-015](../opportunities/OPP-015-token-recovery-governance.md); budgets derive from saved baselines and default-inert; recovery block in every adapter rendering; advisory line never blocks.
|
|
131
|
+
- [ ] **OPP-016** Behavioral evaluation phase 2 — scenario registry replayed against real harness CLIs. _Acceptance:_ see [OPP-016](../opportunities/OPP-016-behavioral-eval-scenarios.md); deterministic state-outcome assertions, skipped-harness markers, advisory-first before gates, usage recorded per run.
|
|
132
|
+
- [ ] **OPP-017** Orchestration depth — integration forecast, doctor leak reporting, diff-stat evidence, per-harness honor matrix. _Acceptance:_ see [OPP-017](../opportunities/OPP-017-orchestration-depth.md); wave-ordered merge forecast with merge-tree dry run, typed cleanup continuation from doctor, no auto-merge.
|
|
133
|
+
- [x] **OPP-018** Supported migration path for legacy (pre-ownership) installations. _Acceptance:_ see [OPP-018](../opportunities/OPP-018-legacy-migration.md); dry-run lists archived/rebuilt/preserved/manual, transactional apply with rollback, fail-closed on ambiguity, doctor/install point at `aiw migrate`.
|
|
124
134
|
|
|
125
135
|
## Backlog rules
|
|
126
136
|
|
|
@@ -6,19 +6,24 @@ Every blocking `aiw` exit carries a stable reason code plus either an exact runn
|
|
|
6
6
|
- With `--json`, blocking exits return `{ "reason_code": ..., "next_command" | "next_action" | "terminal": ... }`.
|
|
7
7
|
- The table below is generated from the registry; a test in `src/blocking-reasons.test.ts` fails CI when this page drifts from the code.
|
|
8
8
|
|
|
9
|
-
| Reason code | Description
|
|
10
|
-
| ------------------------------- |
|
|
11
|
-
| `gate_evidence_missing` | The quality gate is blocked because the required evidence artifact is missing.
|
|
12
|
-
| `gate_incomplete` | The quality gate found completeness issues in the stage artifact.
|
|
13
|
-
| `stage_prerequisite_pending` | A previous stage is awaiting human approval before this stage can start.
|
|
14
|
-
| `stage_prerequisite_unresolved` | Previous stages need human approval or an accepted optional skip before this stage can start.
|
|
15
|
-
| `stage_awaiting_approval` | This stage is awaiting human approval; approve or reject it before continuing.
|
|
16
|
-
| `stage_already_approved` | This stage is already approved; changed evidence requires rerunning its quality gate.
|
|
17
|
-
| `approval_evidence_missing` | Approval cannot be recorded because the stage evidence artifact is missing.
|
|
18
|
-
| `approval_stale` | The artifact changed after review; its approval is stale and the gate must run again.
|
|
19
|
-
| `approval_gate_missing` | No passing quality gate is recorded for this stage; gate it before approving.
|
|
20
|
-
| `lane_escalation_required` | Light-lane workflows do not author stage artifacts; escalate to the full lane to continue.
|
|
21
|
-
| `light_evidence_missing` | A light-lane workflow cannot complete without at least one recorded validation entry.
|
|
22
|
-
| `workflow_still_active` | A workflow is already active; finish or archive it before starting another.
|
|
23
|
-
| `workflow_not_active` | No SDD workflow is active for this command to operate on.
|
|
24
|
-
| `workflow_complete` | This workflow is already complete; start a new workflow to continue working.
|
|
9
|
+
| Reason code | Description | Continuation |
|
|
10
|
+
| ------------------------------- | -------------------------------------------------------------------------------------------------------- | ---------------- |
|
|
11
|
+
| `gate_evidence_missing` | The quality gate is blocked because the required evidence artifact is missing. | explicit action |
|
|
12
|
+
| `gate_incomplete` | The quality gate found completeness issues in the stage artifact. | explicit action |
|
|
13
|
+
| `stage_prerequisite_pending` | A previous stage is awaiting human approval before this stage can start. | runnable command |
|
|
14
|
+
| `stage_prerequisite_unresolved` | Previous stages need human approval or an accepted optional skip before this stage can start. | runnable command |
|
|
15
|
+
| `stage_awaiting_approval` | This stage is awaiting human approval; approve or reject it before continuing. | runnable command |
|
|
16
|
+
| `stage_already_approved` | This stage is already approved; changed evidence requires rerunning its quality gate. | runnable command |
|
|
17
|
+
| `approval_evidence_missing` | Approval cannot be recorded because the stage evidence artifact is missing. | explicit action |
|
|
18
|
+
| `approval_stale` | The artifact changed after review; its approval is stale and the gate must run again. | runnable command |
|
|
19
|
+
| `approval_gate_missing` | No passing quality gate is recorded for this stage; gate it before approving. | runnable command |
|
|
20
|
+
| `lane_escalation_required` | Light-lane workflows do not author stage artifacts; escalate to the full lane to continue. | explicit action |
|
|
21
|
+
| `light_evidence_missing` | A light-lane workflow cannot complete without at least one recorded validation entry. | explicit action |
|
|
22
|
+
| `workflow_still_active` | A workflow is already active; finish or archive it before starting another. | explicit action |
|
|
23
|
+
| `workflow_not_active` | No SDD workflow is active for this command to operate on. | runnable command |
|
|
24
|
+
| `workflow_complete` | This workflow is already complete; start a new workflow to continue working. | runnable command |
|
|
25
|
+
| `installation_required` | AI Workflow is not installed in this repository; install it to continue. | runnable command |
|
|
26
|
+
| `state_mismatch` | AI Workflow state files are inconsistent with each other; run diagnostics to inspect and repair. | runnable command |
|
|
27
|
+
| `context_budget_exceeded` | The served context exceeds the declared ceiling for this stage; trim it or raise the ceiling explicitly. | explicit action |
|
|
28
|
+
| `stage_not_in_process` | This stage does not exist in the active workflow's process; use the process's own stages. | explicit action |
|
|
29
|
+
| `migration_required` | Legacy pre-ownership state detected; review the migration plan before continuing. | runnable command |
|
|
@@ -0,0 +1,56 @@
|
|
|
1
|
+
# OPP-009: Processes as entry points — bug-fixing and idea assessment alongside SDD
|
|
2
|
+
|
|
3
|
+
- Status: Implemented (2026-11-01): feature/bugfix/idea-assessment processes with per-process stage lists, gates, and typed stage_not_in_process blocks
|
|
4
|
+
- Priority: P2
|
|
5
|
+
- Added: 2026-11-01
|
|
6
|
+
- Decision: extends [ADR-001](../../.context/adrs/ADR-001-odd-rdd-proportionality-scope.md) Decision A (lane architecture)
|
|
7
|
+
- Trend signal: `github/spec-kit` (145K★) ships bug-fixing and idea-assessment as **independent entry points, not mandatory phases** — the market confirmed the proportionality direction ADR-001 adopted; this opportunity matches its breadth.
|
|
8
|
+
|
|
9
|
+
## Problem
|
|
10
|
+
|
|
11
|
+
AI Workflow's staged pipeline is feature-centric. Real work arrives in at least two other shapes that today are force-fitted into the full lane (generating unread artifacts) or bypass the workflow entirely (generating no evidence):
|
|
12
|
+
|
|
13
|
+
1. **Bug reports** — nobody brainstorms a bug. The honest process is: reproduce/diagnose → scoped fix → verification that the reproduction now passes and the original behavior holds.
|
|
14
|
+
2. **Idea evaluation** — before any spec exists, the question is "is this worth building?" — an evidence-backed go / clarify / stop decision, not an implementation plan.
|
|
15
|
+
|
|
16
|
+
spec-kit's model validates the shape: processes are independent entry points assembled from shared primitives, some bundled as extensions. AI Workflow's lanes, reason codes, gates, and approval ledger are exactly such primitives — this opportunity composes them into first-class processes.
|
|
17
|
+
|
|
18
|
+
## Proposal
|
|
19
|
+
|
|
20
|
+
1. **Process definitions** (data, not code paths): each process declares its ordered stages and optional stages, extending the lane concept (OPP-001) from two lanes of one process to a small process registry:
|
|
21
|
+
- `feature` (default, current 8 stages, unchanged);
|
|
22
|
+
- `bugfix`: `diagnosis` (reproduction evidence is the artifact — observed behavior vs expected, environment) → scoped fix implementation (TDD: the failing reproduction test is the RED test) → verification → review. Brainstorming/specification/technical-design do not exist in this process; requirements are the reproduction itself.
|
|
23
|
+
- `idea-assessment`: evidence gathering (read-only research with attributed sources) → assessment artifact with a single outcome (`go` / `clarify` / `stop`) + human decision; `go` may chain into a `feature` workflow carrying the evidence forward.
|
|
24
|
+
2. **CLI**: `aiw task start --ticket=X --process=bugfix` (feature stays the default; the process choice is human-made like the lane). Process names validate against the registry; `--process` composes with `--lane=light` where the process supports it (bugfix supports light; idea-assessment is inherently its own artifact shape).
|
|
25
|
+
3. **Ledger**: `ApprovalLedger` gains a `process` field (default `feature`); stage validation (`assertStageReady`, `getUnresolvedPrerequisites`) reads the process's stage list instead of the fixed `APPROVAL_STAGES`. Gates and fingerprint binding are reused as-is — every stage keeps the same "automated gate ≠ human approval" contract.
|
|
26
|
+
4. **Templates/skills**: `resources/templates/` gains diagnosis and assessment templates; a `bug-fixing` skill guides the diagnosis artifact (reproduction command, observed/expected, root cause with evidence, scope statement); an `idea-assessment` skill guides evidence-backed go/clarify/stop with attributed sources.
|
|
27
|
+
5. **Traceability**: `aiw trace` accepts diagnosis artifacts as evidence sources for bugfix workflows (requirement = the reproduction statement).
|
|
28
|
+
|
|
29
|
+
## Affected areas
|
|
30
|
+
|
|
31
|
+
- `src/types.ts` + `src/human-approval.ts` (process registry, per-process stage lists, ledger `process` field, validation per process)
|
|
32
|
+
- `src/workflow.ts` (`task start --process`, stage command routing per process, `workflow complete` per-process stage set)
|
|
33
|
+
- `src/quality-gates.ts` (diagnosis and assessment completeness validators)
|
|
34
|
+
- `src/traceability.ts` (diagnosis as evidence source)
|
|
35
|
+
- `resources/templates/`, `resources/skills/bug-fixing.md`, `resources/skills/idea-assessment.md`, `resources/package.yaml`
|
|
36
|
+
- `docs/architecture.md` (process registry section), `README.md`
|
|
37
|
+
|
|
38
|
+
## Acceptance criteria
|
|
39
|
+
|
|
40
|
+
- Given a bugfix task, when the workflow runs, then only diagnosis → implementation → verification → review stages exist; `aiw spec` blocks with a typed reason (`stage_not_in_process`); the diagnosis artifact requires a reproduction command and observed/expected behavior to pass its gate.
|
|
41
|
+
- Given an idea-assessment task, when it completes, then the assessment artifact records an outcome (`go`/`clarify`/`stop`) with attributed evidence, and a human decision on the outcome is required before the workflow closes; `go` records a forward link to the created feature workflow.
|
|
42
|
+
- Given the default no-`--process` invocation, when any stage command runs, then behavior is byte-identical to the feature process today.
|
|
43
|
+
- Given an archived ledger from ≤0.10.0, when 0.11+ reads it, then it is treated as `process: feature` (backward compatible, no migration).
|
|
44
|
+
- Given any process, when a stage blocks, then the blocking exit carries a reason code and exact continuation per the OPP-004 contract.
|
|
45
|
+
|
|
46
|
+
## Non-goals
|
|
47
|
+
|
|
48
|
+
- No user-defined custom processes in this iteration (the registry is code-defined; arbitrary composition is a later extension).
|
|
49
|
+
- No changes to the feature process's stage order, gates, or approvals.
|
|
50
|
+
- No automatic process inference from the ticket — the human picks; automation may suggest.
|
|
51
|
+
|
|
52
|
+
## Open questions
|
|
53
|
+
|
|
54
|
+
- Should `idea-assessment` outcomes feed recommendations (a `stop` recorded against a capability suppresses re-suggestion)?
|
|
55
|
+
- Does the bugfix process need a `regression-test` gate item distinct from verification evidence?
|
|
56
|
+
- Naming: `bugfix` vs `fix`; `idea-assessment` vs `assessment`.
|
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
# OPP-010: Worktree isolation for orchestrated execution
|
|
2
|
+
|
|
3
|
+
- Status: Implemented (2026-11-01): orchestrate --isolate with branch-per-task worktrees, evidence-bound checkpoints, kept-on-failure worktrees, and --cleanup
|
|
4
|
+
- Priority: P2
|
|
5
|
+
- Added: 2026-11-01
|
|
6
|
+
- Decision: extends [ADR-001](../../.context/adrs/ADR-001-odd-rdd-proportionality-scope.md) Decision A
|
|
7
|
+
- Trend signal: Spec Kitty sells isolated worktrees as its core ("implementation can happen in parallel without branch chaos"); Conductor (3.7K★) and Grasp are built entirely around parallel agent worktrees; AI Workflow's own `docs/quality.md` already mandates isolated-worktree validation runs — currently performed manually.
|
|
8
|
+
|
|
9
|
+
## Problem
|
|
10
|
+
|
|
11
|
+
`aiw orchestrate` schedules a validated task graph and delegates each task to an adapter, but every task runs against the **same working tree**. Consequences:
|
|
12
|
+
|
|
13
|
+
- parallel tasks conflict on shared files (the reason `--max-parallel` exists but parallelism stays risky);
|
|
14
|
+
- the quality.md requirement — "externally visible tasks also require an isolated integration run by an AI subagent in a separate worktree" — is satisfied by hand, with no evidence that the worktree was actually isolated;
|
|
15
|
+
- a failed task leaves the main tree dirty, mixing orchestration debris with human work.
|
|
16
|
+
|
|
17
|
+
## Proposal
|
|
18
|
+
|
|
19
|
+
1. **`aiw orchestrate --isolate`**: before executing a task (or an independent wave), create a git worktree from the current HEAD under `.aiw/worktrees/<plan-id>/<task-id>/`; pass that path as the task's working directory; remove the worktree on success (after evidence capture) and keep it on failure for diagnosis.
|
|
20
|
+
2. **Adapter contract extension** (backward compatible): `OrchestrationTask` gains an optional `worktree?: string`; the adapter contract keeps "the adapter owns process isolation" — aiw owns **git** isolation and hands the path over. Adapters that cannot use a worktree reject `--isolate` with a typed error naming the capability (consistent with the adapter capability matrix).
|
|
21
|
+
3. **Evidence binding**: the per-task checkpoint (already `orchestration-<plan-id>.yml`) records the worktree path, base commit, and final commit/diff stat, so the "isolated run happened" requirement becomes verifiable evidence instead of prose.
|
|
22
|
+
4. **Merge strategy**: on success, a fast-forward/merge step records which task branches are ready to integrate; merge remains a human-approved action (delivery is never automatic — consistent with the approval contract).
|
|
23
|
+
5. **Cleanup guard**: `aiw orchestrate --cleanup` lists and removes stale worktrees for completed/failed plans; doctor reports leaked worktrees (a worktree whose plan checkpoint shows completion but still exists on disk).
|
|
24
|
+
|
|
25
|
+
## Affected areas
|
|
26
|
+
|
|
27
|
+
- `src/orchestration.ts` (worktree lifecycle, evidence fields, cleanup)
|
|
28
|
+
- `src/adapter-contract.ts` (optional `worktree` task field, capability declaration)
|
|
29
|
+
- Adapters (`src/claude-adapter.ts`, codex, gemini, copilot, universal) — pass-through or capability-declared refusal
|
|
30
|
+
- `src/doctor`/`src/workflow.ts` diagnostics (leaked worktree check)
|
|
31
|
+
- `docs/orchestration.md`, `docs/quality.md` (manual requirement → tooling requirement)
|
|
32
|
+
|
|
33
|
+
## Acceptance criteria
|
|
34
|
+
|
|
35
|
+
- Given `--isolate` with two independent tasks, when the plan executes with `--max-parallel=2`, then each task runs in its own worktree from the same base commit and their file writes never touch the main tree.
|
|
36
|
+
- Given a task failure under isolation, when the plan finishes, then the failed task's worktree is preserved on disk and the checkpoint names its path and base commit.
|
|
37
|
+
- Given a successful task under isolation, when evidence is recorded, then the checkpoint contains base commit, final commit, and diff stat for that task.
|
|
38
|
+
- Given an adapter that does not declare worktree support, when `--isolate` is used, then the plan fails before any task runs with a typed error naming the adapter and capability.
|
|
39
|
+
- Given no `--isolate` flag, when a plan executes, then behavior is byte-identical to today.
|
|
40
|
+
- Given leaked worktrees, when `aiw doctor` runs, then they are reported with plan/task attribution; `--cleanup` removes them after explicit confirmation.
|
|
41
|
+
|
|
42
|
+
## Non-goals
|
|
43
|
+
|
|
44
|
+
- No automatic merge/push of task results — integration stays human-approved.
|
|
45
|
+
- No cross-repository worktrees; isolation is within the current repository.
|
|
46
|
+
- No rebase/-conflict resolution automation: conflicting task branches surface for human merge.
|
|
47
|
+
|
|
48
|
+
## Open questions
|
|
49
|
+
|
|
50
|
+
- Branch-per-task inside the worktree (easier merge later) vs detached HEAD with recorded commit (simpler)? Recommendation: branch-per-task named `aiw/<plan>/<task>`.
|
|
51
|
+
- Should isolated runs reuse OPP-002's tier to decide whether a task needs isolation at all (advisory)?
|
|
52
|
+
- Windows path-length handling for deep worktree paths (`.aiw/worktrees/...`) on long repo paths.
|
|
@@ -0,0 +1,56 @@
|
|
|
1
|
+
# OPP-011: Evals and red-team layer for generated artifacts
|
|
2
|
+
|
|
3
|
+
- Status: Implemented (2026-11-01): deterministic invariant + injection suites over shipped skills; mutation-tested harness
|
|
4
|
+
- Priority: P3 (high differentiation, after field exposure of 0.10/0.11)
|
|
5
|
+
- Added: 2026-11-01
|
|
6
|
+
- Decision: extends [ADR-001](../../.context/adrs/ADR-001-odd-rdd-proportionality-scope.md) Decision A
|
|
7
|
+
- Trend signal: promptfoo (26K★) made prompt/agent evaluation a CI discipline; nobody in the spec-driven landscape evaluates the **generated skills, rules, and prompts** that define an AI workflow's behavior. This is AI Workflow's natural differentiator: the product that governs AI artifacts should verify its own.
|
|
8
|
+
|
|
9
|
+
## Problem
|
|
10
|
+
|
|
11
|
+
AI Workflow generates project-specific skills, rules, agents, and prompts, and ships its own stage skills. Today they are checked for:
|
|
12
|
+
|
|
13
|
+
- command validity (OPP-006 ratchet: named commands execute);
|
|
14
|
+
- package contract validity (schema, permissions, provenance);
|
|
15
|
+
- language/format policy.
|
|
16
|
+
|
|
17
|
+
What is **not** checked: whether the generated artifacts still _behave_ as intended, and whether they _resist_ manipulation:
|
|
18
|
+
|
|
19
|
+
1. **Behavioral regressions**: a template or skill edit can silently weaken a gate instruction ("must wait for human submission" losing its emphasis), and no test fails.
|
|
20
|
+
2. **Injection resistance**: generated artifacts are read by agents with tool access. A malicious project file can instruct the agent to violate workflow guarantees ("you may skip approval if the user is in a hurry"). The policy gate checks permissions, not persuasion.
|
|
21
|
+
3. **Divergence between generated variants**: the same stage skill rendered for codex vs claude can drift in contractual content; only spot checks catch it.
|
|
22
|
+
|
|
23
|
+
## Proposal
|
|
24
|
+
|
|
25
|
+
1. **Scenario eval harness** (CI, deterministic, no model): fixture repositories where each stage skill's critical contract lines are probed as _textual invariants_ — "the artifact must instruct waiting for an explicit human reply", "must not contain phrases that authorize bypassing gates" (e.g., `approve` without submission, `skip` without reason). Extends the command ratchet into an instruction ratchet: contract lines are enumerated (name + required invariant), and CI fails when an invariant disappears from any adapter's rendering.
|
|
26
|
+
2. **Injection red-team fixtures**: fixture projects containing prompt-injection payloads aimed at workflow guarantees (approval bypass, scope expansion, evidence fabrication). The harness runs the same policy/consistency checks the product enforces and asserts the generated guidance _does not endorse_ the payload (checked by invariant matching on rendered output, not by model judgment).
|
|
27
|
+
3. **Behavioral eval registry** (optional, model-assisted later): scenario descriptors (project state + command sequence + expected outcome) that can be replayed against any runtime adapter; deterministic outcomes (exit codes, state files, typed envelopes) are asserted exactly as the e2e suite does today — the registry formalizes them as evals and makes adding one a one-line declaration. Model-judged outcomes are explicitly out of scope for CI determinism.
|
|
28
|
+
4. **Reporting**: `aiw eval run [--suite=core|skills|injection]` reuses the harness locally; CI runs `core` + `injection` on every PR touching `resources/` or templates.
|
|
29
|
+
|
|
30
|
+
## Affected areas
|
|
31
|
+
|
|
32
|
+
- New `src/eval/` (scenario registry, invariant extraction from skill markdown, harness)
|
|
33
|
+
- `resources/skills/*.md`, `resources/templates/*` (invariant annotations: stable anchor comments the harness counts on)
|
|
34
|
+
- `.github/workflows/ci.yml` (eval suites on resource-touching PRs)
|
|
35
|
+
- `src/package-audit.ts` / `src/policy-gate.ts` (reuse permission and policy surfaces for injection checks)
|
|
36
|
+
- `docs/quality.md` (eval suites as completion criteria for skill/template changes)
|
|
37
|
+
|
|
38
|
+
## Acceptance criteria
|
|
39
|
+
|
|
40
|
+
- Given a stage skill rendered for every adapter, when invariants run, then each declared contract invariant (approval waiting, skip reasons, gate-before-approve) is present in every rendering; a missing one fails CI naming the adapter and invariant.
|
|
41
|
+
- Given an injection fixture (approval-bypass payload in a project file), when the eval suite runs, then no generated artifact endorses the bypass (invariant match), and the run is deterministic across executions.
|
|
42
|
+
- Given a template edit that removes a gate instruction, when CI runs, then the behavioral invariant fails naming the removed anchor.
|
|
43
|
+
- Given `aiw eval run --suite=skills`, when run locally, then it reproduces the CI result offline with no model configured.
|
|
44
|
+
- Given a new skill/template PR, when it touches rendered contract content, then the PR must include invariant updates (ratchet like OPP-006).
|
|
45
|
+
|
|
46
|
+
## Non-goals
|
|
47
|
+
|
|
48
|
+
- No LLM-as-judge in CI (determinism is the whole point; model-assisted evaluation stays a local, advisory tool).
|
|
49
|
+
- No red-teaming of end-user code — the target is AI Workflow's own generated artifacts.
|
|
50
|
+
- No runtime enforcement changes: this hardens what ships, it does not add new gates to user workflows.
|
|
51
|
+
|
|
52
|
+
## Open questions
|
|
53
|
+
|
|
54
|
+
- Invariant anchor format: HTML comments in skill markdown vs a sidecar manifest mapping invariants to files?
|
|
55
|
+
- Should injection fixtures live in the package contract tests (per third-party package) or the core harness?
|
|
56
|
+
- Do rendered-variant diffs (codex vs claude) warrant a dedicated divergence check beyond shared invariants?
|
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
# OPP-012: Standards discovery — extract codebase conventions into versioned standards
|
|
2
|
+
|
|
3
|
+
- Status: Implemented (2026-11-01, deterministic floor): aiw standards extract|list|confirm|reject with evidence-cited inferred standards; OPP-008 retrieval extension remains future
|
|
4
|
+
- Priority: P3 (depends on OPP-008 retrieval plumbing)
|
|
5
|
+
- Added: 2026-11-01
|
|
6
|
+
- Decision: extends [ADR-001](../../.context/adrs/ADR-001-odd-rdd-proportionality-scope.md) Decision A
|
|
7
|
+
- Trend signal: Agent OS (5.5K★) built its positioning on "Discover Standards — extract patterns and conventions from your codebase into documented standards" and "Deploy Standards — inject relevant standards based on what you're building". AI Workflow generates rules from the detected **stack**; the trend is to extract them from the **codebase's actual conventions**, with evidence.
|
|
8
|
+
|
|
9
|
+
## Problem
|
|
10
|
+
|
|
11
|
+
AI Workflow generates project rules from detected and confirmed facts (stack, tools, scripts). What it does not capture: **how this codebase actually does things** — error-handling idiom, naming conventions, test structure, state management patterns. Consequences:
|
|
12
|
+
|
|
13
|
+
- generated guidance is generic ("this is a TypeScript project") rather than specific ("this repo wraps errors in `AppError` with `code` fields; tests live beside sources as `*.test.ts` using vitest `describe/it`");
|
|
14
|
+
- agents routinely propose idiomatic-but-foreign code that reviewers must correct;
|
|
15
|
+
- conventions live in senior developers' heads — exactly the knowledge agents lack.
|
|
16
|
+
|
|
17
|
+
## Proposal
|
|
18
|
+
|
|
19
|
+
1. **`aiw standards extract`** (read-only, evidence-linked): analyzes the repository (deterministic detectors first: file layout, test naming, config presence; semantic retrieval via OPP-008 when enabled for pattern instances) and produces **standards documents** under `.aiw/standards/<topic>.md` — each statement carries: the convention, **evidence citations** (file:line examples, N occurrences), and a confidence tier (detected-strong / detected-weak / proposed).
|
|
20
|
+
2. **Confirmation contract** (consistent with profile facts): standards are `inferred` until the human accepts/rejects/edits them via `aiw confirm`; accepted standards become `confirmed` and are versioned in the lockfile. AI output never silently becomes rule content.
|
|
21
|
+
3. **Injection into generated rules**: confirmed standards are rendered into project rules (and referenced from stage skills: "follow `.aiw/standards/error-handling.md`") through the existing generator — the same preserved-override/drift semantics as other generated content (GEN-002).
|
|
22
|
+
4. **Deterministic floor**: with no semantic mapping configured, extraction covers deterministic conventions only (naming patterns via glob/AST-lite heuristics, test layout, error-class presence) — no network, useful even offline. Retrieval only extends coverage (OPP-008's consumer, as anticipated).
|
|
23
|
+
5. **Freshness**: standards carry their extraction commit; `aiw scan` reports drift when the codebase diverges from recorded standards (files matching a standard's evidence changed) — advisory, consistent with drift detection elsewhere.
|
|
24
|
+
|
|
25
|
+
## Affected areas
|
|
26
|
+
|
|
27
|
+
- New `src/standards.ts` (extractors, confirmation state, rendering)
|
|
28
|
+
- `src/scanner.ts` (deterministic convention detectors), `src/semantic.ts` (OPP-008 retrieval for pattern instances)
|
|
29
|
+
- `src/generator.ts` (rule injection referencing confirmed standards), `src/drift.ts` (standards drift reporting)
|
|
30
|
+
- `.aiw/standards/` (new state directory), lockfile (confirmed standards)
|
|
31
|
+
- `docs/README.md`, `docs/architecture.md`, README
|
|
32
|
+
|
|
33
|
+
## Acceptance criteria
|
|
34
|
+
|
|
35
|
+
- Given a repository with consistent error-wrapping, when `aiw standards extract` runs offline, then the error-handling standard cites ≥2 file:line examples with occurrence counts and tier `detected-strong`, pending confirmation.
|
|
36
|
+
- Given inferred standards, when the human confirms one, then it becomes `confirmed`, is recorded in the lockfile, and appears in regenerated rules; rejected ones are never rendered.
|
|
37
|
+
- Given no semantic mapping configured, when extracting, then deterministic conventions still produce standards and no network request is made.
|
|
38
|
+
- Given confirmed standards, when a stage skill renders, then it references applicable standards by path (just-in-time, per the token-cost principle — standards content is loaded on demand, not inlined everywhere).
|
|
39
|
+
- Given a codebase change contradicting a confirmed standard, when `aiw scan` runs, then drift is reported as advisory with the diverging files named.
|
|
40
|
+
|
|
41
|
+
## Non-goals
|
|
42
|
+
|
|
43
|
+
- No auto-confirmation: all standards start inferred (profile-facts contract).
|
|
44
|
+
- No style-lint replacement: standards describe conventions; enforcement stays with the project's own linters.
|
|
45
|
+
- No cross-repository standards sharing in this iteration (org-level standards live in ENT-001/002 territory).
|
|
46
|
+
|
|
47
|
+
## Open questions
|
|
48
|
+
|
|
49
|
+
- Extractor taxonomy for v1: error handling, testing, naming, module layout — what else earns a dedicated extractor?
|
|
50
|
+
- Should standards participate in traceability (a rule citing a standard citing files)?
|
|
51
|
+
- Granularity: one document per topic vs one per convention-cluster (affects drift reporting precision)?
|
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
# OPP-013: Read-Optimized Artifacts — the Digest Layer
|
|
2
|
+
|
|
3
|
+
- Status: Implemented (2026-11-01): fingerprint-bound digests computed at approval; aiw digest and context --stage serve them stale-safe
|
|
4
|
+
- Priority: P1 (largest measurable token lever in the product)
|
|
5
|
+
- Added: 2026-11-01
|
|
6
|
+
- Decision: extends [ADR-001](../../.context/adrs/ADR-001-odd-rdd-proportionality-scope.md) Decision A
|
|
7
|
+
- Measured baseline (this repository's own artifacts): a full-lane feature whose specification is ~2.2K tokens and plan ~4.1K tokens pays **~25-30K tokens in downstream re-reads alone** — the specification is re-read ~6-7× (plan, implementation, verification, review, traceability) and the plan ~4×. Writing artifacts is ~20% of their lifetime token cost; re-reading them is ~80%.
|
|
8
|
+
|
|
9
|
+
## Problem
|
|
10
|
+
|
|
11
|
+
Every artifact in the staged workflow is written once and read many times. The single dominant token cost of a full-lane feature is not authoring — it is downstream stages re-ingesting full artifacts to extract the handful of facts they need:
|
|
12
|
+
|
|
13
|
+
- `plan` needs the spec's requirement IDs and acceptance criteria — re-reads everything;
|
|
14
|
+
- `implementation` needs its task's linked requirements — re-reads the spec and the whole plan per task;
|
|
15
|
+
- `verification`/`review`/`traceability` need requirement→evidence mappings — re-read both.
|
|
16
|
+
|
|
17
|
+
Three methodologies shrink artifacts (ODD: fewer; HDAE: smaller; spec-kit: better templates). None attack the multiplier: **write once, read N×**. AI Workflow can be the first to formalize the separation: **full artifacts are for humans; agents get a derived, verifiable view**.
|
|
18
|
+
|
|
19
|
+
## Proposal
|
|
20
|
+
|
|
21
|
+
1. **Digest at the approval boundary**: when a stage passes its gate and receives human approval, aiw computes a structured **digest** from the exact approved bytes and stores it fingerprint-bound: `goal`, `scope` (one line), `requirements: [{id, one-line, criteria-ids}]`, `non-goals`, `decisions-open`, `links` (artifacts and commits). Deterministic extraction from required sections (the gate already enforces their presence) — **never a model-generated summary**.
|
|
22
|
+
2. **Downstream reads target digests by default**: `aiw context --stage=<stage>` (CTX-002's delta retrieval) returns the manifest of digests plus only the artifact sections a stage's manifest declares; full reads remain available (`--full`, or task-linked requirement sections by ID anchor).
|
|
23
|
+
3. **Digest integrity**: the digest records the artifact's SHA-256 fingerprint (already bound in `.aiw/approvals.yml`); a stale digest (artifact changed without re-gating) is reported as stale, never silently served — same semantics as approval staleness.
|
|
24
|
+
4. **Template-declared fields**: digests extract from the sections templates already mandate (`## Requirements`, `## Acceptance Criteria`, `## Tasks`, `## Validation`), so extraction is parse-not-guess; templates may declare additional digest fields per process (OPP-009).
|
|
25
|
+
5. **Read-path reporting**: `aiw token-usage` gains a digest row (served-digest vs full-read counts) so the saving is measured, not assumed.
|
|
26
|
+
|
|
27
|
+
## Affected areas
|
|
28
|
+
|
|
29
|
+
- `src/context-retrieval.ts` / `src/context-cache.ts` (digest store + default-read routing)
|
|
30
|
+
- `src/quality-gates.ts` (digest extraction validators per artifact type)
|
|
31
|
+
- `src/workflow.ts` (compute at approve; serve in stage flows), `src/token-usage.ts` (read-path accounting)
|
|
32
|
+
- `resources/templates/*` (digest field declarations), `docs/architecture.md`, `docs/token-efficiency.md`
|
|
33
|
+
|
|
34
|
+
## Acceptance criteria
|
|
35
|
+
|
|
36
|
+
- Given an approved specification, when a downstream stage requests context, then it receives the digest plus only declared sections; a full read happens only with an explicit flag or a linked-section reference.
|
|
37
|
+
- Given a digest, when the artifact's fingerprint no longer matches the approval ledger, then the digest is reported stale and the full artifact is required (fail-safe toward fidelity).
|
|
38
|
+
- Given the digest extraction, when the artifact lacks a mandated section, then the gate fails before approval (digests only exist for gate-passing artifacts).
|
|
39
|
+
- Given no flag changes, when a light-lane workflow runs, then behavior is unchanged (light tasks produce no durable artifacts to digest).
|
|
40
|
+
- Given a full-lane feature at the measured baseline, when downstream stages complete, then recorded full-artifact reads drop by at least 60% (served-digest accounting in `token-usage`).
|
|
41
|
+
|
|
42
|
+
## Non-goals
|
|
43
|
+
|
|
44
|
+
- No model-generated summaries, ever (a stochastic digest would void the fingerprint guarantee).
|
|
45
|
+
- No digest replacing the artifact for humans; no digest content beyond extraction from mandated sections.
|
|
46
|
+
- No change to gate or approval semantics — digests exist only downstream of approval.
|
|
47
|
+
|
|
48
|
+
## Open questions
|
|
49
|
+
|
|
50
|
+
- Digest storage: sidecar files (`.aiw/digests/<artifact>.yml`) vs front-matter appended to the artifact itself (one file, but touches approved bytes — likely sidecar, to keep approved artifacts immutable)?
|
|
51
|
+
- Anchor format for per-requirement reads (headings are stable; `REQ-\d+` IDs are already mandated).
|
|
52
|
+
- Do plan digests include per-task validation commands (OPP-014 pairs), or do those stay task-scoped?
|
|
@@ -0,0 +1,54 @@
|
|
|
1
|
+
# OPP-014: Paired Verification Contract — expected vs observed, mechanically compared
|
|
2
|
+
|
|
3
|
+
- Status: Implemented (2026-11-01): plan declares command → expected, reports record command → observed, implementation/verification gates compare pairs
|
|
4
|
+
- Priority: P2
|
|
5
|
+
- Added: 2026-11-01
|
|
6
|
+
- Decision: extends [ADR-001](../../.context/adrs/ADR-001-odd-rdd-proportionality-scope.md) Decision A
|
|
7
|
+
- Trend signal: HDAE's gate structure pairs **Verification plan** (`command → expected result`) with **Evidence** (`command → observed result`) as first-class fields; Gentle-AI's field report showed the failure mode this prevents — agents declaring success without observed proof. AI Workflow's plan template already carries `Validation:` lines and its quality gates verify section _presence_; this contract makes the proof **mechanically comparable**.
|
|
8
|
+
|
|
9
|
+
## Problem
|
|
10
|
+
|
|
11
|
+
"Checkoffs require observed proof" is today an instruction, not a mechanism:
|
|
12
|
+
|
|
13
|
+
- the plan declares validation commands (`Validation: npm test`), but the gate cannot tell whether the recorded evidence actually ran _that_ command;
|
|
14
|
+
- expected outcomes are implicit ("tests pass") — nothing records what result was _expected_ before the run, so a wrong expectation passes as easily as a right one;
|
|
15
|
+
- evidence sections are free text — a gate can verify presence, never correspondence.
|
|
16
|
+
|
|
17
|
+
A gate that cannot compare declared vs observed is checking paperwork, not proof.
|
|
18
|
+
|
|
19
|
+
## Proposal
|
|
20
|
+
|
|
21
|
+
1. **Plan side (declaration)**: each task in the implementation plan declares its verification pair, machine-parseable:
|
|
22
|
+
- `Validation: <command> → <expected>` where expected is a minimal assertion (exit code plus optional output pattern), e.g. `npm test → exit 0`, `aiw trace --requirement=REQ-001 → contains REQ-001`.
|
|
23
|
+
2. **Evidence side (observation)**: implementation/verification artifacts record the same command with its observed result: `<command> → <observed>` (exit code and the claimed observable, e.g. `132 passed`).
|
|
24
|
+
3. **Gate comparison** (deterministic, parse-only): the verification and implementation gates check, per declared pair: the command appears verbatim in evidence, the observed section is non-empty, and expected/observed exit codes agree. Mismatch, absence, or a evidence command that was never declared → typed gate failure naming the task and pair (`verification_pair_mismatch`, `verification_pair_missing` — new reason codes, inventory updated).
|
|
25
|
+
4. **Light lane**: one pair per task is the whole evidence contract (the compact checkpoint already records command and result — formalize the same shape).
|
|
26
|
+
5. **No execution**: the gate compares declarations; it does not run project commands (preserving `verify`'s contract and keeping gates fast and side-effect-free). Execution evidence must come from a recorded run.
|
|
27
|
+
|
|
28
|
+
## Affected areas
|
|
29
|
+
|
|
30
|
+
- `resources/templates/` (plan task format gains the pair; implementation/verification report templates gain the observed side)
|
|
31
|
+
- `src/quality-gates.ts` (pair parser + comparator validators; new gate failures)
|
|
32
|
+
- `src/blocking-reasons.ts` (`verification_pair_mismatch`, `verification_pair_missing` + inventory rows)
|
|
33
|
+
- `src/workflow.ts` (gate wiring), `docs/architecture.md`, `docs/quality.md` (contract statement)
|
|
34
|
+
|
|
35
|
+
## Acceptance criteria
|
|
36
|
+
|
|
37
|
+
- Given a plan declaring `npm test → exit 0`, when evidence records `npm test → exit 0 (132 passed)`, then the gate passes.
|
|
38
|
+
- Given evidence recording a different command than declared, when the gate runs, then it fails with `verification_pair_mismatch` naming the task.
|
|
39
|
+
- Given a declared pair with no matching observed record, when the gate runs, then it fails with `verification_pair_missing`.
|
|
40
|
+
- Given artifacts authored before this contract (no pairs), when gates run, then the legacy validation format remains accepted with an advisory notice (backward-compatible parsing, no forced migration).
|
|
41
|
+
- Given a light-lane evidence entry, when completion runs, then the pair shape is the compact checkpoint's command/result fields.
|
|
42
|
+
- Given any pair failure, when `--json` is used, then the typed reason code and the failing task id are machine-readable.
|
|
43
|
+
|
|
44
|
+
## Non-goals
|
|
45
|
+
|
|
46
|
+
- The gate never executes project commands (side-effect-free gates; execution evidence comes from recorded runs).
|
|
47
|
+
- No expected/observed semantic comparison beyond exit codes and declared patterns (no model judgment in gates).
|
|
48
|
+
- No retroactive migration of existing artifacts (ratchet: new plans use pairs; legacy format accepted with notice).
|
|
49
|
+
|
|
50
|
+
## Open questions
|
|
51
|
+
|
|
52
|
+
- Expected-result grammar: exit code only, or `exit N` + substring/regex pattern? (Recommendation: exit code required, pattern optional.)
|
|
53
|
+
- Should `verify` (REQ→evidence coverage) surface pairs per requirement in the verification report?
|
|
54
|
+
- Do orchestration-delegated tasks (OPP-010) inherit pairs via their task prompt or their own declaration?
|
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
# OPP-015: Token and Recovery Governance — enforced budgets, recovery block, advisory suggestions
|
|
2
|
+
|
|
3
|
+
- Status: Implemented (2026-11-01): enforced context budgets, recovery managed block, advisory lane suggestion, regression-pinning advisory
|
|
4
|
+
- Priority: P2 (small, independent mechanisms; bundle ships together)
|
|
5
|
+
- Added: 2026-11-01
|
|
6
|
+
- Decision: extends [ADR-001](../../.context/adrs/ADR-001-odd-rdd-proportionality-scope.md) Decision A
|
|
7
|
+
- Sources: HDAE's context-recovery managed block and "do not fill gaps with assumptions" interview discipline; ODD's advisory lane guidance; AI Workflow's own CTX-004 (per-stage budgets, currently report-only) and OPP-002 (`risk assess`, already shipped).
|
|
8
|
+
|
|
9
|
+
## Problem
|
|
10
|
+
|
|
11
|
+
Four small gaps share one theme — the workflow knows the right thing but does not _act_ on it:
|
|
12
|
+
|
|
13
|
+
1. **Stage budgets are advisory**: CTX-004 measures per-stage token usage, but exceeding a budget changes nothing — there is no enforced ceiling, so drift accumulates silently.
|
|
14
|
+
2. **Post-compaction recovery is left to the agent's memory**: after context compaction or handoff, the agent re-derives state from chat remnants instead of reading the authoritative files — expensive and lossy.
|
|
15
|
+
3. **Lane choice is uninformed**: `task start` does not surface what `risk assess` already knows; humans pick lanes blind when a one-line advisory would do.
|
|
16
|
+
4. **Regression pinning is folklore**: "touched behavior keeps a test proving its previous behavior" exists in docs/quality culture but not as a declared, checkable artifact item.
|
|
17
|
+
|
|
18
|
+
## Proposal
|
|
19
|
+
|
|
20
|
+
1. **Enforced stage budgets**: stage/context manifests (per process and lane, OPP-009-ready) declare ceilings; exceeding one fails with a typed block (`context_budget_exceeded`) whose continuation names what to trim or the `--budget=<n>` override requiring an explicit reason recorded in the checkpoint. Budgets default from measured baselines (`context-bench --save`), never invented.
|
|
21
|
+
2. **Context-recovery managed block**: the root context file gains an aiw-managed block (same preserve/drift semantics as other managed content): _"After any context compaction, reset, resume, or handoff: run `aiw task status`, read the artifacts this workflow declares, and do not rely on compacted chat as the source of project state."_ Points at digests when OPP-013 exists (declared forward reference).
|
|
22
|
+
3. **Advisory lane suggestion**: `task start --lane=light` (and plain `task start`) prints, when `aiw risk assess` data is available for the current changes, one advisory line — `risk: high -> consider the full lane` or `risk: passive -> light lane viable` — always advisory; the human's choice is recorded as today. No new flags, no blocking.
|
|
23
|
+
4. **Regression-pinning declaration**: the plan template's task format gains an optional `Regression: <behavior> → <test>` line; `verify`/traceability report tasks that touched existing behavior (declared `Touches:` existing paths) without a regression declaration, as advisory output in the verification report and a documentation rule in `docs/quality.md`. Optional to declare, reported when absent — declaration, not bureaucracy.
|
|
24
|
+
|
|
25
|
+
## Affected areas
|
|
26
|
+
|
|
27
|
+
- `src/workflow.ts` (budget enforcement points, lane suggestion line), `src/context-quality.ts` / CTX-004 ceilings
|
|
28
|
+
- `src/human-approval.ts` (budget override reason recorded in checkpoints)
|
|
29
|
+
- `src/generator.ts` / `resources/templates/*` (recovery block, task `Regression:` line)
|
|
30
|
+
- `src/blocking-reasons.ts` (`context_budget_exceeded` + inventory row)
|
|
31
|
+
- `docs/quality.md`, `docs/token-efficiency.md`, `docs/architecture.md`
|
|
32
|
+
|
|
33
|
+
## Acceptance criteria
|
|
34
|
+
|
|
35
|
+
- Given a stage context exceeding its declared ceiling, when the stage runs, then it blocks with `context_budget_exceeded` and the continuation names trim options or the explicit override; the override is recorded with its reason in the checkpoint.
|
|
36
|
+
- Given `context-bench --save` baselines, when budgets are absent, then defaults derive from them (never arbitrary constants).
|
|
37
|
+
- Given the recovery block rendered for each target, when an agent resumes after compaction, then the block instructs reading `aiw task status` and workflow-declared artifacts before continuing (block present in all adapter renderings; preserved on user edits per drift rules).
|
|
38
|
+
- Given pending changes classified `high` by risk assess, when `task start --lane=light` runs, then the advisory line appears and the lane still starts as chosen (advisory never blocks).
|
|
39
|
+
- Given a task declaring `Touches: src/auth/token.ts` without a `Regression:` line, when verification runs, then the report lists the gap as advisory; declaring one removes it.
|
|
40
|
+
- Given the `npm`-published package on any target, when installed fresh, then behavior without baselines/budgets configured is byte-identical to today (all four mechanisms default-inert until declared).
|
|
41
|
+
|
|
42
|
+
## Non-goals
|
|
43
|
+
|
|
44
|
+
- No hard token ceilings on artifact authoring (authors write what the work needs; budgets govern _context served_, not writing).
|
|
45
|
+
- No automatic lane switching, budget auto-tuning, or model-driven anything (deterministic floor per spec NFR-2/3).
|
|
46
|
+
- No compaction simulator: the recovery block is an instruction artifact; testing it is OPP-011 invariant territory.
|
|
47
|
+
|
|
48
|
+
## Open questions
|
|
49
|
+
|
|
50
|
+
- Default budget derivation: per-stage from `context-bench` baseline, or per-workflow pooled?
|
|
51
|
+
- Should the lane suggestion also appear in `aiw recommend` output for ticket-driven starts?
|
|
52
|
+
- Does the `Regression:` line belong in the plan task format (OPP-014 pairs live there too) or the implementation report?
|
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
# OPP-016: Behavioral evaluation phase 2 — scenario registry replayed against real harness CLIs
|
|
2
|
+
|
|
3
|
+
- Status: Proposed
|
|
4
|
+
- Priority: P2
|
|
5
|
+
- Added: 2026-11-01
|
|
6
|
+
- Decision: extends [ADR-001](../../.context/adrs/ADR-001-odd-rdd-proportionality-scope.md) Decision A
|
|
7
|
+
- Source signal: external qualitative assessment scored "Behavioral evaluation 3/5 — deterministic evals; lacks real behavioral validation". Agreed: OPP-011 proves the skills **say** the right things; nothing yet proves agents **do** them.
|
|
8
|
+
|
|
9
|
+
## Problem
|
|
10
|
+
|
|
11
|
+
The shipped stage skills carry contract invariants (wait for explicit human reply, gate before approve, reasons required), and OPP-011's harness checks those invariants as text. What remains unvalidated is the behavior that emerges when a real coding agent reads them in a real repository: does the agent actually stop at the approval boundary, or does it proceed? Does a light-lane task really skip durable artifacts? Text invariants cannot answer behavior questions — only replaying scenarios against real harness CLIs can.
|
|
12
|
+
|
|
13
|
+
## Proposal
|
|
14
|
+
|
|
15
|
+
1. **Scenario registry**: declarative descriptors (YAML) per scenario — fixture repository state (files, git history), the harness command to run (`claude -p "…"` with `--permission-mode` bounded, `codex exec …`, `gemini -p …`), a bounded prompt driving one workflow action, and the **deterministic expected outcomes**: exit code, `.aiw/` state files (workflow.yml fields, approvals ledger statuses, checkpoint existence), and typed output envelopes (reason codes, digests). No prose assertions — state asserts only.
|
|
16
|
+
2. **Runner**: `aiw eval run --scenarios` executes each scenario in an isolated fixture copy, enforces wall-clock and token-usage ceilings per scenario, and asserts state outcomes exactly as the e2e suite does today. Missing harness CLI → the scenario is **skipped with an explicit marker** in results (never a silent pass). Retry policy: one retry on harness transport failure; a retry-passing scenario is reported as `flaky-suspect`, not green.
|
|
17
|
+
3. **CI shape**: a bounded smoke set (one scenario per stage family) runs on PRs touching `resources/` or templates; the full registry runs nightly. Both are advisory-first (report-only) for two release cycles before becoming gates — behavioral flakiness must be characterized before it can block.
|
|
18
|
+
4. **Determinism floor unchanged**: OPP-011 textual invariants and OPP-006's command ratchet remain the fast deterministic layer; scenarios are the behavioral layer on top.
|
|
19
|
+
|
|
20
|
+
## Affected areas
|
|
21
|
+
|
|
22
|
+
- New `src/eval/scenarios.ts` (registry parser, runner, result schema `aiw.scenario-results/v1`)
|
|
23
|
+
- `scenarios/*.yml` (initial registry: approval boundary, light-lane skip, escalation, bugfix process, rejection → re-gate)
|
|
24
|
+
- `.github/workflows/` (nightly behavioral job; advisory mode first)
|
|
25
|
+
- `docs/quality.md`, OPP-011 cross-link
|
|
26
|
+
|
|
27
|
+
## Acceptance criteria
|
|
28
|
+
|
|
29
|
+
- Given a scenario run twice on unchanged inputs, when results compare, then outcomes are identical (determinism), with token counts recorded per run via the usage sidecar.
|
|
30
|
+
- Given a harness CLI missing from the machine, when its scenarios are evaluated, then results mark them `skipped-harness-unavailable` and CI does not count them as passing.
|
|
31
|
+
- Given a fixture where the agent must stop at the approval boundary, when the scenario runs, then `.aiw/approvals.yml` shows `pending-human` and no later stage is started; an agent that proceeded fails the scenario.
|
|
32
|
+
- Given two consecutive release cycles, when the advisory period elapses, then at least the approval-boundary scenarios become CI gates (ratchet declared up front).
|
|
33
|
+
- Given no harness installed at all, when `aiw eval run --scenarios` runs, then it exits 0 with every scenario marked skipped and a summary naming what is missing.
|
|
34
|
+
|
|
35
|
+
## Non-goals
|
|
36
|
+
|
|
37
|
+
- No LLM-as-judge in CI; no free-form prose assertions; no behavioral guarantees for arbitrary user projects (fixtures only).
|
|
38
|
+
- No long-horizon scenarios (multi-hour sessions) — bounded, single-action scenarios first.
|
|
39
|
+
|
|
40
|
+
## Open questions
|
|
41
|
+
|
|
42
|
+
- Which harness CLIs are CI-available vs local-only, and how is availability detected robustly?
|
|
43
|
+
- Token ceiling per scenario run (real harness calls cost money) — pooled nightly budget vs per-scenario?
|
|
44
|
+
- Should scenario fixtures double as documentation examples?
|
|
@@ -0,0 +1,50 @@
|
|
|
1
|
+
# OPP-017: Orchestration depth — integration tooling, leak reporting, per-harness guidance
|
|
2
|
+
|
|
3
|
+
- Status: Proposed
|
|
4
|
+
- Priority: P2
|
|
5
|
+
- Added: 2026-11-01
|
|
6
|
+
- Decision: extends [ADR-001](../../.context/adrs/ADR-001-odd-rdd-proportionality-scope.md) Decision A
|
|
7
|
+
- Source signal: external qualitative assessment scored "Multi-agent orchestration 3/5 — scheduler and isolation; runtime delegated to host". Agreed: the delegation is by design, but three concrete pieces of depth on aiw's side of the contract are missing.
|
|
8
|
+
|
|
9
|
+
## Problem
|
|
10
|
+
|
|
11
|
+
After OPP-010, isolated execution exists: branch-per-task worktrees, evidence-bound checkpoints, cleanup. What remains missing keeps orchestration at 3/5:
|
|
12
|
+
|
|
13
|
+
1. **No integration tooling**: after a plan passes, the task branches (`aiw/<plan>/<task>`) sit without guidance — the human must remember merge order (waves) and discover conflicts by trial.
|
|
14
|
+
2. **No leak reporting**: `orchestrate --cleanup` removes managed worktrees, but `aiw doctor` does not report them — leaked worktrees from crashed runs are invisible in diagnostics.
|
|
15
|
+
3. **No per-harness guidance**: whether a given harness integration honors `OrchestrationTask.worktree` is undocumented — teams cannot know before running.
|
|
16
|
+
|
|
17
|
+
## Proposal
|
|
18
|
+
|
|
19
|
+
1. **`aiw orchestrate integrate --plan=<id>`** (read-only): prints the merge order derived from the execution waves, and for each task branch runs `git merge-tree` (dry-run) against the target branch, reporting conflicts per task before anything is merged. Merges stay human-executed (delivery boundary unchanged); the command is a conflict _forecast_, not an executor.
|
|
20
|
+
2. **Doctor leak reporting**: doctor gains a managed-worktree check (reuse `listManagedWorktrees`) — leaked worktrees are reported with plan/task attribution and the cleanup continuation (`orchestrate --cleanup --execute`), per the OPP-004 typed-continuation contract.
|
|
21
|
+
3. **Diff-stat evidence**: the checkpoint gains per-task changed-file stats (`git -C <worktree> diff --stat <base>..<final>`), so integration review starts from real scope numbers.
|
|
22
|
+
4. **Per-harness honor guidance**: a documented matrix in `docs/orchestration.md` — for each documented harness integration, whether (and how) it consumes `task.worktree`, plus the minimal adapter contract statement ("run the task with the worktree as cwd, or fail with a typed error").
|
|
23
|
+
5. **Parallel evidence**: when a plan executes with `--max-parallel > 1`, the checkpoint records wall-clock duration and per-task overlap so parallel execution is evidenced, not assumed.
|
|
24
|
+
|
|
25
|
+
## Affected areas
|
|
26
|
+
|
|
27
|
+
- `src/orchestration.ts` (integrate preview: wave-ordered merge plan + merge-tree dry run; diff-stat collection)
|
|
28
|
+
- `src/workflow.ts` (orchestrate subcommand wiring; doctor leak check)
|
|
29
|
+
- `docs/orchestration.md` (honor matrix + adapter contract statement)
|
|
30
|
+
- `docs/backlog.md` (this file)
|
|
31
|
+
|
|
32
|
+
## Acceptance criteria
|
|
33
|
+
|
|
34
|
+
- Given a passed isolated plan, when `orchestrate integrate --plan=<id>` runs, then it prints merge order respecting waves and flags conflicting task branches via merge-tree without merging anything.
|
|
35
|
+
- Given leaked managed worktrees, when `aiw doctor` runs, then they are reported with plan/task attribution and the cleanup continuation command (typed reason, OPP-004 contract).
|
|
36
|
+
- Given a task branch in conflict with the target, when the integrate forecast runs, then the conflict is reported before any merge is attempted.
|
|
37
|
+
- Given an isolated execution with `--max-parallel=2`, when the checkpoint is written, then per-task wall-clock and overlap evidence are present.
|
|
38
|
+
- Given no isolation, when orchestrate/integrate/doctor run, then behavior is unchanged.
|
|
39
|
+
|
|
40
|
+
## Non-goals
|
|
41
|
+
|
|
42
|
+
- No automatic merge, push, or orphan-branch deletion (human-owned integration).
|
|
43
|
+
- No remote/cloud orchestration; no cross-repository worktrees.
|
|
44
|
+
- No behavioral parity claims for harnesses that do not declare worktree support.
|
|
45
|
+
|
|
46
|
+
## Open questions
|
|
47
|
+
|
|
48
|
+
- Integrate target: main directly, or stacked branches per wave (easier review, harder bookkeeping)?
|
|
49
|
+
- Should integrate also propose squash strategy per task from the checkpoint's commit evidence?
|
|
50
|
+
- Doctor leak threshold: report all, or only worktrees older than N hours?
|