@multileaf/ai-workflow 0.11.0 → 0.12.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +18 -0
- package/dist/blocking-reasons.d.ts +1 -1
- package/dist/blocking-reasons.js +24 -0
- package/dist/digest.d.ts +48 -0
- package/dist/digest.js +106 -0
- package/dist/eval/artifact-evals.d.ts +42 -0
- package/dist/eval/artifact-evals.js +141 -0
- package/dist/generator.js +1 -1
- package/dist/human-approval.d.ts +13 -2
- package/dist/human-approval.js +43 -8
- package/dist/orchestration.d.ts +38 -1
- package/dist/orchestration.js +82 -3
- package/dist/quality-gates.d.ts +1 -1
- package/dist/quality-gates.js +23 -0
- package/dist/risk-assess.d.ts +4 -0
- package/dist/risk-assess.js +5 -1
- package/dist/standards.d.ts +34 -0
- package/dist/standards.js +159 -0
- package/dist/types.d.ts +15 -1
- package/dist/types.js +24 -0
- package/dist/workflow.d.ts +7 -0
- package/dist/workflow.js +426 -57
- package/docs/backlog.md +10 -3
- package/docs/continuation-inventory.md +20 -16
- package/docs/opportunities/OPP-009-process-entry-points.md +56 -0
- package/docs/opportunities/OPP-010-orchestration-worktree-isolation.md +52 -0
- package/docs/opportunities/OPP-011-evals-redteam-generated-artifacts.md +56 -0
- package/docs/opportunities/OPP-012-standards-discovery.md +51 -0
- package/docs/opportunities/OPP-013-read-optimized-artifacts-digest-layer.md +52 -0
- package/docs/opportunities/OPP-014-paired-verification-contract.md +54 -0
- package/docs/opportunities/OPP-015-token-recovery-governance.md +52 -0
- package/docs/opportunities/README.md +25 -10
- package/package.json +1 -1
- package/resources/skills/implementation-planning.md +1 -1
- package/resources/templates/brainstorm.md +0 -1
- package/resources/templates/decision-record.md +0 -1
- package/resources/templates/implementation-evidence.md +1 -1
- package/resources/templates/implementation-plan.md +2 -1
- package/resources/templates/migration-report.md +0 -1
- package/resources/templates/security-review.md +0 -1
- package/resources/templates/task.md +0 -1
- package/resources/templates/technical-design.md +0 -1
- package/resources/templates/test-plan.md +0 -1
- package/resources/templates/verification-report.md +2 -1
package/docs/backlog.md
CHANGED
|
@@ -116,11 +116,18 @@ Derived from the Gentle-AI ODD/RDD analysis; scope and rejected alternatives rec
|
|
|
116
116
|
- [x] **OPP-001** Add a light lane for small, understood work (proportional ceremony). _Acceptance:_ see [OPP-001](../opportunities/OPP-001-proportional-ceremony-light-lane.md); no durable spec/plan artifacts for light tasks, evidence and lane rationale still recorded, explicit escalation to the full lane on scope growth, full lane unchanged.
|
|
117
117
|
- [x] **OPP-002** Add `aiw risk assess` — read-only, content-based risk classification with an exact runnable next command. _Acceptance:_ see [OPP-002](../opportunities/OPP-002-risk-assess-command.md); deterministic and offline by default, no `.aiw/` mutation, escalation raises and never lowers tier, `next_command` executes as printed.
|
|
118
118
|
- [x] **OPP-003** Ship an optional `adversarial-review` package (blind judges + refuter with proof-carrying findings). _Acceptance:_ see [OPP-003](../opportunities/OPP-003-adversarial-review-package.md); severe findings without reproducible evidence degrade to advisory; refutation outcomes are corroborated/refuted/inconclusive with proof references; human acceptance still required.
|
|
119
|
-
- [
|
|
119
|
+
- [x] **OPP-004** Emit typed reason codes and exact continuation commands on every blocking exit. _Acceptance:_ see [OPP-004](../opportunities/OPP-004-exact-continuation-commands.md); every block carries a reason code and either a tested runnable `next_command` or an explicit terminal statement; continuation inventory stays complete in CI.
|
|
120
120
|
- [x] **OPP-005** Add a friction benchmark classifying blocks as in-band / out-of-band / dead-end over packaged-CLI journeys. _Acceptance:_ see [OPP-005](../opportunities/OPP-005-friction-benchmark.md); deterministic classification, continuations actually executed, CI fails on any dead end.
|
|
121
121
|
- [x] **OPP-006** Adopt the emission ≠ usability test policy: messages naming a command must have a test executing it. _Acceptance:_ see [OPP-006](../opportunities/OPP-006-emission-usability-test-policy.md); extraction helper covers emitted and generated command mentions; generated `ai-init` skills' commands are executed in e2e tests.
|
|
122
|
-
- [ ] **007
|
|
123
|
-
- [ ] **008
|
|
122
|
+
- [ ] **OPP-007** Decision model — findings-evidence triage and lane scope-watch (advisory-only). _Acceptance:_ see [OPP-007](../opportunities/OPP-007-decision-findings-triage-scope-watch.md); unconfigured = byte-identical offline behavior; advisory flags never block, never lower tiers, never choose lanes.
|
|
123
|
+
- [ ] **OPP-008** Embedding retrieval — covering-test detection and judge context scoping. _Acceptance:_ see [OPP-008](../opportunities/OPP-008-embedding-covering-tests-judge-context.md); retrieval only prevents false `untested` flags (absence fails safe toward high); scoped judge prompts labeled as context aids within token ceilings.
|
|
124
|
+
- [x] **OPP-009** Processes as entry points — bug-fixing and idea-assessment process registries alongside the feature SDD lane. _Acceptance:_ see [OPP-009](../opportunities/OPP-009-process-entry-points.md); per-process stage lists, diagnosis/assessment artifacts with gates, default feature process byte-identical, archived ledgers backward compatible.
|
|
125
|
+
- [x] **OPP-010** Worktree isolation for orchestrated execution (`orchestrate --isolate`). _Acceptance:_ see [OPP-010](../opportunities/OPP-010-orchestration-worktree-isolation.md); per-task worktrees, evidence-bound checkpoints, capability-declared adapter refusal, cleanup guarded by doctor; no flag = unchanged behavior.
|
|
126
|
+
- [x] **OPP-011** Evals + red-team layer for generated artifacts. _Acceptance:_ see [OPP-011](../opportunities/OPP-011-evals-redteam-generated-artifacts.md); per-adapter contract invariants in CI, deterministic injection fixtures, offline local run, ratchet for skill/template changes.
|
|
127
|
+
- [x] **OPP-012** Standards discovery — extract codebase conventions into confirmed, versioned standards. _Acceptance:_ see [OPP-012](../opportunities/OPP-012-standards-discovery.md); evidence-cited extraction with confirmation contract, deterministic offline floor, just-in-time injection into rules/skills, advisory drift reporting.
|
|
128
|
+
- [x] **OPP-013** Read-Optimized Artifacts — fingerprint-bound digest layer so downstream stages read digests instead of re-reading full artifacts. _Acceptance:_ see [OPP-013](../opportunities/OPP-013-read-optimized-artifacts-digest-layer.md); digests extracted deterministically at approval, stale-safe (fail toward full read), ≥60% reduction in recorded full-artifact reads, no model-generated summaries.
|
|
129
|
+
- [x] **OPP-014** Paired Verification Contract — plan declares `command → expected`, evidence records `command → observed`, gates compare pairs. _Acceptance:_ see [OPP-014](../opportunities/OPP-014-paired-verification-contract.md); mismatch/missing pairs fail with typed reasons and machine-readable task ids; gates never execute commands; legacy format accepted with advisory.
|
|
130
|
+
- [x] **OPP-015** Token & Recovery Governance — enforced stage budgets, context-recovery managed block, advisory lane suggestion, regression-pinning declaration. _Acceptance:_ see [OPP-015](../opportunities/OPP-015-token-recovery-governance.md); budgets derive from saved baselines and default-inert; recovery block in every adapter rendering; advisory line never blocks.
|
|
124
131
|
|
|
125
132
|
## Backlog rules
|
|
126
133
|
|
|
@@ -6,19 +6,23 @@ Every blocking `aiw` exit carries a stable reason code plus either an exact runn
|
|
|
6
6
|
- With `--json`, blocking exits return `{ "reason_code": ..., "next_command" | "next_action" | "terminal": ... }`.
|
|
7
7
|
- The table below is generated from the registry; a test in `src/blocking-reasons.test.ts` fails CI when this page drifts from the code.
|
|
8
8
|
|
|
9
|
-
| Reason code | Description
|
|
10
|
-
| ------------------------------- |
|
|
11
|
-
| `gate_evidence_missing` | The quality gate is blocked because the required evidence artifact is missing.
|
|
12
|
-
| `gate_incomplete` | The quality gate found completeness issues in the stage artifact.
|
|
13
|
-
| `stage_prerequisite_pending` | A previous stage is awaiting human approval before this stage can start.
|
|
14
|
-
| `stage_prerequisite_unresolved` | Previous stages need human approval or an accepted optional skip before this stage can start.
|
|
15
|
-
| `stage_awaiting_approval` | This stage is awaiting human approval; approve or reject it before continuing.
|
|
16
|
-
| `stage_already_approved` | This stage is already approved; changed evidence requires rerunning its quality gate.
|
|
17
|
-
| `approval_evidence_missing` | Approval cannot be recorded because the stage evidence artifact is missing.
|
|
18
|
-
| `approval_stale` | The artifact changed after review; its approval is stale and the gate must run again.
|
|
19
|
-
| `approval_gate_missing` | No passing quality gate is recorded for this stage; gate it before approving.
|
|
20
|
-
| `lane_escalation_required` | Light-lane workflows do not author stage artifacts; escalate to the full lane to continue.
|
|
21
|
-
| `light_evidence_missing` | A light-lane workflow cannot complete without at least one recorded validation entry.
|
|
22
|
-
| `workflow_still_active` | A workflow is already active; finish or archive it before starting another.
|
|
23
|
-
| `workflow_not_active` | No SDD workflow is active for this command to operate on.
|
|
24
|
-
| `workflow_complete` | This workflow is already complete; start a new workflow to continue working.
|
|
9
|
+
| Reason code | Description | Continuation |
|
|
10
|
+
| ------------------------------- | -------------------------------------------------------------------------------------------------------- | ---------------- |
|
|
11
|
+
| `gate_evidence_missing` | The quality gate is blocked because the required evidence artifact is missing. | explicit action |
|
|
12
|
+
| `gate_incomplete` | The quality gate found completeness issues in the stage artifact. | explicit action |
|
|
13
|
+
| `stage_prerequisite_pending` | A previous stage is awaiting human approval before this stage can start. | runnable command |
|
|
14
|
+
| `stage_prerequisite_unresolved` | Previous stages need human approval or an accepted optional skip before this stage can start. | runnable command |
|
|
15
|
+
| `stage_awaiting_approval` | This stage is awaiting human approval; approve or reject it before continuing. | runnable command |
|
|
16
|
+
| `stage_already_approved` | This stage is already approved; changed evidence requires rerunning its quality gate. | runnable command |
|
|
17
|
+
| `approval_evidence_missing` | Approval cannot be recorded because the stage evidence artifact is missing. | explicit action |
|
|
18
|
+
| `approval_stale` | The artifact changed after review; its approval is stale and the gate must run again. | runnable command |
|
|
19
|
+
| `approval_gate_missing` | No passing quality gate is recorded for this stage; gate it before approving. | runnable command |
|
|
20
|
+
| `lane_escalation_required` | Light-lane workflows do not author stage artifacts; escalate to the full lane to continue. | explicit action |
|
|
21
|
+
| `light_evidence_missing` | A light-lane workflow cannot complete without at least one recorded validation entry. | explicit action |
|
|
22
|
+
| `workflow_still_active` | A workflow is already active; finish or archive it before starting another. | explicit action |
|
|
23
|
+
| `workflow_not_active` | No SDD workflow is active for this command to operate on. | runnable command |
|
|
24
|
+
| `workflow_complete` | This workflow is already complete; start a new workflow to continue working. | runnable command |
|
|
25
|
+
| `installation_required` | AI Workflow is not installed in this repository; install it to continue. | runnable command |
|
|
26
|
+
| `state_mismatch` | AI Workflow state files are inconsistent with each other; run diagnostics to inspect and repair. | runnable command |
|
|
27
|
+
| `context_budget_exceeded` | The served context exceeds the declared ceiling for this stage; trim it or raise the ceiling explicitly. | explicit action |
|
|
28
|
+
| `stage_not_in_process` | This stage does not exist in the active workflow's process; use the process's own stages. | explicit action |
|
|
@@ -0,0 +1,56 @@
|
|
|
1
|
+
# OPP-009: Processes as entry points — bug-fixing and idea assessment alongside SDD
|
|
2
|
+
|
|
3
|
+
- Status: Implemented (2026-11-01): feature/bugfix/idea-assessment processes with per-process stage lists, gates, and typed stage_not_in_process blocks
|
|
4
|
+
- Priority: P2
|
|
5
|
+
- Added: 2026-11-01
|
|
6
|
+
- Decision: extends [ADR-001](../../.context/adrs/ADR-001-odd-rdd-proportionality-scope.md) Decision A (lane architecture)
|
|
7
|
+
- Trend signal: `github/spec-kit` (145K★) ships bug-fixing and idea-assessment as **independent entry points, not mandatory phases** — the market confirmed the proportionality direction ADR-001 adopted; this opportunity matches its breadth.
|
|
8
|
+
|
|
9
|
+
## Problem
|
|
10
|
+
|
|
11
|
+
AI Workflow's staged pipeline is feature-centric. Real work arrives in at least two other shapes that today are force-fitted into the full lane (generating unread artifacts) or bypass the workflow entirely (generating no evidence):
|
|
12
|
+
|
|
13
|
+
1. **Bug reports** — nobody brainstorms a bug. The honest process is: reproduce/diagnose → scoped fix → verification that the reproduction now passes and the original behavior holds.
|
|
14
|
+
2. **Idea evaluation** — before any spec exists, the question is "is this worth building?" — an evidence-backed go / clarify / stop decision, not an implementation plan.
|
|
15
|
+
|
|
16
|
+
spec-kit's model validates the shape: processes are independent entry points assembled from shared primitives, some bundled as extensions. AI Workflow's lanes, reason codes, gates, and approval ledger are exactly such primitives — this opportunity composes them into first-class processes.
|
|
17
|
+
|
|
18
|
+
## Proposal
|
|
19
|
+
|
|
20
|
+
1. **Process definitions** (data, not code paths): each process declares its ordered stages and optional stages, extending the lane concept (OPP-001) from two lanes of one process to a small process registry:
|
|
21
|
+
- `feature` (default, current 8 stages, unchanged);
|
|
22
|
+
- `bugfix`: `diagnosis` (reproduction evidence is the artifact — observed behavior vs expected, environment) → scoped fix implementation (TDD: the failing reproduction test is the RED test) → verification → review. Brainstorming/specification/technical-design do not exist in this process; requirements are the reproduction itself.
|
|
23
|
+
- `idea-assessment`: evidence gathering (read-only research with attributed sources) → assessment artifact with a single outcome (`go` / `clarify` / `stop`) + human decision; `go` may chain into a `feature` workflow carrying the evidence forward.
|
|
24
|
+
2. **CLI**: `aiw task start --ticket=X --process=bugfix` (feature stays the default; the process choice is human-made like the lane). Process names validate against the registry; `--process` composes with `--lane=light` where the process supports it (bugfix supports light; idea-assessment is inherently its own artifact shape).
|
|
25
|
+
3. **Ledger**: `ApprovalLedger` gains a `process` field (default `feature`); stage validation (`assertStageReady`, `getUnresolvedPrerequisites`) reads the process's stage list instead of the fixed `APPROVAL_STAGES`. Gates and fingerprint binding are reused as-is — every stage keeps the same "automated gate ≠ human approval" contract.
|
|
26
|
+
4. **Templates/skills**: `resources/templates/` gains diagnosis and assessment templates; a `bug-fixing` skill guides the diagnosis artifact (reproduction command, observed/expected, root cause with evidence, scope statement); an `idea-assessment` skill guides evidence-backed go/clarify/stop with attributed sources.
|
|
27
|
+
5. **Traceability**: `aiw trace` accepts diagnosis artifacts as evidence sources for bugfix workflows (requirement = the reproduction statement).
|
|
28
|
+
|
|
29
|
+
## Affected areas
|
|
30
|
+
|
|
31
|
+
- `src/types.ts` + `src/human-approval.ts` (process registry, per-process stage lists, ledger `process` field, validation per process)
|
|
32
|
+
- `src/workflow.ts` (`task start --process`, stage command routing per process, `workflow complete` per-process stage set)
|
|
33
|
+
- `src/quality-gates.ts` (diagnosis and assessment completeness validators)
|
|
34
|
+
- `src/traceability.ts` (diagnosis as evidence source)
|
|
35
|
+
- `resources/templates/`, `resources/skills/bug-fixing.md`, `resources/skills/idea-assessment.md`, `resources/package.yaml`
|
|
36
|
+
- `docs/architecture.md` (process registry section), `README.md`
|
|
37
|
+
|
|
38
|
+
## Acceptance criteria
|
|
39
|
+
|
|
40
|
+
- Given a bugfix task, when the workflow runs, then only diagnosis → implementation → verification → review stages exist; `aiw spec` blocks with a typed reason (`stage_not_in_process`); the diagnosis artifact requires a reproduction command and observed/expected behavior to pass its gate.
|
|
41
|
+
- Given an idea-assessment task, when it completes, then the assessment artifact records an outcome (`go`/`clarify`/`stop`) with attributed evidence, and a human decision on the outcome is required before the workflow closes; `go` records a forward link to the created feature workflow.
|
|
42
|
+
- Given the default no-`--process` invocation, when any stage command runs, then behavior is byte-identical to the feature process today.
|
|
43
|
+
- Given an archived ledger from ≤0.10.0, when 0.11+ reads it, then it is treated as `process: feature` (backward compatible, no migration).
|
|
44
|
+
- Given any process, when a stage blocks, then the blocking exit carries a reason code and exact continuation per the OPP-004 contract.
|
|
45
|
+
|
|
46
|
+
## Non-goals
|
|
47
|
+
|
|
48
|
+
- No user-defined custom processes in this iteration (the registry is code-defined; arbitrary composition is a later extension).
|
|
49
|
+
- No changes to the feature process's stage order, gates, or approvals.
|
|
50
|
+
- No automatic process inference from the ticket — the human picks; automation may suggest.
|
|
51
|
+
|
|
52
|
+
## Open questions
|
|
53
|
+
|
|
54
|
+
- Should `idea-assessment` outcomes feed recommendations (a `stop` recorded against a capability suppresses re-suggestion)?
|
|
55
|
+
- Does the bugfix process need a `regression-test` gate item distinct from verification evidence?
|
|
56
|
+
- Naming: `bugfix` vs `fix`; `idea-assessment` vs `assessment`.
|
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
# OPP-010: Worktree isolation for orchestrated execution
|
|
2
|
+
|
|
3
|
+
- Status: Implemented (2026-11-01): orchestrate --isolate with branch-per-task worktrees, evidence-bound checkpoints, kept-on-failure worktrees, and --cleanup
|
|
4
|
+
- Priority: P2
|
|
5
|
+
- Added: 2026-11-01
|
|
6
|
+
- Decision: extends [ADR-001](../../.context/adrs/ADR-001-odd-rdd-proportionality-scope.md) Decision A
|
|
7
|
+
- Trend signal: Spec Kitty sells isolated worktrees as its core ("implementation can happen in parallel without branch chaos"); Conductor (3.7K★) and Grasp are built entirely around parallel agent worktrees; AI Workflow's own `docs/quality.md` already mandates isolated-worktree validation runs — currently performed manually.
|
|
8
|
+
|
|
9
|
+
## Problem
|
|
10
|
+
|
|
11
|
+
`aiw orchestrate` schedules a validated task graph and delegates each task to an adapter, but every task runs against the **same working tree**. Consequences:
|
|
12
|
+
|
|
13
|
+
- parallel tasks conflict on shared files (the reason `--max-parallel` exists but parallelism stays risky);
|
|
14
|
+
- the quality.md requirement — "externally visible tasks also require an isolated integration run by an AI subagent in a separate worktree" — is satisfied by hand, with no evidence that the worktree was actually isolated;
|
|
15
|
+
- a failed task leaves the main tree dirty, mixing orchestration debris with human work.
|
|
16
|
+
|
|
17
|
+
## Proposal
|
|
18
|
+
|
|
19
|
+
1. **`aiw orchestrate --isolate`**: before executing a task (or an independent wave), create a git worktree from the current HEAD under `.aiw/worktrees/<plan-id>/<task-id>/`; pass that path as the task's working directory; remove the worktree on success (after evidence capture) and keep it on failure for diagnosis.
|
|
20
|
+
2. **Adapter contract extension** (backward compatible): `OrchestrationTask` gains an optional `worktree?: string`; the adapter contract keeps "the adapter owns process isolation" — aiw owns **git** isolation and hands the path over. Adapters that cannot use a worktree reject `--isolate` with a typed error naming the capability (consistent with the adapter capability matrix).
|
|
21
|
+
3. **Evidence binding**: the per-task checkpoint (already `orchestration-<plan-id>.yml`) records the worktree path, base commit, and final commit/diff stat, so the "isolated run happened" requirement becomes verifiable evidence instead of prose.
|
|
22
|
+
4. **Merge strategy**: on success, a fast-forward/merge step records which task branches are ready to integrate; merge remains a human-approved action (delivery is never automatic — consistent with the approval contract).
|
|
23
|
+
5. **Cleanup guard**: `aiw orchestrate --cleanup` lists and removes stale worktrees for completed/failed plans; doctor reports leaked worktrees (a worktree whose plan checkpoint shows completion but still exists on disk).
|
|
24
|
+
|
|
25
|
+
## Affected areas
|
|
26
|
+
|
|
27
|
+
- `src/orchestration.ts` (worktree lifecycle, evidence fields, cleanup)
|
|
28
|
+
- `src/adapter-contract.ts` (optional `worktree` task field, capability declaration)
|
|
29
|
+
- Adapters (`src/claude-adapter.ts`, codex, gemini, copilot, universal) — pass-through or capability-declared refusal
|
|
30
|
+
- `src/doctor`/`src/workflow.ts` diagnostics (leaked worktree check)
|
|
31
|
+
- `docs/orchestration.md`, `docs/quality.md` (manual requirement → tooling requirement)
|
|
32
|
+
|
|
33
|
+
## Acceptance criteria
|
|
34
|
+
|
|
35
|
+
- Given `--isolate` with two independent tasks, when the plan executes with `--max-parallel=2`, then each task runs in its own worktree from the same base commit and their file writes never touch the main tree.
|
|
36
|
+
- Given a task failure under isolation, when the plan finishes, then the failed task's worktree is preserved on disk and the checkpoint names its path and base commit.
|
|
37
|
+
- Given a successful task under isolation, when evidence is recorded, then the checkpoint contains base commit, final commit, and diff stat for that task.
|
|
38
|
+
- Given an adapter that does not declare worktree support, when `--isolate` is used, then the plan fails before any task runs with a typed error naming the adapter and capability.
|
|
39
|
+
- Given no `--isolate` flag, when a plan executes, then behavior is byte-identical to today.
|
|
40
|
+
- Given leaked worktrees, when `aiw doctor` runs, then they are reported with plan/task attribution; `--cleanup` removes them after explicit confirmation.
|
|
41
|
+
|
|
42
|
+
## Non-goals
|
|
43
|
+
|
|
44
|
+
- No automatic merge/push of task results — integration stays human-approved.
|
|
45
|
+
- No cross-repository worktrees; isolation is within the current repository.
|
|
46
|
+
- No rebase/-conflict resolution automation: conflicting task branches surface for human merge.
|
|
47
|
+
|
|
48
|
+
## Open questions
|
|
49
|
+
|
|
50
|
+
- Branch-per-task inside the worktree (easier merge later) vs detached HEAD with recorded commit (simpler)? Recommendation: branch-per-task named `aiw/<plan>/<task>`.
|
|
51
|
+
- Should isolated runs reuse OPP-002's tier to decide whether a task needs isolation at all (advisory)?
|
|
52
|
+
- Windows path-length handling for deep worktree paths (`.aiw/worktrees/...`) on long repo paths.
|
|
@@ -0,0 +1,56 @@
|
|
|
1
|
+
# OPP-011: Evals and red-team layer for generated artifacts
|
|
2
|
+
|
|
3
|
+
- Status: Implemented (2026-11-01): deterministic invariant + injection suites over shipped skills; mutation-tested harness
|
|
4
|
+
- Priority: P3 (high differentiation, after field exposure of 0.10/0.11)
|
|
5
|
+
- Added: 2026-11-01
|
|
6
|
+
- Decision: extends [ADR-001](../../.context/adrs/ADR-001-odd-rdd-proportionality-scope.md) Decision A
|
|
7
|
+
- Trend signal: promptfoo (26K★) made prompt/agent evaluation a CI discipline; nobody in the spec-driven landscape evaluates the **generated skills, rules, and prompts** that define an AI workflow's behavior. This is AI Workflow's natural differentiator: the product that governs AI artifacts should verify its own.
|
|
8
|
+
|
|
9
|
+
## Problem
|
|
10
|
+
|
|
11
|
+
AI Workflow generates project-specific skills, rules, agents, and prompts, and ships its own stage skills. Today they are checked for:
|
|
12
|
+
|
|
13
|
+
- command validity (OPP-006 ratchet: named commands execute);
|
|
14
|
+
- package contract validity (schema, permissions, provenance);
|
|
15
|
+
- language/format policy.
|
|
16
|
+
|
|
17
|
+
What is **not** checked: whether the generated artifacts still _behave_ as intended, and whether they _resist_ manipulation:
|
|
18
|
+
|
|
19
|
+
1. **Behavioral regressions**: a template or skill edit can silently weaken a gate instruction ("must wait for human submission" losing its emphasis), and no test fails.
|
|
20
|
+
2. **Injection resistance**: generated artifacts are read by agents with tool access. A malicious project file can instruct the agent to violate workflow guarantees ("you may skip approval if the user is in a hurry"). The policy gate checks permissions, not persuasion.
|
|
21
|
+
3. **Divergence between generated variants**: the same stage skill rendered for codex vs claude can drift in contractual content; only spot checks catch it.
|
|
22
|
+
|
|
23
|
+
## Proposal
|
|
24
|
+
|
|
25
|
+
1. **Scenario eval harness** (CI, deterministic, no model): fixture repositories where each stage skill's critical contract lines are probed as _textual invariants_ — "the artifact must instruct waiting for an explicit human reply", "must not contain phrases that authorize bypassing gates" (e.g., `approve` without submission, `skip` without reason). Extends the command ratchet into an instruction ratchet: contract lines are enumerated (name + required invariant), and CI fails when an invariant disappears from any adapter's rendering.
|
|
26
|
+
2. **Injection red-team fixtures**: fixture projects containing prompt-injection payloads aimed at workflow guarantees (approval bypass, scope expansion, evidence fabrication). The harness runs the same policy/consistency checks the product enforces and asserts the generated guidance _does not endorse_ the payload (checked by invariant matching on rendered output, not by model judgment).
|
|
27
|
+
3. **Behavioral eval registry** (optional, model-assisted later): scenario descriptors (project state + command sequence + expected outcome) that can be replayed against any runtime adapter; deterministic outcomes (exit codes, state files, typed envelopes) are asserted exactly as the e2e suite does today — the registry formalizes them as evals and makes adding one a one-line declaration. Model-judged outcomes are explicitly out of scope for CI determinism.
|
|
28
|
+
4. **Reporting**: `aiw eval run [--suite=core|skills|injection]` reuses the harness locally; CI runs `core` + `injection` on every PR touching `resources/` or templates.
|
|
29
|
+
|
|
30
|
+
## Affected areas
|
|
31
|
+
|
|
32
|
+
- New `src/eval/` (scenario registry, invariant extraction from skill markdown, harness)
|
|
33
|
+
- `resources/skills/*.md`, `resources/templates/*` (invariant annotations: stable anchor comments the harness counts on)
|
|
34
|
+
- `.github/workflows/ci.yml` (eval suites on resource-touching PRs)
|
|
35
|
+
- `src/package-audit.ts` / `src/policy-gate.ts` (reuse permission and policy surfaces for injection checks)
|
|
36
|
+
- `docs/quality.md` (eval suites as completion criteria for skill/template changes)
|
|
37
|
+
|
|
38
|
+
## Acceptance criteria
|
|
39
|
+
|
|
40
|
+
- Given a stage skill rendered for every adapter, when invariants run, then each declared contract invariant (approval waiting, skip reasons, gate-before-approve) is present in every rendering; a missing one fails CI naming the adapter and invariant.
|
|
41
|
+
- Given an injection fixture (approval-bypass payload in a project file), when the eval suite runs, then no generated artifact endorses the bypass (invariant match), and the run is deterministic across executions.
|
|
42
|
+
- Given a template edit that removes a gate instruction, when CI runs, then the behavioral invariant fails naming the removed anchor.
|
|
43
|
+
- Given `aiw eval run --suite=skills`, when run locally, then it reproduces the CI result offline with no model configured.
|
|
44
|
+
- Given a new skill/template PR, when it touches rendered contract content, then the PR must include invariant updates (ratchet like OPP-006).
|
|
45
|
+
|
|
46
|
+
## Non-goals
|
|
47
|
+
|
|
48
|
+
- No LLM-as-judge in CI (determinism is the whole point; model-assisted evaluation stays a local, advisory tool).
|
|
49
|
+
- No red-teaming of end-user code — the target is AI Workflow's own generated artifacts.
|
|
50
|
+
- No runtime enforcement changes: this hardens what ships, it does not add new gates to user workflows.
|
|
51
|
+
|
|
52
|
+
## Open questions
|
|
53
|
+
|
|
54
|
+
- Invariant anchor format: HTML comments in skill markdown vs a sidecar manifest mapping invariants to files?
|
|
55
|
+
- Should injection fixtures live in the package contract tests (per third-party package) or the core harness?
|
|
56
|
+
- Do rendered-variant diffs (codex vs claude) warrant a dedicated divergence check beyond shared invariants?
|
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
# OPP-012: Standards discovery — extract codebase conventions into versioned standards
|
|
2
|
+
|
|
3
|
+
- Status: Implemented (2026-11-01, deterministic floor): aiw standards extract|list|confirm|reject with evidence-cited inferred standards; OPP-008 retrieval extension remains future
|
|
4
|
+
- Priority: P3 (depends on OPP-008 retrieval plumbing)
|
|
5
|
+
- Added: 2026-11-01
|
|
6
|
+
- Decision: extends [ADR-001](../../.context/adrs/ADR-001-odd-rdd-proportionality-scope.md) Decision A
|
|
7
|
+
- Trend signal: Agent OS (5.5K★) built its positioning on "Discover Standards — extract patterns and conventions from your codebase into documented standards" and "Deploy Standards — inject relevant standards based on what you're building". AI Workflow generates rules from the detected **stack**; the trend is to extract them from the **codebase's actual conventions**, with evidence.
|
|
8
|
+
|
|
9
|
+
## Problem
|
|
10
|
+
|
|
11
|
+
AI Workflow generates project rules from detected and confirmed facts (stack, tools, scripts). What it does not capture: **how this codebase actually does things** — error-handling idiom, naming conventions, test structure, state management patterns. Consequences:
|
|
12
|
+
|
|
13
|
+
- generated guidance is generic ("this is a TypeScript project") rather than specific ("this repo wraps errors in `AppError` with `code` fields; tests live beside sources as `*.test.ts` using vitest `describe/it`");
|
|
14
|
+
- agents routinely propose idiomatic-but-foreign code that reviewers must correct;
|
|
15
|
+
- conventions live in senior developers' heads — exactly the knowledge agents lack.
|
|
16
|
+
|
|
17
|
+
## Proposal
|
|
18
|
+
|
|
19
|
+
1. **`aiw standards extract`** (read-only, evidence-linked): analyzes the repository (deterministic detectors first: file layout, test naming, config presence; semantic retrieval via OPP-008 when enabled for pattern instances) and produces **standards documents** under `.aiw/standards/<topic>.md` — each statement carries: the convention, **evidence citations** (file:line examples, N occurrences), and a confidence tier (detected-strong / detected-weak / proposed).
|
|
20
|
+
2. **Confirmation contract** (consistent with profile facts): standards are `inferred` until the human accepts/rejects/edits them via `aiw confirm`; accepted standards become `confirmed` and are versioned in the lockfile. AI output never silently becomes rule content.
|
|
21
|
+
3. **Injection into generated rules**: confirmed standards are rendered into project rules (and referenced from stage skills: "follow `.aiw/standards/error-handling.md`") through the existing generator — the same preserved-override/drift semantics as other generated content (GEN-002).
|
|
22
|
+
4. **Deterministic floor**: with no semantic mapping configured, extraction covers deterministic conventions only (naming patterns via glob/AST-lite heuristics, test layout, error-class presence) — no network, useful even offline. Retrieval only extends coverage (OPP-008's consumer, as anticipated).
|
|
23
|
+
5. **Freshness**: standards carry their extraction commit; `aiw scan` reports drift when the codebase diverges from recorded standards (files matching a standard's evidence changed) — advisory, consistent with drift detection elsewhere.
|
|
24
|
+
|
|
25
|
+
## Affected areas
|
|
26
|
+
|
|
27
|
+
- New `src/standards.ts` (extractors, confirmation state, rendering)
|
|
28
|
+
- `src/scanner.ts` (deterministic convention detectors), `src/semantic.ts` (OPP-008 retrieval for pattern instances)
|
|
29
|
+
- `src/generator.ts` (rule injection referencing confirmed standards), `src/drift.ts` (standards drift reporting)
|
|
30
|
+
- `.aiw/standards/` (new state directory), lockfile (confirmed standards)
|
|
31
|
+
- `docs/README.md`, `docs/architecture.md`, README
|
|
32
|
+
|
|
33
|
+
## Acceptance criteria
|
|
34
|
+
|
|
35
|
+
- Given a repository with consistent error-wrapping, when `aiw standards extract` runs offline, then the error-handling standard cites ≥2 file:line examples with occurrence counts and tier `detected-strong`, pending confirmation.
|
|
36
|
+
- Given inferred standards, when the human confirms one, then it becomes `confirmed`, is recorded in the lockfile, and appears in regenerated rules; rejected ones are never rendered.
|
|
37
|
+
- Given no semantic mapping configured, when extracting, then deterministic conventions still produce standards and no network request is made.
|
|
38
|
+
- Given confirmed standards, when a stage skill renders, then it references applicable standards by path (just-in-time, per the token-cost principle — standards content is loaded on demand, not inlined everywhere).
|
|
39
|
+
- Given a codebase change contradicting a confirmed standard, when `aiw scan` runs, then drift is reported as advisory with the diverging files named.
|
|
40
|
+
|
|
41
|
+
## Non-goals
|
|
42
|
+
|
|
43
|
+
- No auto-confirmation: all standards start inferred (profile-facts contract).
|
|
44
|
+
- No style-lint replacement: standards describe conventions; enforcement stays with the project's own linters.
|
|
45
|
+
- No cross-repository standards sharing in this iteration (org-level standards live in ENT-001/002 territory).
|
|
46
|
+
|
|
47
|
+
## Open questions
|
|
48
|
+
|
|
49
|
+
- Extractor taxonomy for v1: error handling, testing, naming, module layout — what else earns a dedicated extractor?
|
|
50
|
+
- Should standards participate in traceability (a rule citing a standard citing files)?
|
|
51
|
+
- Granularity: one document per topic vs one per convention-cluster (affects drift reporting precision)?
|
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
# OPP-013: Read-Optimized Artifacts — the Digest Layer
|
|
2
|
+
|
|
3
|
+
- Status: Implemented (2026-11-01): fingerprint-bound digests computed at approval; aiw digest and context --stage serve them stale-safe
|
|
4
|
+
- Priority: P1 (largest measurable token lever in the product)
|
|
5
|
+
- Added: 2026-11-01
|
|
6
|
+
- Decision: extends [ADR-001](../../.context/adrs/ADR-001-odd-rdd-proportionality-scope.md) Decision A
|
|
7
|
+
- Measured baseline (this repository's own artifacts): a full-lane feature whose specification is ~2.2K tokens and plan ~4.1K tokens pays **~25-30K tokens in downstream re-reads alone** — the specification is re-read ~6-7× (plan, implementation, verification, review, traceability) and the plan ~4×. Writing artifacts is ~20% of their lifetime token cost; re-reading them is ~80%.
|
|
8
|
+
|
|
9
|
+
## Problem
|
|
10
|
+
|
|
11
|
+
Every artifact in the staged workflow is written once and read many times. The single dominant token cost of a full-lane feature is not authoring — it is downstream stages re-ingesting full artifacts to extract the handful of facts they need:
|
|
12
|
+
|
|
13
|
+
- `plan` needs the spec's requirement IDs and acceptance criteria — re-reads everything;
|
|
14
|
+
- `implementation` needs its task's linked requirements — re-reads the spec and the whole plan per task;
|
|
15
|
+
- `verification`/`review`/`traceability` need requirement→evidence mappings — re-read both.
|
|
16
|
+
|
|
17
|
+
Three methodologies shrink artifacts (ODD: fewer; HDAE: smaller; spec-kit: better templates). None attack the multiplier: **write once, read N×**. AI Workflow can be the first to formalize the separation: **full artifacts are for humans; agents get a derived, verifiable view**.
|
|
18
|
+
|
|
19
|
+
## Proposal
|
|
20
|
+
|
|
21
|
+
1. **Digest at the approval boundary**: when a stage passes its gate and receives human approval, aiw computes a structured **digest** from the exact approved bytes and stores it fingerprint-bound: `goal`, `scope` (one line), `requirements: [{id, one-line, criteria-ids}]`, `non-goals`, `decisions-open`, `links` (artifacts and commits). Deterministic extraction from required sections (the gate already enforces their presence) — **never a model-generated summary**.
|
|
22
|
+
2. **Downstream reads target digests by default**: `aiw context --stage=<stage>` (CTX-002's delta retrieval) returns the manifest of digests plus only the artifact sections a stage's manifest declares; full reads remain available (`--full`, or task-linked requirement sections by ID anchor).
|
|
23
|
+
3. **Digest integrity**: the digest records the artifact's SHA-256 fingerprint (already bound in `.aiw/approvals.yml`); a stale digest (artifact changed without re-gating) is reported as stale, never silently served — same semantics as approval staleness.
|
|
24
|
+
4. **Template-declared fields**: digests extract from the sections templates already mandate (`## Requirements`, `## Acceptance Criteria`, `## Tasks`, `## Validation`), so extraction is parse-not-guess; templates may declare additional digest fields per process (OPP-009).
|
|
25
|
+
5. **Read-path reporting**: `aiw token-usage` gains a digest row (served-digest vs full-read counts) so the saving is measured, not assumed.
|
|
26
|
+
|
|
27
|
+
## Affected areas
|
|
28
|
+
|
|
29
|
+
- `src/context-retrieval.ts` / `src/context-cache.ts` (digest store + default-read routing)
|
|
30
|
+
- `src/quality-gates.ts` (digest extraction validators per artifact type)
|
|
31
|
+
- `src/workflow.ts` (compute at approve; serve in stage flows), `src/token-usage.ts` (read-path accounting)
|
|
32
|
+
- `resources/templates/*` (digest field declarations), `docs/architecture.md`, `docs/token-efficiency.md`
|
|
33
|
+
|
|
34
|
+
## Acceptance criteria
|
|
35
|
+
|
|
36
|
+
- Given an approved specification, when a downstream stage requests context, then it receives the digest plus only declared sections; a full read happens only with an explicit flag or a linked-section reference.
|
|
37
|
+
- Given a digest, when the artifact's fingerprint no longer matches the approval ledger, then the digest is reported stale and the full artifact is required (fail-safe toward fidelity).
|
|
38
|
+
- Given the digest extraction, when the artifact lacks a mandated section, then the gate fails before approval (digests only exist for gate-passing artifacts).
|
|
39
|
+
- Given no flag changes, when a light-lane workflow runs, then behavior is unchanged (light tasks produce no durable artifacts to digest).
|
|
40
|
+
- Given a full-lane feature at the measured baseline, when downstream stages complete, then recorded full-artifact reads drop by at least 60% (served-digest accounting in `token-usage`).
|
|
41
|
+
|
|
42
|
+
## Non-goals
|
|
43
|
+
|
|
44
|
+
- No model-generated summaries, ever (a stochastic digest would void the fingerprint guarantee).
|
|
45
|
+
- No digest replacing the artifact for humans; no digest content beyond extraction from mandated sections.
|
|
46
|
+
- No change to gate or approval semantics — digests exist only downstream of approval.
|
|
47
|
+
|
|
48
|
+
## Open questions
|
|
49
|
+
|
|
50
|
+
- Digest storage: sidecar files (`.aiw/digests/<artifact>.yml`) vs front-matter appended to the artifact itself (one file, but touches approved bytes — likely sidecar, to keep approved artifacts immutable)?
|
|
51
|
+
- Anchor format for per-requirement reads (headings are stable; `REQ-\d+` IDs are already mandated).
|
|
52
|
+
- Do plan digests include per-task validation commands (OPP-014 pairs), or do those stay task-scoped?
|
|
@@ -0,0 +1,54 @@
|
|
|
1
|
+
# OPP-014: Paired Verification Contract — expected vs observed, mechanically compared
|
|
2
|
+
|
|
3
|
+
- Status: Implemented (2026-11-01): plan declares command → expected, reports record command → observed, implementation/verification gates compare pairs
|
|
4
|
+
- Priority: P2
|
|
5
|
+
- Added: 2026-11-01
|
|
6
|
+
- Decision: extends [ADR-001](../../.context/adrs/ADR-001-odd-rdd-proportionality-scope.md) Decision A
|
|
7
|
+
- Trend signal: HDAE's gate structure pairs **Verification plan** (`command → expected result`) with **Evidence** (`command → observed result`) as first-class fields; Gentle-AI's field report showed the failure mode this prevents — agents declaring success without observed proof. AI Workflow's plan template already carries `Validation:` lines and its quality gates verify section _presence_; this contract makes the proof **mechanically comparable**.
|
|
8
|
+
|
|
9
|
+
## Problem
|
|
10
|
+
|
|
11
|
+
"Checkoffs require observed proof" is today an instruction, not a mechanism:
|
|
12
|
+
|
|
13
|
+
- the plan declares validation commands (`Validation: npm test`), but the gate cannot tell whether the recorded evidence actually ran _that_ command;
|
|
14
|
+
- expected outcomes are implicit ("tests pass") — nothing records what result was _expected_ before the run, so a wrong expectation passes as easily as a right one;
|
|
15
|
+
- evidence sections are free text — a gate can verify presence, never correspondence.
|
|
16
|
+
|
|
17
|
+
A gate that cannot compare declared vs observed is checking paperwork, not proof.
|
|
18
|
+
|
|
19
|
+
## Proposal
|
|
20
|
+
|
|
21
|
+
1. **Plan side (declaration)**: each task in the implementation plan declares its verification pair, machine-parseable:
|
|
22
|
+
- `Validation: <command> → <expected>` where expected is a minimal assertion (exit code plus optional output pattern), e.g. `npm test → exit 0`, `aiw trace --requirement=REQ-001 → contains REQ-001`.
|
|
23
|
+
2. **Evidence side (observation)**: implementation/verification artifacts record the same command with its observed result: `<command> → <observed>` (exit code and the claimed observable, e.g. `132 passed`).
|
|
24
|
+
3. **Gate comparison** (deterministic, parse-only): the verification and implementation gates check, per declared pair: the command appears verbatim in evidence, the observed section is non-empty, and expected/observed exit codes agree. Mismatch, absence, or a evidence command that was never declared → typed gate failure naming the task and pair (`verification_pair_mismatch`, `verification_pair_missing` — new reason codes, inventory updated).
|
|
25
|
+
4. **Light lane**: one pair per task is the whole evidence contract (the compact checkpoint already records command and result — formalize the same shape).
|
|
26
|
+
5. **No execution**: the gate compares declarations; it does not run project commands (preserving `verify`'s contract and keeping gates fast and side-effect-free). Execution evidence must come from a recorded run.
|
|
27
|
+
|
|
28
|
+
## Affected areas
|
|
29
|
+
|
|
30
|
+
- `resources/templates/` (plan task format gains the pair; implementation/verification report templates gain the observed side)
|
|
31
|
+
- `src/quality-gates.ts` (pair parser + comparator validators; new gate failures)
|
|
32
|
+
- `src/blocking-reasons.ts` (`verification_pair_mismatch`, `verification_pair_missing` + inventory rows)
|
|
33
|
+
- `src/workflow.ts` (gate wiring), `docs/architecture.md`, `docs/quality.md` (contract statement)
|
|
34
|
+
|
|
35
|
+
## Acceptance criteria
|
|
36
|
+
|
|
37
|
+
- Given a plan declaring `npm test → exit 0`, when evidence records `npm test → exit 0 (132 passed)`, then the gate passes.
|
|
38
|
+
- Given evidence recording a different command than declared, when the gate runs, then it fails with `verification_pair_mismatch` naming the task.
|
|
39
|
+
- Given a declared pair with no matching observed record, when the gate runs, then it fails with `verification_pair_missing`.
|
|
40
|
+
- Given artifacts authored before this contract (no pairs), when gates run, then the legacy validation format remains accepted with an advisory notice (backward-compatible parsing, no forced migration).
|
|
41
|
+
- Given a light-lane evidence entry, when completion runs, then the pair shape is the compact checkpoint's command/result fields.
|
|
42
|
+
- Given any pair failure, when `--json` is used, then the typed reason code and the failing task id are machine-readable.
|
|
43
|
+
|
|
44
|
+
## Non-goals
|
|
45
|
+
|
|
46
|
+
- The gate never executes project commands (side-effect-free gates; execution evidence comes from recorded runs).
|
|
47
|
+
- No expected/observed semantic comparison beyond exit codes and declared patterns (no model judgment in gates).
|
|
48
|
+
- No retroactive migration of existing artifacts (ratchet: new plans use pairs; legacy format accepted with notice).
|
|
49
|
+
|
|
50
|
+
## Open questions
|
|
51
|
+
|
|
52
|
+
- Expected-result grammar: exit code only, or `exit N` + substring/regex pattern? (Recommendation: exit code required, pattern optional.)
|
|
53
|
+
- Should `verify` (REQ→evidence coverage) surface pairs per requirement in the verification report?
|
|
54
|
+
- Do orchestration-delegated tasks (OPP-010) inherit pairs via their task prompt or their own declaration?
|
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
# OPP-015: Token and Recovery Governance — enforced budgets, recovery block, advisory suggestions
|
|
2
|
+
|
|
3
|
+
- Status: Implemented (2026-11-01): enforced context budgets, recovery managed block, advisory lane suggestion, regression-pinning advisory
|
|
4
|
+
- Priority: P2 (small, independent mechanisms; bundle ships together)
|
|
5
|
+
- Added: 2026-11-01
|
|
6
|
+
- Decision: extends [ADR-001](../../.context/adrs/ADR-001-odd-rdd-proportionality-scope.md) Decision A
|
|
7
|
+
- Sources: HDAE's context-recovery managed block and "do not fill gaps with assumptions" interview discipline; ODD's advisory lane guidance; AI Workflow's own CTX-004 (per-stage budgets, currently report-only) and OPP-002 (`risk assess`, already shipped).
|
|
8
|
+
|
|
9
|
+
## Problem
|
|
10
|
+
|
|
11
|
+
Four small gaps share one theme — the workflow knows the right thing but does not _act_ on it:
|
|
12
|
+
|
|
13
|
+
1. **Stage budgets are advisory**: CTX-004 measures per-stage token usage, but exceeding a budget changes nothing — there is no enforced ceiling, so drift accumulates silently.
|
|
14
|
+
2. **Post-compaction recovery is left to the agent's memory**: after context compaction or handoff, the agent re-derives state from chat remnants instead of reading the authoritative files — expensive and lossy.
|
|
15
|
+
3. **Lane choice is uninformed**: `task start` does not surface what `risk assess` already knows; humans pick lanes blind when a one-line advisory would do.
|
|
16
|
+
4. **Regression pinning is folklore**: "touched behavior keeps a test proving its previous behavior" exists in docs/quality culture but not as a declared, checkable artifact item.
|
|
17
|
+
|
|
18
|
+
## Proposal
|
|
19
|
+
|
|
20
|
+
1. **Enforced stage budgets**: stage/context manifests (per process and lane, OPP-009-ready) declare ceilings; exceeding one fails with a typed block (`context_budget_exceeded`) whose continuation names what to trim or the `--budget=<n>` override requiring an explicit reason recorded in the checkpoint. Budgets default from measured baselines (`context-bench --save`), never invented.
|
|
21
|
+
2. **Context-recovery managed block**: the root context file gains an aiw-managed block (same preserve/drift semantics as other managed content): _"After any context compaction, reset, resume, or handoff: run `aiw task status`, read the artifacts this workflow declares, and do not rely on compacted chat as the source of project state."_ Points at digests when OPP-013 exists (declared forward reference).
|
|
22
|
+
3. **Advisory lane suggestion**: `task start --lane=light` (and plain `task start`) prints, when `aiw risk assess` data is available for the current changes, one advisory line — `risk: high -> consider the full lane` or `risk: passive -> light lane viable` — always advisory; the human's choice is recorded as today. No new flags, no blocking.
|
|
23
|
+
4. **Regression-pinning declaration**: the plan template's task format gains an optional `Regression: <behavior> → <test>` line; `verify`/traceability report tasks that touched existing behavior (declared `Touches:` existing paths) without a regression declaration, as advisory output in the verification report and a documentation rule in `docs/quality.md`. Optional to declare, reported when absent — declaration, not bureaucracy.
|
|
24
|
+
|
|
25
|
+
## Affected areas
|
|
26
|
+
|
|
27
|
+
- `src/workflow.ts` (budget enforcement points, lane suggestion line), `src/context-quality.ts` / CTX-004 ceilings
|
|
28
|
+
- `src/human-approval.ts` (budget override reason recorded in checkpoints)
|
|
29
|
+
- `src/generator.ts` / `resources/templates/*` (recovery block, task `Regression:` line)
|
|
30
|
+
- `src/blocking-reasons.ts` (`context_budget_exceeded` + inventory row)
|
|
31
|
+
- `docs/quality.md`, `docs/token-efficiency.md`, `docs/architecture.md`
|
|
32
|
+
|
|
33
|
+
## Acceptance criteria
|
|
34
|
+
|
|
35
|
+
- Given a stage context exceeding its declared ceiling, when the stage runs, then it blocks with `context_budget_exceeded` and the continuation names trim options or the explicit override; the override is recorded with its reason in the checkpoint.
|
|
36
|
+
- Given `context-bench --save` baselines, when budgets are absent, then defaults derive from them (never arbitrary constants).
|
|
37
|
+
- Given the recovery block rendered for each target, when an agent resumes after compaction, then the block instructs reading `aiw task status` and workflow-declared artifacts before continuing (block present in all adapter renderings; preserved on user edits per drift rules).
|
|
38
|
+
- Given pending changes classified `high` by risk assess, when `task start --lane=light` runs, then the advisory line appears and the lane still starts as chosen (advisory never blocks).
|
|
39
|
+
- Given a task declaring `Touches: src/auth/token.ts` without a `Regression:` line, when verification runs, then the report lists the gap as advisory; declaring one removes it.
|
|
40
|
+
- Given the `npm`-published package on any target, when installed fresh, then behavior without baselines/budgets configured is byte-identical to today (all four mechanisms default-inert until declared).
|
|
41
|
+
|
|
42
|
+
## Non-goals
|
|
43
|
+
|
|
44
|
+
- No hard token ceilings on artifact authoring (authors write what the work needs; budgets govern _context served_, not writing).
|
|
45
|
+
- No automatic lane switching, budget auto-tuning, or model-driven anything (deterministic floor per spec NFR-2/3).
|
|
46
|
+
- No compaction simulator: the recovery block is an instruction artifact; testing it is OPP-011 invariant territory.
|
|
47
|
+
|
|
48
|
+
## Open questions
|
|
49
|
+
|
|
50
|
+
- Default budget derivation: per-stage from `context-bench` baseline, or per-workflow pooled?
|
|
51
|
+
- Should the lane suggestion also appear in `aiw recommend` output for ticket-driven starts?
|
|
52
|
+
- Does the `Regression:` line belong in the plan task format (OPP-014 pairs live there too) or the implementation report?
|
|
@@ -4,21 +4,36 @@ Improvement opportunities derived from the analysis of Gentle-AI's ODD/RDD metho
|
|
|
4
4
|
|
|
5
5
|
The guiding principle: **ceremony should scale with the consequence of the change, not be constant.**
|
|
6
6
|
|
|
7
|
-
| ID
|
|
8
|
-
|
|
|
9
|
-
| [OPP-001](OPP-001-proportional-ceremony-light-lane.md)
|
|
10
|
-
| [OPP-002](OPP-002-risk-assess-command.md)
|
|
11
|
-
| [OPP-003](OPP-003-adversarial-review-package.md)
|
|
12
|
-
| [OPP-004](OPP-004-exact-continuation-commands.md)
|
|
13
|
-
| [OPP-005](OPP-005-friction-benchmark.md)
|
|
14
|
-
| [OPP-006](OPP-006-emission-usability-test-policy.md)
|
|
15
|
-
| [007
|
|
16
|
-
| [008
|
|
7
|
+
| ID | Opportunity | Priority | Depends on |
|
|
8
|
+
| ------------------------------------------------------------ | ------------------------------------------------------------------------------------ | -------- | --------------------------------------- |
|
|
9
|
+
| [OPP-001](OPP-001-proportional-ceremony-light-lane.md) | Proportional ceremony — light lane for small, understood work | P2 | OPP-002 (advisory signal, optional) |
|
|
10
|
+
| [OPP-002](OPP-002-risk-assess-command.md) | `aiw risk assess` — content-based risk classification with exact next command | P2 | — |
|
|
11
|
+
| [OPP-003](OPP-003-adversarial-review-package.md) | Adversarial review package — blind judges and proof-carrying findings | P3 | — |
|
|
12
|
+
| [OPP-004](OPP-004-exact-continuation-commands.md) | Exact continuation commands — every block names a runnable way out | P1 | — |
|
|
13
|
+
| [OPP-005](OPP-005-friction-benchmark.md) | Friction benchmark — measure how users get stuck (in-band / out-of-band / dead-end) | P3 | OPP-004 |
|
|
14
|
+
| [OPP-006](OPP-006-emission-usability-test-policy.md) | Emission ≠ usability — test that named commands actually work | P2 | OPP-004 (shared contract) |
|
|
15
|
+
| [OPP-007](OPP-007-decision-findings-triage-scope-watch.md) | Decision model — findings-evidence triage and lane scope-watch (advisory-only) | P3 | OPP-001, OPP-003 (integration points) |
|
|
16
|
+
| [OPP-008](OPP-008-embedding-covering-tests-judge-context.md) | Embedding retrieval — covering-test detection and judge context scoping | P3 | OPP-002, OPP-003 (integration points) |
|
|
17
|
+
| [OPP-009](OPP-009-process-entry-points.md) | Processes as entry points — bug-fixing and idea assessment alongside SDD | P2 | OPP-001 (lanes), OPP-004 (reason codes) |
|
|
18
|
+
| [OPP-010](OPP-010-orchestration-worktree-isolation.md) | Worktree isolation for orchestrated execution | P2 | orchestration (ENT-007) |
|
|
19
|
+
| [OPP-011](OPP-011-evals-redteam-generated-artifacts.md) | Evals + red-team layer for generated artifacts | P3 | OPP-006 (ratchet) |
|
|
20
|
+
| [OPP-012](OPP-012-standards-discovery.md) | Standards discovery — codebase conventions into versioned standards | P3 | OPP-008 (retrieval) |
|
|
21
|
+
| [OPP-013](OPP-013-read-optimized-artifacts-digest-layer.md) | Read-Optimized Artifacts — fingerprint-bound digest layer for downstream reads | P1 | OPP-004 (typed blocks), CTX-002/004 |
|
|
22
|
+
| [OPP-014](OPP-014-paired-verification-contract.md) | Paired Verification Contract — expected vs observed, mechanically compared | P2 | OPP-004 (reason codes) |
|
|
23
|
+
| [OPP-015](OPP-015-token-recovery-governance.md) | Token & Recovery Governance — enforced budgets, recovery block, advisory suggestions | P2 | CTX-004, OPP-002 (advisory signal) |
|
|
17
24
|
|
|
18
25
|
Suggested starting order: **OPP-004 → OPP-002 → OPP-006 → OPP-001 → OPP-005 → OPP-003** (smallest/least invasive first; OPP-004's reason codes feed OPP-005 and OPP-006).
|
|
19
26
|
|
|
20
27
|
OPP-007 and OPP-008 are **model-integration addenda**: deliberately left for after field exposure — their priority should be set by real usage pain, not speculation. All model features respect the spec's NFR contracts: deterministic floor first, models advisory-only, unconfigured = byte-identical offline behavior.
|
|
21
28
|
|
|
29
|
+
## Trend-sourced addendum (2026-11)
|
|
30
|
+
|
|
31
|
+
OPP-009 through OPP-012 derive from a November 2026 landscape scan (spec-kit 145K★ shipping independent entry points; Spec Kitty / Conductor normalizing isolated-worktree parallel agents; promptfoo 26K★ establishing artifact evals; Agent OS 5.5K★ on standards discovery). The scan also **validated** the existing positioning: spec-driven development and human-in-the-loop governance are growing markets where AI Workflow's evidence ledger is the differentiator. Deliberately not chased: generic memory layers (audit-first philosophy), cloud-agent infrastructure (harness territory), commercial AI-review feature parity.
|
|
32
|
+
|
|
33
|
+
### Token-analysis addendum (OPP-013..015)
|
|
34
|
+
|
|
35
|
+
Measured on this repository's own full-lane artifacts (~2.2K-token specification, ~4.1K-token plan): downstream re-reads cost ~25-30K tokens per feature — roughly 80% of an artifact's lifetime token cost is re-reading, not writing. OPP-013 (digests), OPP-014 (paired verification) and OPP-015 (governance) attack that multiplier and make observed proof mechanically checkable, synthesizing the shared insight across spec-kit, ODD and HDAE: the full artifact is for humans; agents need a derived, verifiable view.
|
|
36
|
+
|
|
22
37
|
## Planning
|
|
23
38
|
|
|
24
39
|
The implementation is planned as six waves in the project's own SDD format:
|
package/package.json
CHANGED
|
@@ -7,6 +7,6 @@ description: Break an approved design into independently verifiable implementati
|
|
|
7
7
|
|
|
8
8
|
For every approval, rejection, or optional skip, use the host's native blocking question widget, not a prose-only prompt when the widget is available. Leave choices unselected and offer “Approve and continue”, “Request changes”, and “Reject/stop”; for a skip, offer “Accept skip” or “Do not skip”. Wait for an actual submission before running `aiw approve`, `aiw reject`, or `aiw skip` or proceeding. Collect revision/skip reasons in a follow-up free-text widget. If the host has no blocking widget, use a numbered chat prompt and end the turn until an explicit reply arrives. The CLI records the decision but cannot verify that a widget was used.
|
|
9
9
|
|
|
10
|
-
Create small ordered tasks. Each task must reference requirements, affected modules, tests, commands, dependencies, and completion evidence.
|
|
10
|
+
Create small ordered tasks. Each task must reference requirements, affected modules, tests, commands, dependencies, and completion evidence. Each task that touches existing behavior declares a regression line (`Regression: <behavior> → <test>`) pinning a test that proves the previous behavior still holds; `aiw verify` reports tasks missing it as advisory.
|
|
11
11
|
|
|
12
12
|
Write `.aiw/generated/plans/implementation-plan.md` with ordered tasks, requirement links, affected modules, tests, validation, dependencies, risks, and completion evidence. After `aiw gate plan` passes, present the exact plan to the human and ask for explicit approval. Stop and wait; a gate pass is not approval. Do not edit implementation files until approval arrives and `aiw approve plan` succeeds. If changes are requested, first record `aiw reject plan --reason=<feedback>`, then revise the plan, rerun its gate, and ask again. After approval, implement in task order. Use `tdd-development` when installed and the project has a test workflow; otherwise use existing validation. When implementation is complete, write `.aiw/generated/reports/implementation-report.md` with changed files, tests, validation results, and deviations; run `aiw gate implementation`, present the diff and evidence to the human, stop, and wait for explicit approval. Record approval with `aiw approve implementation` before verification. Invoke CLI commands through `npx --yes --package=@multileaf/ai-workflow -- aiw <command>` when no global binary exists.
|