@kontourai/flow-agents 3.6.0 → 3.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +7 -0
- package/build/src/builder-flow-run-adapter.d.ts +22 -2
- package/build/src/builder-flow-run-adapter.js +93 -28
- package/build/src/builder-flow-runtime.d.ts +8 -3
- package/build/src/builder-flow-runtime.js +308 -47
- package/build/src/cli/assignment-provider.d.ts +4 -7
- package/build/src/cli/assignment-provider.js +67 -13
- package/build/src/cli/builder-run.js +14 -18
- package/build/src/cli/workflow-sidecar.d.ts +28 -4
- package/build/src/cli/workflow-sidecar.js +801 -97
- package/build/src/cli/workflow.js +290 -42
- package/build/src/flow-kit/validate.d.ts +17 -0
- package/build/src/flow-kit/validate.js +340 -2
- package/build/src/index.js +5 -5
- package/build/src/lib/observed-command.d.ts +7 -0
- package/build/src/lib/observed-command.js +100 -0
- package/build/src/tools/generate-context-map.js +5 -3
- package/context/scripts/hooks/lib/kit-catalog.js +1 -1
- package/context/scripts/hooks/stop-goal-fit.js +78 -9
- package/docs/context-map.md +22 -20
- package/docs/developer-architecture.md +1 -1
- package/docs/public-workflow-cli.md +76 -7
- package/docs/skills-map.md +51 -27
- package/docs/spec/builder-flow-runtime.md +41 -40
- package/docs/workflow-usage-guide.md +109 -42
- package/evals/fixtures/hook-influence/cases.json +2 -2
- package/evals/integration/test_builder_entry_enforcement.sh +52 -41
- package/evals/integration/test_builder_step_producers.sh +297 -6
- package/evals/integration/test_bundle_install.sh +212 -63
- package/evals/integration/test_critique_supersession_roundtrip.sh +21 -8
- package/evals/integration/test_current_json_per_actor.sh +12 -0
- package/evals/integration/test_dual_emit_flow_step.sh +62 -43
- package/evals/integration/test_flowdef_session_activation.sh +145 -25
- package/evals/integration/test_flowdef_session_history_preservation.sh +23 -21
- package/evals/integration/test_gate_lockdown.sh +3 -5
- package/evals/integration/test_goal_fit_hook.sh +60 -2
- package/evals/integration/test_liveness_verdict.sh +14 -17
- package/evals/integration/test_phase_map_and_gate_claim.sh +63 -38
- package/evals/integration/test_public_workflow_cli.sh +325 -11
- package/evals/integration/test_pull_work_liveness_preflight.sh +22 -66
- package/evals/integration/test_sidecar_field_preservation.sh +36 -11
- package/evals/integration/test_workflow_sidecar_writer.sh +277 -44
- package/evals/integration/test_workflow_steering_hook.sh +15 -38
- package/evals/run.sh +2 -0
- package/evals/static/test_builder_skill_coherence.sh +247 -0
- package/evals/static/test_library_exports.sh +5 -2
- package/evals/static/test_workflow_skills.sh +13 -325
- package/kits/builder/flows/build.flow.json +22 -0
- package/kits/builder/flows/shape.flow.json +9 -9
- package/kits/builder/kit.json +70 -16
- package/kits/builder/skills/builder-shape/SKILL.md +75 -75
- package/kits/builder/skills/continue-work/SKILL.md +45 -106
- package/kits/builder/skills/deliver/SKILL.md +96 -442
- package/kits/builder/skills/design-probe/SKILL.md +40 -139
- package/kits/builder/skills/evidence-gate/SKILL.md +59 -201
- package/kits/builder/skills/execute-plan/SKILL.md +54 -125
- package/kits/builder/skills/fix-bug/SKILL.md +42 -132
- package/kits/builder/skills/gate-review/SKILL.md +60 -211
- package/kits/builder/skills/idea-to-backlog/SKILL.md +63 -244
- package/kits/builder/skills/learning-review/SKILL.md +63 -170
- package/kits/builder/skills/pickup-probe/SKILL.md +54 -111
- package/kits/builder/skills/plan-work/SKILL.md +51 -185
- package/kits/builder/skills/pull-work/SKILL.md +136 -485
- package/kits/builder/skills/release-readiness/SKILL.md +66 -107
- package/kits/builder/skills/review-work/SKILL.md +89 -176
- package/kits/builder/skills/tdd-workflow/SKILL.md +53 -147
- package/kits/builder/skills/verify-work/SKILL.md +101 -113
- package/package.json +2 -2
- package/scripts/hooks/lib/kit-catalog.js +1 -1
- package/scripts/hooks/stop-goal-fit.js +78 -9
- package/src/builder-flow-run-adapter.ts +118 -32
- package/src/builder-flow-runtime.ts +331 -61
- package/src/cli/assignment-provider-lock.test.mjs +83 -0
- package/src/cli/assignment-provider.ts +62 -13
- package/src/cli/builder-flow-run-adapter.test.mjs +4 -2
- package/src/cli/builder-flow-runtime.test.mjs +194 -8
- package/src/cli/builder-run.ts +3 -9
- package/src/cli/kit-metadata-security.test.mjs +232 -6
- package/src/cli/public-api.test.mjs +15 -0
- package/src/cli/workflow-sidecar-execution-proof.test.mjs +90 -0
- package/src/cli/workflow-sidecar.ts +724 -97
- package/src/cli/workflow.ts +277 -40
- package/src/flow-kit/validate.ts +320 -2
- package/src/index.ts +6 -5
- package/src/lib/observed-command.ts +96 -0
- package/src/tools/generate-context-map.ts +5 -3
|
@@ -1,277 +1,96 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: "idea-to-backlog"
|
|
3
|
-
description: "Turn raw product or technical ideas into shaped, prioritized,
|
|
3
|
+
description: "Turn raw product or technical ideas into a shaped, prioritized, provider-neutral executable backlog before implementation starts."
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# Idea To Backlog
|
|
7
7
|
|
|
8
|
-
Convert raw ideas into shaped, prioritized, executable
|
|
8
|
+
Convert raw ideas into shaped, prioritized, executable work without starting implementation.
|
|
9
9
|
|
|
10
|
-
##
|
|
11
|
-
|
|
12
|
-
- Produce durable reasoning artifacts before creating execution issues.
|
|
13
|
-
- Own the human-readable backlog issue contract; downstream delivery workflows inherit and refine it rather than inventing a new shape.
|
|
14
|
-
- Create GitHub issues only for committed or near-committed work.
|
|
15
|
-
- Do not write production code or edit implementation files.
|
|
16
|
-
- Do not invoke downstream delivery skills while shaping upstream work.
|
|
17
|
-
- After the backlog gate, hand off to `pull-work` only if the user explicitly asks to continue.
|
|
18
|
-
- Never invoke `plan-work`, `execute-plan`, `review-work`, `verify-work`, `evidence-gate`, or release skills from this workflow.
|
|
19
|
-
- Stop at gates unless the user explicitly asks to continue.
|
|
20
|
-
- Treat GitHub issues as executable backlog, not the whole reasoning store.
|
|
21
|
-
- Keep separate ideas separate until a shared outcome, hard dependency, or sequencing reason justifies bundling them.
|
|
22
|
-
- Push back when the user blends unrelated ideas, and ask them to justify why the ideas belong together before shaping bundled work.
|
|
23
|
-
|
|
24
|
-
## Model Routing
|
|
25
|
-
|
|
26
|
-
When this skill delegates its shaping work (opportunity review, option
|
|
27
|
-
exploration, slicing), resolve the `delegate-design` role from
|
|
28
|
-
`.datum/config.json` (`npx @kontourai/datum resolve delegate-design --json`) and
|
|
29
|
-
pass the resolved model explicitly — shaping a raw idea into a backlog needs
|
|
30
|
-
design latitude, so it routes to the design tier. See
|
|
31
|
-
`context/contracts/execution-contract.md` § Delegation: Model Routing. Fallback:
|
|
32
|
-
inherit the session model when datum/config is absent, noted in the artifact.
|
|
33
|
-
|
|
34
|
-
## Artifact Contract
|
|
35
|
-
|
|
36
|
-
Create or update `.kontourai/flow-agents/<slug>/<slug>--idea-to-backlog.md` with:
|
|
37
|
-
|
|
38
|
-
- `source_ideas`: raw inputs and dedupe links
|
|
39
|
-
- `idea_inventory`: one record per distinct idea, classification, outcome, and reason
|
|
40
|
-
- `slice_candidates`: thinnest meaningful slice for each buildable idea, success signal, and non-goals
|
|
41
|
-
- `bundle_justification`: why grouped ideas belong together, or explicit decision to split
|
|
42
|
-
- `dependency_map`: blocks / blocked-by / related-only relationships between ideas or slices
|
|
43
|
-
- `phase`: intake, opportunity, explore, shape, prioritize, backlog, blocked, complete
|
|
44
|
-
- `decisions`: decision, rationale, decision maker, date
|
|
45
|
-
- `opportunity_briefs`: problem, stakeholder, outcome, confidence, size
|
|
46
|
-
- `shaped_work`: scope, non-goals, requirements, acceptance criteria
|
|
47
|
-
- `risk_release_notes`: risk class, rollout, rollback, observability
|
|
48
|
-
- `backlog_links`: GitHub issue URLs and status
|
|
49
|
-
- `parked_or_rejected`: reason and revisit trigger
|
|
50
|
-
- `open_questions`: owner and needed evidence
|
|
51
|
-
- `next_gate`: gate name and pass/fail/block status
|
|
52
|
-
|
|
53
|
-
## Workflow
|
|
54
|
-
|
|
55
|
-
### 1. Intake
|
|
56
|
-
|
|
57
|
-
Normalize and deduplicate raw ideas.
|
|
58
|
-
|
|
59
|
-
Classify each item as:
|
|
60
|
-
|
|
61
|
-
- feature
|
|
62
|
-
- bug
|
|
63
|
-
- research question
|
|
64
|
-
- spike
|
|
65
|
-
- prototype
|
|
66
|
-
- chore
|
|
67
|
-
- cleanup
|
|
68
|
-
- parked thought
|
|
69
|
-
|
|
70
|
-
Outcome: `discard`, `park`, `merge`, `research`, `shape`, or `commit`.
|
|
71
|
-
|
|
72
|
-
Gate: every idea has exactly one outcome and a recorded reason.
|
|
73
|
-
|
|
74
|
-
### 2. Separate Ideas And Slice
|
|
75
|
-
|
|
76
|
-
Create an idea inventory before shaping. Treat each distinct user problem, workflow, stakeholder, risk, or success metric as a separate idea unless proven otherwise.
|
|
77
|
-
|
|
78
|
-
When multiple ideas arrive together:
|
|
79
|
-
|
|
80
|
-
- name each idea separately
|
|
81
|
-
- identify the thinnest meaningful slice for each buildable idea
|
|
82
|
-
- challenge accidental bundles
|
|
83
|
-
- require a bundle justification before shaping grouped work
|
|
84
|
-
- distinguish hard dependencies from "related-only" context
|
|
85
|
-
|
|
86
|
-
A thinnest meaningful slice is the smallest independently valuable and testable unit that can reach evidence/release gates without leaving dependent half-work behind.
|
|
87
|
-
|
|
88
|
-
Bundled work is allowed only when at least one is true:
|
|
10
|
+
## Role And Binding
|
|
89
11
|
|
|
90
|
-
-
|
|
91
|
-
-
|
|
92
|
-
- the
|
|
93
|
-
-
|
|
12
|
+
- **Role:** canonical Builder shape-step producer.
|
|
13
|
+
- **Binding:** `builder.shape` steps `shape`, `breakdown`, and `file-issues`.
|
|
14
|
+
- **Produces:** the `trust.bundle` shaping slices for an active matching run: `shaped-problem`, `shaped-outcome`, `shaped-constraints`, `shaped-non-goals`, `shaped-success`, `shaped-risk`, optional `open-decisions`, `slices-defined`, and `work-items-filed`.
|
|
15
|
+
- **Standalone no-run behavior:** create a local shaping artifact or return the shaped backlog to the caller. Do not start a Builder run, record Builder evidence, or imply that any Builder gate passed.
|
|
94
16
|
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
Gate: every shaped candidate has one outcome, one thinnest meaningful slice, and either a split decision or a recorded bundle justification and dependency map.
|
|
98
|
-
|
|
99
|
-
### 3. Opportunity Review
|
|
100
|
-
|
|
101
|
-
For ideas worth shaping, capture:
|
|
102
|
-
|
|
103
|
-
- product/theme goal
|
|
104
|
-
- user or stakeholder
|
|
105
|
-
- problem / opportunity
|
|
106
|
-
- expected outcome
|
|
107
|
-
- confidence
|
|
108
|
-
- rough investment size
|
|
109
|
-
- tradeoffs and what this displaces
|
|
110
|
-
|
|
111
|
-
Gate: the opportunity is worth shaping, or it is parked/rejected.
|
|
112
|
-
|
|
113
|
-
### 4. Explore Options
|
|
114
|
-
|
|
115
|
-
Use `search-first` or `explore` when context is missing.
|
|
116
|
-
|
|
117
|
-
Decide the path:
|
|
118
|
-
|
|
119
|
-
- `shape-work`: enough is known to define the feature.
|
|
120
|
-
- `spike-research`: unknowns need investigation, no production code.
|
|
121
|
-
- `prototype-mvp`: visual or interaction proof is needed in an isolated worktree.
|
|
122
|
-
- `reject/park`: not worth current backlog.
|
|
123
|
-
|
|
124
|
-
For spikes and prototypes, define the learning question, timebox, artifact, and cleanup expectation. Prototype work must be isolated and must not silently become production implementation.
|
|
125
|
-
|
|
126
|
-
### 5. Shape Work
|
|
127
|
-
|
|
128
|
-
Create a brief with:
|
|
17
|
+
## Contract
|
|
129
18
|
|
|
130
|
-
-
|
|
131
|
-
-
|
|
132
|
-
-
|
|
133
|
-
-
|
|
134
|
-
-
|
|
135
|
-
- requirements with stable ids, such as `R1`, `R2`
|
|
136
|
-
- UX/API implications
|
|
137
|
-
- acceptance criteria with stable ids, such as `AC1`, `AC2`; criteria must be testable and map back to requirements
|
|
138
|
-
- verification expectations that preserve AC ids and name expected command/test evidence plus source evidence/permalink expectations when implementation behavior is claimed
|
|
139
|
-
- risk class
|
|
140
|
-
- rollout / rollback notes
|
|
141
|
-
- observability or release concerns
|
|
142
|
-
- open questions
|
|
19
|
+
- Produce durable reasoning before creating executable work items.
|
|
20
|
+
- Do not write production code or invoke downstream build, review, verification, or release skills.
|
|
21
|
+
- Keep distinct ideas separate unless a shared outcome, hard dependency, or sequencing reason justifies a bundle.
|
|
22
|
+
- Stop after shaping unless the user explicitly asks to continue with another workflow.
|
|
23
|
+
- Treat provider work items as an execution queue, not as the sole reasoning store.
|
|
143
24
|
|
|
144
|
-
|
|
25
|
+
## Provider Boundaries
|
|
145
26
|
|
|
146
|
-
|
|
27
|
+
Use configured capabilities rather than provider-specific commands or data shapes:
|
|
147
28
|
|
|
148
|
-
- `
|
|
149
|
-
- `
|
|
150
|
-
- `
|
|
151
|
-
- `
|
|
152
|
-
- `Requirements` with stable `R*` ids
|
|
153
|
-
- `Acceptance Criteria` with stable `AC*` ids
|
|
154
|
-
- `Verification Expectation`
|
|
155
|
-
- `Acceptance Evidence` expectation: closure/provider comments must map AC ids to status, command/test evidence, source evidence/permalinks, and gaps
|
|
156
|
-
- `Milestone / Delivery Outcome`
|
|
157
|
-
- `Dependencies / Blockers`
|
|
158
|
-
- `Source Artifact`
|
|
29
|
+
- `BoardProvider` supplies candidate queue, priority, milestone, and board state.
|
|
30
|
+
- `WorkItemProvider` creates or updates executable work items and records their references.
|
|
31
|
+
- `RepositoryAdapter` supplies repository identity, target revision, and source context.
|
|
32
|
+
- `AssignmentProvider` may provide ownership context; shaping does not claim work.
|
|
159
33
|
|
|
160
|
-
|
|
34
|
+
GitHub is only an optional adapter example. Do not require GitHub issues, Projects, labels, milestones, URLs, or `gh` for this skill to work.
|
|
161
35
|
|
|
162
|
-
|
|
36
|
+
## Model Routing
|
|
163
37
|
|
|
164
|
-
|
|
38
|
+
For delegated opportunity review, option exploration, or slicing, resolve `delegate-design` from `.datum/config.json`. If that mapping is unavailable, inherit the session model and record the fallback in the artifact.
|
|
39
|
+
Apply the routing and escalation contract in `context/contracts/execution-contract.md`.
|
|
165
40
|
|
|
166
|
-
|
|
167
|
-
- why now
|
|
168
|
-
- expected outcome
|
|
169
|
-
- confidence
|
|
170
|
-
- cost / size
|
|
171
|
-
- risks
|
|
172
|
-
- dependencies
|
|
173
|
-
- alternatives
|
|
174
|
-
- human decision
|
|
41
|
+
## Artifact
|
|
175
42
|
|
|
176
|
-
|
|
43
|
+
Create or update `<slug>--idea-to-backlog.md` with:
|
|
177
44
|
|
|
178
|
-
|
|
45
|
+
- source ideas, deduplication decisions, and an inventory of distinct ideas
|
|
46
|
+
- the thinnest meaningful slice, outcome, non-goals, and bundle justification for each buildable idea
|
|
47
|
+
- dependency map, opportunity briefs, options considered, and priority recommendation
|
|
48
|
+
- requirements and acceptance criteria with stable `R*` and `AC*` identifiers
|
|
49
|
+
- risk, rollout, rollback, observability, open-question, parked, and rejected-work records
|
|
50
|
+
- created work-item references and the selected provider/adapter capability gaps
|
|
51
|
+
- source revision assumptions: target ref/SHA when available, shaping time, and relevant scope references
|
|
179
52
|
|
|
180
|
-
|
|
53
|
+
## Shape
|
|
181
54
|
|
|
182
|
-
|
|
55
|
+
1. Normalize and deduplicate the inputs. Classify each as a feature, bug, research question, spike, prototype, chore, cleanup, or parked thought. Give every idea one outcome: discard, park, merge, research, shape, or commit.
|
|
56
|
+
2. Separate ideas before slicing. Name the user problem, stakeholder, risk, and success signal for each idea. Require a recorded split decision or bundle justification.
|
|
57
|
+
3. Review opportunity: product goal, affected user or operator, expected outcome, confidence, investment, tradeoffs, and what the work displaces.
|
|
58
|
+
4. Choose the path: shape work, time-boxed research, isolated prototype, or park/reject. A spike or prototype must state its learning question, artifact, cleanup expectation, and why it is not production implementation.
|
|
59
|
+
5. Shape the work with a problem statement, scope, non-goals, requirements, testable acceptance criteria, UX/API implications, verification expectations, risks, rollout and rollback notes, observability, and open questions.
|
|
60
|
+
6. Prioritize explicitly: do, defer, or reject; why now; expected outcome; confidence; size; risk; dependencies; alternatives; and decision owner.
|
|
61
|
+
7. File only committed or near-committed slices through `WorkItemProvider`. Preserve the readable brief and provider-neutral metadata needed for later selection: dependencies, blockers, source revisions, planning scope references, and the shaping artifact reference.
|
|
183
62
|
|
|
184
|
-
|
|
185
|
-
- problem
|
|
186
|
-
- scope
|
|
187
|
-
- non-goals
|
|
188
|
-
- requirements with stable `R*` ids
|
|
189
|
-
- acceptance criteria
|
|
190
|
-
- links to source brief / session artifact
|
|
191
|
-
- priority rationale
|
|
192
|
-
- milestone or milestone decision: assign a milestone when the work contributes to a named delivery outcome, or record why no milestone is appropriate
|
|
193
|
-
- expected size
|
|
194
|
-
- thinnest meaningful slice
|
|
195
|
-
- dependencies / blockers
|
|
196
|
-
- bundle justification if grouped with other issues
|
|
197
|
-
- verification expectation
|
|
198
|
-
- release/evidence expectation
|
|
199
|
-
- acceptance evidence expectation, including source evidence refs for behavior claims and immutable GitHub blob permalinks pinned to commit SHA when a provider change is available
|
|
200
|
-
- owner or assignee when known
|
|
63
|
+
A thinnest meaningful slice is independently valuable and testable. Bundle only when the same outcome requires all included work, a hard dependency makes sequencing necessary, or grouping demonstrably reduces delivery risk.
|
|
201
64
|
|
|
202
|
-
|
|
65
|
+
## Work Item Contract
|
|
203
66
|
|
|
204
|
-
|
|
67
|
+
Each filed work item should include:
|
|
205
68
|
|
|
206
|
-
-
|
|
207
|
-
- `
|
|
208
|
-
-
|
|
209
|
-
-
|
|
210
|
-
-
|
|
69
|
+
- story or operator outcome, problem, scope, and non-goals
|
|
70
|
+
- stable `R*` requirements and `AC*` acceptance criteria
|
|
71
|
+
- verification expectation and a future acceptance-evidence table: `AC id`, `Status`, `Command/Test Evidence`, `Source Evidence`, and `Gaps`
|
|
72
|
+
- priority rationale, expected size, dependencies/blockers, milestone or delivery outcome decision, and source artifact reference
|
|
73
|
+
- source revision groups with `planned_base_ref`, `planned_base_sha`, `planned_at`, and `planning_scope_refs` when the `RepositoryAdapter` can supply them
|
|
211
74
|
|
|
212
|
-
|
|
75
|
+
Use an adapter-neutral structured metadata attachment or field when supported. Native dependency, project, milestone, assignee, and custom-field features remain optional adapter enhancements.
|
|
213
76
|
|
|
214
|
-
|
|
77
|
+
## Active Builder Evidence
|
|
215
78
|
|
|
216
|
-
|
|
79
|
+
Only when an active `builder.shape` run is already bound to this work, inspect
|
|
80
|
+
its current state with the public workflow status command. Record each completed
|
|
81
|
+
expectation with the public workflow evidence command; use reviewable shaping
|
|
82
|
+
references and never an internal writer:
|
|
217
83
|
|
|
218
|
-
```
|
|
219
|
-
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
"
|
|
223
|
-
|
|
224
|
-
"repo": "owner/repo",
|
|
225
|
-
"planned_base_ref": "main",
|
|
226
|
-
"planned_base_sha": "0123456789abcdef0123456789abcdef01234567",
|
|
227
|
-
"planned_at": "2026-06-03T03:23:14Z",
|
|
228
|
-
"planning_artifact_ref": ".flow-agents/example/example--idea-to-backlog.md",
|
|
229
|
-
"planning_scope_refs": [
|
|
230
|
-
"kits/builder/skills/idea-to-backlog/SKILL.md",
|
|
231
|
-
"context/contracts/work-item-contract.md"
|
|
232
|
-
]
|
|
233
|
-
}
|
|
234
|
-
],
|
|
235
|
-
"blockers": [
|
|
236
|
-
{
|
|
237
|
-
"type": "work_item",
|
|
238
|
-
"ref": "provider://work-items/123",
|
|
239
|
-
"status": "blocked",
|
|
240
|
-
"summary": "Needs the upstream contract issue to land first."
|
|
241
|
-
},
|
|
242
|
-
{
|
|
243
|
-
"type": "text",
|
|
244
|
-
"status": "blocked",
|
|
245
|
-
"summary": "Needs product decision on rollout scope."
|
|
246
|
-
}
|
|
247
|
-
]
|
|
248
|
-
}
|
|
249
|
-
-->
|
|
84
|
+
```bash
|
|
85
|
+
flow-agents workflow status --session-dir <session-dir>
|
|
86
|
+
flow-agents workflow evidence --session-dir <session-dir> \
|
|
87
|
+
--expectation <expectation-id> --status pass \
|
|
88
|
+
--summary "Shaped work and filed Work Item evidence are recorded." \
|
|
89
|
+
--evidence-ref-json '{"kind":"artifact","file":"<session-dir>/<slug>--idea-to-backlog.md","summary":"Reviewable shaping report with slices and filed Work Item links."}'
|
|
250
90
|
```
|
|
251
91
|
|
|
252
|
-
|
|
253
|
-
|
|
254
|
-
Avoid creating tiny implementation tasks too early; `plan-work` can decompose inside an execution artifact later.
|
|
255
|
-
|
|
256
|
-
Gate: every created issue is ready for `pull-work` or explicitly marked blocked/needs shaping.
|
|
257
|
-
|
|
258
|
-
Backlog issue bodies should not require live provider calls to create permalinks during shaping. They should require downstream closure comments to include an `Acceptance Evidence` table with columns `AC id`, `Status`, `Command/Test Evidence`, `Source Evidence / Permalinks`, and `Gaps`; source evidence can start as local file/line refs during verification and should be upgraded to immutable provider permalinks before closure when available.
|
|
259
|
-
|
|
260
|
-
When a GitHub Project is used, decide whether milestones are represented as repo milestones, project fields, issue labels, or intentionally omitted. Record the chosen milestone strategy in the artifact and apply it consistently to created issues/project items when the provider supports it.
|
|
261
|
-
|
|
262
|
-
## Gates
|
|
263
|
-
|
|
264
|
-
- Idea Gate: deduped, classified, and either discarded, parked, merged, researched, shaped, or committed.
|
|
265
|
-
- Slice Gate: every candidate has one outcome, one thinnest meaningful slice, and explicit split/bundle/dependency reasoning.
|
|
266
|
-
- Shape Gate: brief is coherent and acceptance criteria are stable enough.
|
|
267
|
-
- Priority Gate: priority and tradeoff are explicit.
|
|
268
|
-
- Backlog Gate: GitHub issues are ready for `pull-work`.
|
|
269
|
-
|
|
270
|
-
If a gate fails, update the artifact with the missing evidence or decision and stop.
|
|
92
|
+
At `shape`, record the six required shape expectations and `open-decisions` when applicable. At `breakdown`, record `slices-defined`. At `file-issues`, record `work-items-filed`. Use `fail` or `not_verified` when the artifact cannot support the expectation; never convert an accepted uncertainty into a passing claim.
|
|
271
93
|
|
|
272
|
-
##
|
|
94
|
+
## Stop
|
|
273
95
|
|
|
274
|
-
-
|
|
275
|
-
- Use an inbox, opportunity store, shaped queue, executable backlog, and archive.
|
|
276
|
-
- Parked ideas need a reason and revisit trigger.
|
|
277
|
-
- Stale issues should be closed, archived, or returned to discovery.
|
|
96
|
+
Stop when the selected path is shaped and either filed as executable work, intentionally parked, or explicitly rejected. Hand off to `pull-work` only when the user asks to select work for delivery.
|
|
@@ -1,196 +1,89 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: "learning-review"
|
|
3
|
-
description: "Capture
|
|
3
|
+
description: "Capture delivery decisions, outcomes, and routed follow-up. Records Builder publish-learn learning evidence in trust.bundle."
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# Learning Review
|
|
7
7
|
|
|
8
|
-
|
|
8
|
+
## Role
|
|
9
9
|
|
|
10
|
-
|
|
10
|
+
This is a Builder step skill.
|
|
11
11
|
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
- Feed actionable changes back to `idea-to-backlog`, GitHub issues, tests, docs, or knowledge.
|
|
16
|
-
- Use `knowledge-capture` or `observe` when the learning should persist beyond the repo.
|
|
17
|
-
- Compare long-lived docs against the local `.kontourai/flow-agents/<slug>/` plan and the final acceptance artifact so implementation intent is not lost after merge.
|
|
18
|
-
- Treat `learning-review` as the terminal closeout decision point for correction telemetry. Compare intended behavior to observed behavior before writing `learning.json`, then record either `correction.needed: false` for a clean run or `correction.needed: true` for a mismatch.
|
|
12
|
+
Learning Review records what the delivery outcome taught us and where any
|
|
13
|
+
correction belongs. It does not rewrite history, implement fixes, or declare a
|
|
14
|
+
failed outcome successful.
|
|
19
15
|
|
|
20
|
-
##
|
|
21
|
-
|
|
22
|
-
- Release-readiness artifact, evidence-gate artifact, PR/issue links, deploy notes, incidents, telemetry, user feedback, and reviewer/verifier notes.
|
|
23
|
-
- **Delegation routing telemetry** — the per-run economics records (`.kontourai/telemetry/economics.jsonl`) carry `delegations[]` with each sub-agent's `(role, resolved_model, outcome)`. Feed them to the routing-efficiency review (step 2a) to judge whether the model each role routes to is actually efficient.
|
|
24
|
-
- **Kit/gate economics proposal ledger** — the same economics records, turned into a durable, idempotent per-kit/per-gate proposal ledger (`.kontourai/telemetry/learning-review-proposals.jsonl`) by `scripts/telemetry/learning-review-proposals.sh`. Feed it to the kit/gate economics review (step 2b).
|
|
25
|
-
|
|
26
|
-
## Artifact Contract
|
|
27
|
-
|
|
28
|
-
Create or update `.kontourai/flow-agents/<slug>/<slug>--learning-review.md` with:
|
|
29
|
-
|
|
30
|
-
- `scope`: delivered work, issue/PR/release links, dates
|
|
31
|
-
- `outcomes`: expected vs observed behavior and signals
|
|
32
|
-
- `evidence`: telemetry, support notes, incidents, CI/review patterns
|
|
33
|
-
- `decisions`: what changed, why, and who decided
|
|
34
|
-
- `gaps`: process, tests, docs, skill, tooling, product, or ownership gaps
|
|
35
|
-
- `followups`: GitHub issues, idea-to-backlog items, docs/tests/eval work
|
|
36
|
-
- `knowledge_updates`: notes captured or observations proposed
|
|
37
|
-
- `docs_promotion`: durable docs updated, source artifacts archived/linked, and gaps routed
|
|
38
|
-
- `runtime_artifact_cleanup`: `.flow-agents/` runtime artifacts remain untracked or a blocker is recorded
|
|
39
|
-
- `verdict`: LEARNED, FOLLOWUP_REQUIRED, or BLOCKED
|
|
40
|
-
|
|
41
|
-
When the repository provides `npm run workflow:sidecar --`, also write `learning.json` with:
|
|
42
|
-
|
|
43
|
-
```bash
|
|
44
|
-
npm run workflow:sidecar -- record-learning .kontourai/flow-agents/<slug> \
|
|
45
|
-
--status learned \
|
|
46
|
-
--record-json '{"id":"...","source_refs":["release.json"],"outcome":"success","facts":["..."],"interpretation":"...","routing":[{"target":"none","action":"No follow-up required after intended-vs-observed closeout.","status":"completed"}],"correction":{"needed":false,"evidence":"Acceptance, release, docs promotion, and learning closeout behaved as intended."}}' \
|
|
47
|
-
--summary "..."
|
|
48
|
-
```
|
|
49
|
-
|
|
50
|
-
Use `followup_required` or `blocked` when facts are captured but follow-up routing is still open.
|
|
51
|
-
|
|
52
|
-
Clean terminal runs must record a lightweight no-correction-needed record, not an invented lesson. Use `correction.needed: false`, a brief `correction.evidence` summary, and closed routing such as `target: "none"` with `status: "completed"`.
|
|
53
|
-
|
|
54
|
-
Mismatch terminal runs must record `correction.needed: true` with:
|
|
55
|
-
|
|
56
|
-
- `correction.type`: one of `workflow`, `skill`, `agent`, `tooling`, `test`, `doc`, `process`, `product`, `provider`, or `none`
|
|
57
|
-
- `correction.recurrence_key`: a stable free-form key for grouping repeated failures
|
|
58
|
-
- `correction.intended_behavior`: what the workflow, artifact, skill, agent, or provider was supposed to do
|
|
59
|
-
- `correction.observed_behavior`: what actually happened
|
|
60
|
-
- `correction.gap`: the difference between intended behavior and observed behavior
|
|
61
|
-
- `correction.prevention`: a routed prevention action, or `correction.no_change_rationale` when no change is intentionally made
|
|
62
|
-
|
|
63
|
-
If a correction-needed prevention route is still open, use `learning.status: followup_required` unless the route is already represented by a durable provider issue, deferred with a trigger, accepted, or rejected with a rationale.
|
|
16
|
+
## Binding
|
|
64
17
|
|
|
65
|
-
|
|
18
|
+
| Context | Binding | Flow expectations |
|
|
19
|
+
| --- | --- | --- |
|
|
20
|
+
| Active Builder run | Parent `builder.build` projection at composed `builder.publish-learn` step `learn` | `decision-evidence` and `learning-evidence` |
|
|
21
|
+
| Standalone invocation | No Flow binding | No workflow mutation. |
|
|
66
22
|
|
|
67
|
-
For
|
|
23
|
+
For an active run, inspect the binding first:
|
|
68
24
|
|
|
69
25
|
```bash
|
|
70
|
-
|
|
71
|
-
--check-json '{"id":"...","kind":"test","status":"pass","summary":"..."}' \
|
|
72
|
-
--require-critique \
|
|
73
|
-
--critique-id "..." \
|
|
74
|
-
--critique-summary "..." \
|
|
75
|
-
--learning-record-json '{"id":"...","source_refs":["evidence.json","critique.json"],"outcome":"success","facts":["..."],"interpretation":"...","routing":[{"target":"none","action":"No correction required.","status":"completed"}],"correction":{"needed":false,"evidence":"Evidence and critique matched the intended closeout behavior."}}'
|
|
26
|
+
flow-agents workflow status --session-dir <session-dir> --json
|
|
76
27
|
```
|
|
77
28
|
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
### 1. Reconstruct Outcome
|
|
83
|
-
|
|
84
|
-
Compare the original intent, acceptance criteria, release decision, and post-deploy signals. Distinguish facts from interpretation.
|
|
85
|
-
|
|
86
|
-
### 1a. Decide Correction State
|
|
87
|
-
|
|
88
|
-
Before identifying durable learnings, write down the intended behavior, observed behavior, and any gap. If there is no gap, record `correction.needed: false`. If there is a gap, record `correction.needed: true` with typed `correction.type`, stable `correction.recurrence_key`, intended behavior, observed behavior, gap, and either a prevention route or an explicit no-change rationale.
|
|
89
|
-
|
|
90
|
-
### 2. Identify Learning
|
|
91
|
-
|
|
92
|
-
Classify learnings as product, technical, operational, workflow, test, documentation, eval, or agent-behavior learning.
|
|
93
|
-
|
|
94
|
-
### 2a. Review Delegation Routing Efficiency
|
|
95
|
-
|
|
96
|
-
Run the routing-efficiency analyzer over the run's economics records and review its proposals — the
|
|
97
|
-
internal mirror of the #409 small-model value proof, applied to our own agent fan-out:
|
|
98
|
-
|
|
99
|
-
```bash
|
|
100
|
-
bash scripts/telemetry/routing-efficiency.sh .kontourai/telemetry/economics.jsonl
|
|
101
|
-
```
|
|
102
|
-
|
|
103
|
-
It emits ADVISORY per-`(role, model)` proposals (`escalate-minimum-tier`, `keep-tier`, `monitor`,
|
|
104
|
-
`insufficient-signal`) with rationales, computed only from **measurable** outcomes — `unavailable`
|
|
105
|
-
outcomes are excluded (a missing verdict is neither success nor failure; see
|
|
106
|
-
`docs/specs/harness-capability-matrix.md`). Fold any actionable proposal into `learning.json`:
|
|
107
|
-
|
|
108
|
-
- an `escalate-minimum-tier` / demote proposal → a `routing` entry (`target: "rule"`, naming the
|
|
109
|
-
`.datum/config.json` role→model change to consider) or `correction` with `type: "agent"`.
|
|
110
|
-
- **These are proposals, never auto-applied.** A human ratifies the `.datum/config.json` change, which
|
|
111
|
-
then travels the normal deliver loop (ADR 0003 call 5). `insufficient-signal` proposals are recorded
|
|
112
|
-
as coverage notes, not routing changes.
|
|
29
|
+
Public status reports the parent definition as `builder.build`. Only that parent
|
|
30
|
+
run at `learn`, whose Flow Definition composes the step from
|
|
31
|
+
`builder.publish-learn`, may publish workflow evidence. A standalone review
|
|
32
|
+
produces its local record only.
|
|
113
33
|
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
Run the kit/gate economics analyzer over a window of economics records whenever enough new
|
|
117
|
-
records have piled up since the last ledger entry — cadenced during a learning-review pass, not
|
|
118
|
-
on a scheduler (see `--help` for the full flag set):
|
|
119
|
-
|
|
120
|
-
```bash
|
|
121
|
-
bash scripts/telemetry/learning-review-proposals.sh --since <last-run-until> --until <now>
|
|
122
|
-
```
|
|
123
|
-
|
|
124
|
-
It emits ADVISORY per-kit (`kit-review-cost-inflation`) and per-gate (`gate-false-block-review`,
|
|
125
|
-
`gate-well-calibrated`) proposals, each citing paired `evidence.cost` + `evidence.defect` — never
|
|
126
|
-
cost alone. `insufficient-data` is a valid, reportable outcome, not a bar to lower; never propose
|
|
127
|
-
from noise. Surface every new (non-`already_proposed`) proposal to the human with its evidence,
|
|
128
|
-
then record the ratify/reject/defer decision BEFORE any follow-on work exists:
|
|
129
|
-
|
|
130
|
-
```bash
|
|
131
|
-
bash scripts/telemetry/learning-review-decide.sh <ledger> <proposal-id> \
|
|
132
|
-
--ratify|--reject|--defer --decided-by <name> --rationale "<why>"
|
|
133
|
-
```
|
|
134
|
-
|
|
135
|
-
- **These are proposals, never auto-applied.** Nothing here writes to `kits/**`,
|
|
136
|
-
`.datum/config.json`, or any gate/flow config file.
|
|
137
|
-
- Only on `--ratify`, create the ordinary follow-on backlog item and record it back onto the
|
|
138
|
-
proposal with `--follow-on-ref <ref>` — a follow-on may never cite an unratified proposal. A
|
|
139
|
-
later pass's effect-fill shows whether the ratified change actually moved the numbers.
|
|
140
|
-
|
|
141
|
-
See `docs/specs/learning-review-proposals-contract.md` for the full contract.
|
|
142
|
-
|
|
143
|
-
### 3. Route Follow-Up
|
|
144
|
-
|
|
145
|
-
Route raw ideas or ambiguous improvements to `idea-to-backlog`. Create GitHub issues only for executable follow-up. Use `evidence-gate` again when unresolved trust questions remain.
|
|
146
|
-
|
|
147
|
-
### 4. Capture Durable Knowledge
|
|
148
|
-
|
|
149
|
-
Use `knowledge-capture` for durable project or relationship knowledge and `observe` for repeated agent workflow patterns or corrections.
|
|
150
|
-
|
|
151
|
-
### 5. Confirm Docs Promotion
|
|
152
|
-
|
|
153
|
-
Check whether accepted delivery artifacts were promoted into long-lived documentation and whether `.flow-agents/` runtime artifacts remained untracked before merge to `main`. If not, route the missing doc or cleanup work as an owned follow-up or explicitly record why the delivery was self-explanatory.
|
|
154
|
-
|
|
155
|
-
### 6. Close The Loop
|
|
156
|
-
|
|
157
|
-
Record which follow-ups were created, which were intentionally deferred, and what trigger should revisit deferred work.
|
|
158
|
-
|
|
159
|
-
## Gate Claims: Record Learning Outcomes
|
|
160
|
-
|
|
161
|
-
After `learning.json` is written and the learning verdict is `LEARNED` or `FOLLOWUP_REQUIRED`, record the two gate claims for the Builder Kit `learn` step. These satisfy the `builder.learn.decisions` and `builder.learn.evidence` gate expectations.
|
|
34
|
+
## Inputs
|
|
162
35
|
|
|
163
|
-
|
|
36
|
+
- Original intent, acceptance evidence, confidence report, and release decision.
|
|
37
|
+
- Observed outcomes from `RepositoryAdapter`, `CheckProvider`, `ReleaseProvider`,
|
|
38
|
+
and `DeployProvider` when those providers exist.
|
|
39
|
+
- Incidents, user feedback, review findings, operational observations, and
|
|
40
|
+
durable documentation decisions.
|
|
41
|
+
|
|
42
|
+
## Learning Work
|
|
43
|
+
|
|
44
|
+
1. Compare intended and observed outcomes. Separate observed facts from
|
|
45
|
+
interpretation.
|
|
46
|
+
2. Record delivery decisions and whether a correction is needed. A clean result
|
|
47
|
+
may record that no correction is needed; do not invent a lesson.
|
|
48
|
+
3. When a correction is needed, name the gap, the affected workflow or product
|
|
49
|
+
area, and a durable follow-up, deferral condition, accepted decision, or
|
|
50
|
+
explicit rationale for no change.
|
|
51
|
+
4. Give each correction a stable identifier, owner or ownership gap, target
|
|
52
|
+
destination, evidence, disposition, and follow-up state. Route product work
|
|
53
|
+
to the backlog, workflow defects to the owning workflow, regression risks to
|
|
54
|
+
tests, durable operating knowledge to the knowledge system, and binding
|
|
55
|
+
decisions or changed contracts to documentation or an ADR.
|
|
56
|
+
5. Record follow-up creation or linkage as observed evidence. A recommendation
|
|
57
|
+
without a durable destination remains open; do not report it as captured.
|
|
58
|
+
6. Preserve `NOT_VERIFIED` for outcomes that could not be observed. Do not infer
|
|
59
|
+
deployment, release, provider, or user outcomes from local completion alone.
|
|
60
|
+
7. Give `decision-evidence` and `learning-evidence` independent verdicts. A
|
|
61
|
+
complete decision record can pass while learning follow-up remains failed or
|
|
62
|
+
`NOT_VERIFIED`.
|
|
63
|
+
8. On a matching active run, publish both expectations through the public CLI:
|
|
164
64
|
|
|
165
65
|
```bash
|
|
166
|
-
|
|
66
|
+
flow-agents workflow evidence --session-dir <session-dir> \
|
|
167
67
|
--expectation decision-evidence \
|
|
168
|
-
--status pass \
|
|
169
|
-
--summary "
|
|
170
|
-
--evidence-ref-json '{"kind":"artifact","file":"
|
|
171
|
-
```
|
|
172
|
-
|
|
173
|
-
**Claim 2 — Learning evidence** (learnings from delivery are recorded for future work):
|
|
68
|
+
--status <pass|fail|not_verified> \
|
|
69
|
+
--summary "Delivery decisions and unresolved gaps are recorded." \
|
|
70
|
+
--evidence-ref-json '{"kind":"artifact","file":"<session-dir>/learning.json","summary":"Recorded delivery decisions, correction state, and routing."}'
|
|
174
71
|
|
|
175
|
-
|
|
176
|
-
npm run workflow:sidecar -- record-gate-claim .kontourai/flow-agents/<slug> \
|
|
72
|
+
flow-agents workflow evidence --session-dir <session-dir> \
|
|
177
73
|
--expectation learning-evidence \
|
|
178
|
-
--status pass \
|
|
179
|
-
--summary "
|
|
180
|
-
--evidence-ref-json '{"kind":"artifact","file":"
|
|
74
|
+
--status <pass|fail|not_verified> \
|
|
75
|
+
--summary "Observed outcomes and learning follow-up are recorded." \
|
|
76
|
+
--evidence-ref-json '{"kind":"artifact","file":"<session-dir>/learning.json","summary":"Observed outcomes, follow-up routing, and unresolved gaps."}'
|
|
181
77
|
```
|
|
182
78
|
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
When the learning verdict is `FOLLOWUP_REQUIRED`, record both claims with `--status pass` and name the open routing in the summary; the follow-up route is separate from gate satisfaction.
|
|
186
|
-
|
|
187
|
-
|
|
79
|
+
## Output
|
|
188
80
|
|
|
189
|
-
|
|
81
|
+
Record scope, intended and observed outcomes, decisions, facts, interpretation,
|
|
82
|
+
correction state, `NOT_VERIFIED` gaps, and follow-up routing as the
|
|
83
|
+
`decision-evidence` and `learning-evidence` slices in `trust.bundle`.
|
|
84
|
+
Corrections include their identifier, owner, destination, evidence, disposition,
|
|
85
|
+
and durable link when one was created.
|
|
190
86
|
|
|
191
|
-
|
|
192
|
-
-
|
|
193
|
-
|
|
194
|
-
- Docs Gate: accepted planning and final acceptance artifacts are archived and linked from durable docs when useful.
|
|
195
|
-
- Closure Gate: the artifact says what is done, what remains, and why.
|
|
196
|
-
- Correction Gate: every terminal learning review records whether correction was needed; clean runs do not invent lessons, and mismatch runs have a recurrence key plus prevention route or no-change rationale.
|
|
87
|
+
For a matching active run, publish both `decision-evidence` and
|
|
88
|
+
`learning-evidence` from those slices. These claims record closeout evidence;
|
|
89
|
+
they do not erase open follow-up work or authorize an external operation.
|