@kontourai/flow-agents 3.6.0 → 3.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +19 -0
- package/build/src/builder-flow-run-adapter.d.ts +22 -2
- package/build/src/builder-flow-run-adapter.js +93 -28
- package/build/src/builder-flow-runtime.d.ts +8 -3
- package/build/src/builder-flow-runtime.js +407 -51
- package/build/src/cli/assignment-provider.d.ts +4 -7
- package/build/src/cli/assignment-provider.js +80 -14
- package/build/src/cli/builder-run.js +14 -18
- package/build/src/cli/workflow-sidecar.d.ts +28 -4
- package/build/src/cli/workflow-sidecar.js +825 -103
- package/build/src/cli/workflow.js +301 -43
- package/build/src/flow-kit/validate.d.ts +17 -0
- package/build/src/flow-kit/validate.js +340 -2
- package/build/src/index.js +5 -5
- package/build/src/lib/observed-command.d.ts +7 -0
- package/build/src/lib/observed-command.js +100 -0
- package/build/src/tools/build-universal-bundles.js +53 -3
- package/build/src/tools/generate-context-map.js +5 -3
- package/context/scripts/hooks/lib/kit-catalog.js +1 -1
- package/context/scripts/hooks/stop-goal-fit.js +78 -9
- package/docs/context-map.md +22 -20
- package/docs/developer-architecture.md +1 -1
- package/docs/getting-started.md +9 -1
- package/docs/public-workflow-cli.md +76 -7
- package/docs/skills-map.md +51 -27
- package/docs/spec/builder-flow-runtime.md +75 -40
- package/docs/workflow-usage-guide.md +109 -42
- package/evals/fixtures/hook-influence/cases.json +2 -2
- package/evals/integration/test_builder_entry_enforcement.sh +52 -41
- package/evals/integration/test_builder_step_producers.sh +330 -6
- package/evals/integration/test_bundle_install.sh +318 -65
- package/evals/integration/test_bundle_lifecycle.sh +4 -6
- package/evals/integration/test_critique_supersession_roundtrip.sh +21 -8
- package/evals/integration/test_current_json_per_actor.sh +12 -0
- package/evals/integration/test_dual_emit_flow_step.sh +62 -43
- package/evals/integration/test_flowdef_session_activation.sh +145 -25
- package/evals/integration/test_flowdef_session_history_preservation.sh +23 -21
- package/evals/integration/test_gate_lockdown.sh +3 -5
- package/evals/integration/test_goal_fit_hook.sh +60 -2
- package/evals/integration/test_install_merge.sh +18 -8
- package/evals/integration/test_liveness_verdict.sh +14 -17
- package/evals/integration/test_phase_map_and_gate_claim.sh +63 -38
- package/evals/integration/test_public_workflow_cli.sh +334 -14
- package/evals/integration/test_pull_work_liveness_preflight.sh +22 -66
- package/evals/integration/test_sidecar_field_preservation.sh +36 -11
- package/evals/integration/test_workflow_sidecar_writer.sh +277 -44
- package/evals/integration/test_workflow_steering_hook.sh +15 -38
- package/evals/run.sh +2 -0
- package/evals/static/test_builder_skill_coherence.sh +247 -0
- package/evals/static/test_library_exports.sh +5 -2
- package/evals/static/test_universal_bundles.sh +44 -1
- package/evals/static/test_workflow_skills.sh +13 -325
- package/kits/builder/flows/build.flow.json +22 -0
- package/kits/builder/flows/shape.flow.json +9 -9
- package/kits/builder/kit.json +70 -16
- package/kits/builder/skills/builder-shape/SKILL.md +75 -75
- package/kits/builder/skills/continue-work/SKILL.md +45 -106
- package/kits/builder/skills/deliver/SKILL.md +96 -442
- package/kits/builder/skills/design-probe/SKILL.md +40 -139
- package/kits/builder/skills/evidence-gate/SKILL.md +59 -201
- package/kits/builder/skills/execute-plan/SKILL.md +54 -125
- package/kits/builder/skills/fix-bug/SKILL.md +42 -132
- package/kits/builder/skills/gate-review/SKILL.md +60 -211
- package/kits/builder/skills/idea-to-backlog/SKILL.md +63 -244
- package/kits/builder/skills/learning-review/SKILL.md +63 -170
- package/kits/builder/skills/pickup-probe/SKILL.md +54 -111
- package/kits/builder/skills/plan-work/SKILL.md +51 -185
- package/kits/builder/skills/pull-work/SKILL.md +136 -485
- package/kits/builder/skills/release-readiness/SKILL.md +66 -107
- package/kits/builder/skills/review-work/SKILL.md +89 -176
- package/kits/builder/skills/tdd-workflow/SKILL.md +53 -147
- package/kits/builder/skills/verify-work/SKILL.md +101 -113
- package/package.json +2 -2
- package/packaging/README.md +1 -1
- package/scripts/README.md +1 -1
- package/scripts/hooks/lib/kit-catalog.js +1 -1
- package/scripts/hooks/stop-goal-fit.js +78 -9
- package/scripts/install-codex-home.sh +63 -23
- package/scripts/install-owned-files.js +18 -2
- package/src/builder-flow-run-adapter.ts +118 -32
- package/src/builder-flow-runtime.ts +426 -64
- package/src/cli/assignment-provider-lock.test.mjs +83 -0
- package/src/cli/assignment-provider.ts +75 -14
- package/src/cli/builder-flow-run-adapter.test.mjs +4 -2
- package/src/cli/builder-flow-runtime.test.mjs +436 -18
- package/src/cli/builder-run.ts +3 -9
- package/src/cli/kit-metadata-security.test.mjs +232 -6
- package/src/cli/public-api.test.mjs +15 -0
- package/src/cli/workflow-sidecar-execution-proof.test.mjs +90 -0
- package/src/cli/workflow-sidecar.ts +746 -103
- package/src/cli/workflow.ts +288 -41
- package/src/flow-kit/validate.ts +320 -2
- package/src/index.ts +6 -5
- package/src/lib/observed-command.ts +96 -0
- package/src/tools/build-universal-bundles.ts +51 -3
- package/src/tools/generate-context-map.ts +5 -3
|
@@ -1,74 +1,54 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: "design-probe"
|
|
3
|
-
description: "Generic one-question-at-a-time design
|
|
3
|
+
description: "Generic one-question-at-a-time design probe for turning unclear goals, designs, and handoffs into shared understanding."
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# Design Probe
|
|
7
7
|
|
|
8
|
-
Use `design-probe` when a goal, design,
|
|
8
|
+
Use `design-probe` when a goal, design, implementation boundary, acceptance criterion, workflow route, or recovery path is not clear enough to plan or execute responsibly.
|
|
9
9
|
|
|
10
|
-
|
|
10
|
+
## Role And Binding
|
|
11
11
|
|
|
12
|
-
|
|
12
|
+
- **Role:** standalone shared alignment primitive.
|
|
13
|
+
- **Binding:** none. This skill is not a Builder step producer.
|
|
14
|
+
- **Produces:** a compact decision record in the caller's existing artifact or conversation; no prescribed Builder artifact or expectation.
|
|
15
|
+
- **Standalone no-run behavior:** always run as an independent probe. Do not start, inspect, advance, or record evidence for any Builder flow.
|
|
13
16
|
|
|
14
17
|
## Contract
|
|
15
18
|
|
|
16
|
-
- Explore
|
|
17
|
-
-
|
|
18
|
-
-
|
|
19
|
-
-
|
|
20
|
-
-
|
|
21
|
-
- Include a recommended answer with every question and briefly explain why it is recommended.
|
|
22
|
-
- Make the recommendation actionable enough that the user can accept it directly.
|
|
23
|
-
- Record decisions, unresolved questions, accepted gaps, and planning readiness as the interview progresses.
|
|
24
|
-
- Stop when shared understanding exists, or when the remaining uncertainty is explicitly recorded as an accepted gap.
|
|
25
|
-
- Do not silently convert uncertainty into implementation work.
|
|
19
|
+
- Explore available docs, plans, artifacts, contracts, code, tests, issue text, and prior decisions before asking for discoverable information.
|
|
20
|
+
- Resolve one decision branch at a time. Ask exactly one alignment question per turn, include a recommended answer, and wait for the response.
|
|
21
|
+
- Keep probing fuzzy goals, overloaded terms, implicit non-goals, missing constraints, and weak success signals until they are resolved or explicitly accepted as gaps.
|
|
22
|
+
- Do not silently turn uncertainty into implementation work.
|
|
23
|
+
- Do not replace backlog shaping, implementation planning, verification, release review, or any flow-specific producer.
|
|
26
24
|
|
|
27
25
|
## Model Routing
|
|
28
26
|
|
|
29
|
-
When
|
|
30
|
-
|
|
31
|
-
(`npx @kontourai/datum resolve delegate-design --json`) and pass the resolved
|
|
32
|
-
model explicitly — probing a goal, design, or recovery path needs design
|
|
33
|
-
latitude, so it routes to the design tier. See
|
|
34
|
-
`context/contracts/execution-contract.md` § Delegation: Model Routing. Fallback:
|
|
35
|
-
inherit the session model when datum/config is absent, noted in the artifact.
|
|
27
|
+
When delegating alignment work, resolve `delegate-design` from `.datum/config.json`. If unavailable, inherit the session model and state the fallback in the decision record.
|
|
28
|
+
Apply the routing and escalation contract in `context/contracts/execution-contract.md`.
|
|
36
29
|
|
|
37
30
|
## When To Use
|
|
38
31
|
|
|
39
|
-
Use this skill
|
|
32
|
+
Use this primitive for ambiguous product or feature goals, conflicting requirements, missing acceptance criteria, architecture decisions, unclear recovery paths, and uncertain next actions. It may be called by a specialized workflow skill, but it owns no workflow gate or claim procedure.
|
|
40
33
|
|
|
41
|
-
|
|
42
|
-
- Conflicting requirements or unclear non-goals.
|
|
43
|
-
- Missing acceptance criteria or unclear evidence expectations.
|
|
44
|
-
- Architecture or workflow decisions that block planning.
|
|
45
|
-
- Direct primitive recovery when upstream context or state is missing.
|
|
46
|
-
- Guided workflow next-step selection when artifacts do not clearly identify whether to ask, plan, execute, verify, or stop.
|
|
34
|
+
## Discovery
|
|
47
35
|
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
Before asking the first question:
|
|
53
|
-
|
|
54
|
-
1. Read the user's request and identify the decision branch that blocks progress.
|
|
55
|
-
2. Search local context that could answer it, such as `README`, `CONTEXT.md`, `docs/`, `context/contracts/`, relevant skills, active workflow artifacts, schemas, tests, and nearby implementation files.
|
|
56
|
-
3. Prefer existing project vocabulary and documented decisions over inventing new terms.
|
|
57
|
-
4. If local evidence resolves the branch, record the inferred decision and move to the next branch.
|
|
58
|
-
5. Ask only when the branch remains ambiguous, contradictory, risky, or value-laden.
|
|
36
|
+
1. Identify the decision branch that blocks progress.
|
|
37
|
+
2. Search local context for the answer and prefer existing vocabulary and documented decisions.
|
|
38
|
+
3. Record an inferred decision when evidence resolves the branch.
|
|
39
|
+
4. Ask only when the decision remains ambiguous, contradictory, risky, or value-laden.
|
|
59
40
|
|
|
60
41
|
## Interview Loop
|
|
61
42
|
|
|
62
43
|
For each unresolved branch:
|
|
63
44
|
|
|
64
|
-
1. State the branch
|
|
65
|
-
2. Ask
|
|
66
|
-
3.
|
|
67
|
-
4.
|
|
68
|
-
5.
|
|
69
|
-
6. Record the outcome before continuing.
|
|
45
|
+
1. State the branch in one short sentence.
|
|
46
|
+
2. Ask one question.
|
|
47
|
+
3. Give a specific recommended answer and its practical consequence.
|
|
48
|
+
4. Wait for the answer.
|
|
49
|
+
5. Record the decision, unresolved question, or accepted gap before moving on.
|
|
70
50
|
|
|
71
|
-
|
|
51
|
+
Use this format:
|
|
72
52
|
|
|
73
53
|
```markdown
|
|
74
54
|
Question: <one alignment question>
|
|
@@ -78,105 +58,26 @@ Recommended answer: <specific answer the user can accept>
|
|
|
78
58
|
Why: <brief reason and consequence>
|
|
79
59
|
```
|
|
80
60
|
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
## Records
|
|
84
|
-
|
|
85
|
-
Maintain a compact running record in the active artifact or conversation when no artifact exists:
|
|
86
|
-
|
|
87
|
-
- `decisions`: choices that are aligned or locally inferable.
|
|
88
|
-
- `unresolved_questions`: questions still blocking planning or execution.
|
|
89
|
-
- `accepted_gaps`: uncertainties the user explicitly accepts, including the consequence.
|
|
90
|
-
- `planning_readiness`: one of `ready`, `needs_more_probe`, or `accepted_gap_ready`.
|
|
91
|
-
- `next_action`: the recommended next workflow step, such as `shape`, `plan-work`, `execute-plan`, `verify-work`, `needs_user`, or `stop`.
|
|
92
|
-
|
|
93
|
-
When workflow artifacts exist, update the appropriate session, handoff, Probe record, or planning artifact according to the local artifact contract. Do not invent a project-specific storage format when the repository already defines one.
|
|
94
|
-
|
|
95
|
-
## Docs-Write: Two-Delta Emission
|
|
96
|
-
|
|
97
|
-
Per `context/contracts/probe-docs-write-contract.md`, when a `decisions` entry above is a **decision subject** — a durable, reusable answer to a named domain question, not an implementation detail or transient planning choice — emit two deltas into durable docs in the same motion, before moving to the next branch:
|
|
98
|
-
|
|
99
|
-
1. **Vocabulary delta**: the subject noun must exist as a `CONTEXT.md` glossary term. If absent, coin it first (a tight one- or two-sentence `### <Term>` entry); if present, refine the definition when the crystallized decision changes what it says (e.g. an "open" pointer becoming a stated answer).
|
|
100
|
-
2. **Decision delta**: consult `docs/decisions/index.md` and propose revise-vs-create against the existing topic slugs (this includes any `needs-decision` stub already naming the subject); revise the existing topic file in place, or create `docs/decisions/<slug>.md` per `context/contracts/decision-registry-contract.md`. Run `npm run gen:decisions-index && npm run check:decisions` after writing.
|
|
101
|
-
3. **Transcript provenance**: add a `session-archive` evidence ref pointing at this Probe's own session artifact (the live `.kontourai/flow-agents/<slug>/...` path — it does not need to be archived yet).
|
|
102
|
-
|
|
103
|
-
Example — coining a new term and creating its topic file:
|
|
104
|
-
|
|
105
|
-
```markdown
|
|
106
|
-
### Retry Budget
|
|
107
|
-
|
|
108
|
-
The maximum number of automatic retries a workflow step may attempt before routing back to the user. Configured per Flow Definition step, not globally.
|
|
109
|
-
```
|
|
110
|
-
|
|
111
|
-
```markdown
|
|
112
|
-
---
|
|
113
|
-
status: current
|
|
114
|
-
subject: Retry budget
|
|
115
|
-
decided: 2026-07-03
|
|
116
|
-
evidence:
|
|
117
|
-
- kind: session-archive
|
|
118
|
-
ref: .kontourai/flow-agents/<slug>/<slug>--design-probe.md
|
|
119
|
-
- kind: issue
|
|
120
|
-
ref: https://github.com/kontourai/flow-agents/issues/<n>
|
|
121
|
-
---
|
|
122
|
-
|
|
123
|
-
# Retry budget
|
|
124
|
-
|
|
125
|
-
Each Flow Definition step declares its own retry budget; there is no global default. Set during design-probe alignment on <n>.
|
|
126
|
-
```
|
|
127
|
-
|
|
128
|
-
Example — revising a `needs-decision` stub seeded by the ADR-freeze cutover (issue #314) once its frozen ADR's decision is confirmed still current: flip `status: needs-decision` to `current`, set `decided` to today, keep the existing `adr` evidence ref, and append the `session-archive` ref for this Probe session plus a `pr` ref once one exists. See `context/contracts/probe-docs-write-contract.md` § Worked Example for the full step-by-step.
|
|
129
|
-
|
|
130
|
-
Do not write a numbered ADR — `docs/adr/` is frozen history (`docs/adr/README.md`); every decision delta targets the topic-keyed registry.
|
|
131
|
-
|
|
132
|
-
## Stop Conditions
|
|
133
|
-
|
|
134
|
-
Stop probing when one of these is true:
|
|
61
|
+
## Decision Record
|
|
135
62
|
|
|
136
|
-
|
|
137
|
-
- The user explicitly accepts a gap and its consequence, and the next action can proceed with that gap recorded.
|
|
138
|
-
- The next action is to stop because the goal is out of scope, not worth pursuing, or blocked by an external dependency.
|
|
63
|
+
Maintain, in an existing caller artifact when one exists:
|
|
139
64
|
|
|
140
|
-
|
|
65
|
+
- decisions
|
|
66
|
+
- unresolved questions
|
|
67
|
+
- accepted gaps and consequences
|
|
68
|
+
- planning readiness: `ready`, `needs_more_probe`, or `accepted_gap_ready`
|
|
69
|
+
- recommended next action
|
|
141
70
|
|
|
142
|
-
-
|
|
143
|
-
- Remaining unresolved questions, if any.
|
|
144
|
-
- Accepted gaps, if any.
|
|
145
|
-
- Planning readiness.
|
|
146
|
-
- Recommended next action.
|
|
71
|
+
For a durable, reusable domain decision, follow the repository's vocabulary and decision-record contract. Keep transient planning choices and open questions in the caller's working artifact.
|
|
147
72
|
|
|
148
|
-
##
|
|
149
|
-
|
|
150
|
-
When `design-probe` runs at the Builder Kit `design-probe` flow step and the probe reaches a stop condition with shared understanding or accepted gaps, record the gate claims before handing off to `plan-work`.
|
|
151
|
-
|
|
152
|
-
This applies whether the probe is run directly (generic) or as part of a Builder Kit productized flow. The `pickup-probe` specialization owns the same two claims when it runs instead.
|
|
153
|
-
|
|
154
|
-
**Claim 1 — Pickup readiness** (probe passed, goal fit and scope confirmed):
|
|
155
|
-
|
|
156
|
-
```bash
|
|
157
|
-
npm run workflow:sidecar -- record-gate-claim .kontourai/flow-agents/<slug> \
|
|
158
|
-
--expectation pickup-probe-readiness \
|
|
159
|
-
--status pass \
|
|
160
|
-
--summary "Design probe passed: goal fit confirmed, scope aligned, planning readiness verified." \
|
|
161
|
-
--evidence-ref-json '{"kind":"artifact","file":".kontourai/flow-agents/<slug>/<slug>--<artifact>.md","summary":"Design-probe artifact with decisions, accepted gaps, and planning readiness."}'
|
|
162
|
-
```
|
|
163
|
-
|
|
164
|
-
**Claim 2 — Probe decisions captured**:
|
|
165
|
-
|
|
166
|
-
```bash
|
|
167
|
-
npm run workflow:sidecar -- record-gate-claim .kontourai/flow-agents/<slug> \
|
|
168
|
-
--expectation probe-decisions-or-accepted-gaps \
|
|
169
|
-
--status pass \
|
|
170
|
-
--summary "Probe decisions recorded: decisions made, unresolved questions explicit, planning readiness confirmed." \
|
|
171
|
-
--evidence-ref-json '{"kind":"artifact","file":".kontourai/flow-agents/<slug>/<slug>--<artifact>.md","summary":"Design-probe artifact with decisions and accepted gaps."}'
|
|
172
|
-
```
|
|
73
|
+
## Stop
|
|
173
74
|
|
|
174
|
-
|
|
75
|
+
Stop when shared understanding exists, an explicit accepted gap permits the next action, or the correct next action is to stop. Summarize decisions, unresolved questions, accepted gaps, readiness, and the recommended next action.
|
|
175
76
|
|
|
176
77
|
## Boundaries
|
|
177
78
|
|
|
178
79
|
- Do not ask multiple questions in one turn.
|
|
179
|
-
- Do not ask for information already discoverable from local
|
|
180
|
-
- Do not broaden
|
|
181
|
-
- Do not
|
|
182
|
-
- Do not
|
|
80
|
+
- Do not ask for information already discoverable from local context.
|
|
81
|
+
- Do not broaden into unrelated architecture review or implementation.
|
|
82
|
+
- Do not use Builder terminology as a prerequisite for this primitive.
|
|
83
|
+
- Do not create or record Builder claims, expectation evidence, or flow state.
|
|
@@ -1,223 +1,81 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: "evidence-gate"
|
|
3
|
-
description: "
|
|
3
|
+
description: "Assess whether verification evidence and scope integrity support merge review. Produces a confidence report and Builder merge-readiness evidence."
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# Evidence Gate
|
|
7
7
|
|
|
8
|
-
|
|
8
|
+
## Role
|
|
9
9
|
|
|
10
|
-
|
|
10
|
+
This is a Builder step skill.
|
|
11
11
|
|
|
12
|
-
|
|
12
|
+
Evidence Gate assesses whether the completed work has trustworthy, scoped
|
|
13
|
+
evidence for merge review. It does not fix code, publish a change, approve a
|
|
14
|
+
release, or deploy.
|
|
13
15
|
|
|
14
|
-
|
|
15
|
-
- Do not fix code.
|
|
16
|
-
- Do not mark unverified work as passing.
|
|
17
|
-
- Treat `NOT_VERIFIED` as a first-class outcome.
|
|
18
|
-
- Separate evidence provenance: human-authored, agent-authored, CI-generated, runtime-observed.
|
|
19
|
-
- Do not approve release readiness.
|
|
20
|
-
- After a clean local evidence verdict, require a publish-change gate before `release-readiness`: verified diff committed, branch pushed, provider change opened or updated by the active `ChangeProvider` or an explicit no-provider-change reason recorded, closing refs recorded, provider checks known, and evidence refs linked.
|
|
21
|
-
- Provider-facing summaries, PR/change descriptions, issue comments, closure comments, and final acceptance comments that claim implementation behavior must include an `Acceptance Evidence` table with columns `AC id`, `Status`, `Command/Test Evidence`, `Source Evidence / Permalinks`, and `Gaps`.
|
|
16
|
+
## Binding
|
|
22
17
|
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
- Verification report.
|
|
28
|
-
- Provider change / branch / check run links when available.
|
|
29
|
-
- Changed-file summary.
|
|
30
|
-
- Active TODOs, issue links, and release/rollback notes.
|
|
31
|
-
- Cross-repo dependency/provider coverage matrix when work spans multiple
|
|
32
|
-
products, package-manager roots, generated artifact locations, install
|
|
33
|
-
scripts, CI, or shared workflow guidance.
|
|
34
|
-
|
|
35
|
-
## Artifact Contract
|
|
36
|
-
|
|
37
|
-
Write or update `.kontourai/flow-agents/<slug>/<slug>--evidence-gate.md` with:
|
|
38
|
-
|
|
39
|
-
- `intent`: issue/brief, acceptance criteria, non-goals, risk class
|
|
40
|
-
- `evidence_manifest`: command/check name, source, timestamp, result, link/output pointer
|
|
41
|
-
- `test_map`: acceptance criterion to evidence tier and gaps
|
|
42
|
-
- `integrity_report`: scope drift, weakened tests/config, sensitive files
|
|
43
|
-
- `ci_report`: checks, reruns, flakes, failures, skipped checks
|
|
44
|
-
- `risk_assessment`: residual risks and required human review
|
|
45
|
-
- `verdict`: PASS, FAIL, or NOT_VERIFIED
|
|
46
|
-
- `next_step`: publish-change, release-readiness, verify-work, execute-plan, plan-work, CI remediation, or human decision
|
|
47
|
-
|
|
48
|
-
Also write or update structured sidecars:
|
|
49
|
-
|
|
50
|
-
- `state.json`: phase `evidence`, current status, and required next action
|
|
51
|
-
- `acceptance.json`: final criterion statuses and goal-fit status
|
|
52
|
-
- `evidence.json`: normalized checks, `standard_refs`, external evidence refs, not-verified gaps, and verdict
|
|
53
|
-
- `handoff.json`: next step and blockers when verdict is not a clean pass
|
|
18
|
+
| Context | Binding | Flow expectation |
|
|
19
|
+
| --- | --- | --- |
|
|
20
|
+
| Active Builder run | `builder.build` at `merge-ready` | `merge-readiness` |
|
|
21
|
+
| Standalone invocation | No Flow binding | No workflow mutation. |
|
|
54
22
|
|
|
55
|
-
|
|
23
|
+
For an active run, confirm the binding before evaluation:
|
|
56
24
|
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
Restate:
|
|
25
|
+
```bash
|
|
26
|
+
flow-agents workflow status --session-dir <session-dir> --json
|
|
27
|
+
```
|
|
62
28
|
|
|
63
|
-
-
|
|
64
|
-
|
|
65
|
-
- non-goals
|
|
66
|
-
- expected risk class
|
|
67
|
-
- authoritative artifacts
|
|
29
|
+
Only a matching run may publish `merge-readiness`. A standalone confidence
|
|
30
|
+
report is local and must not call `workflow evidence`.
|
|
68
31
|
|
|
69
|
-
|
|
32
|
+
## Inputs
|
|
70
33
|
|
|
71
|
-
|
|
34
|
+
- The relevant verification and critique slices in `trust.bundle`.
|
|
35
|
+
- Changed-file scope from `RepositoryAdapter` or the execution record.
|
|
36
|
+
- Change and check records from `ChangeProvider` and `CheckProvider` when available.
|
|
37
|
+
- Known risks, accepted gaps, and repository policy evidence.
|
|
38
|
+
|
|
39
|
+
## Evaluation
|
|
40
|
+
|
|
41
|
+
1. Operate report-only. Do not patch source, alter checks, publish a change, or
|
|
42
|
+
rerun with weaker options to manufacture a passing decision.
|
|
43
|
+
2. Confirm acceptance evidence maps to the intended changed scope. Compare the
|
|
44
|
+
planned scope, actual diff, generated files, and provider-reported change.
|
|
45
|
+
Unexplained drift, weakened checks, or unrelated edits reduce confidence and
|
|
46
|
+
must be named.
|
|
47
|
+
3. Classify evidence by strength: direct runtime or test observation; provider
|
|
48
|
+
check/review record; reproducible command result; source inspection; or
|
|
49
|
+
unsupported assertion. Preserve command, revision, timestamp, scope, and
|
|
50
|
+
artifact provenance where available. Prose-only completion claims are not
|
|
51
|
+
acceptance proof.
|
|
52
|
+
4. Re-run only when the command is safe, relevant, and reproducible. A stale,
|
|
53
|
+
unavailable, changed, or non-reproducible check is degraded to
|
|
54
|
+
`NOT_VERIFIED`; a previous pass must not remain current by assertion.
|
|
55
|
+
5. Produce `PASS`, `FAIL`, or `NOT_VERIFIED` for the confidence decision.
|
|
56
|
+
`NOT_VERIFIED` is the result for missing required proof, unavailable provider
|
|
57
|
+
information, or unresolved integrity questions.
|
|
58
|
+
6. On a matching active run, publish the decision through the public CLI:
|
|
72
59
|
|
|
73
|
-
|
|
60
|
+
```bash
|
|
61
|
+
flow-agents workflow evidence --session-dir <session-dir> \
|
|
62
|
+
--expectation merge-readiness \
|
|
63
|
+
--status <pass|fail|not_verified> \
|
|
64
|
+
--summary "Evidence confidence, scope integrity, and remaining risks are recorded." \
|
|
65
|
+
--evidence-ref-json '{"kind":"artifact","file":"<session-dir>/<slug>--evidence-gate.md","summary":"Evidence confidence, criterion coverage, and scope-integrity decision."}'
|
|
66
|
+
```
|
|
74
67
|
|
|
75
|
-
|
|
76
|
-
- new or modified automated test
|
|
77
|
-
- browser/runtime check
|
|
78
|
-
- static analysis
|
|
79
|
-
- CI check
|
|
80
|
-
- manual/human verification
|
|
81
|
-
- `NOT_VERIFIED` with rationale
|
|
68
|
+
## Output
|
|
82
69
|
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
70
|
+
Record the confidence report as the `merge-readiness` slice in `trust.bundle`.
|
|
71
|
+
It contains the scope assessed, acceptance evidence, integrity findings,
|
|
72
|
+
evidence provenance, residual risks, `PASS`/`FAIL`/`NOT_VERIFIED` decision, and
|
|
73
|
+
recommended route.
|
|
86
74
|
|
|
87
|
-
|
|
75
|
+
Include a readable `Acceptance Evidence` table when behavior is claimed:
|
|
88
76
|
|
|
89
|
-
| AC id | Status | Command/Test Evidence | Source Evidence
|
|
77
|
+
| AC id | Status | Command/Test Evidence | Source Evidence | Gaps |
|
|
90
78
|
| --- | --- | --- | --- | --- |
|
|
91
79
|
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
### 3. Scope And Integrity Check
|
|
95
|
-
|
|
96
|
-
Check for process gaming or accidental drift:
|
|
97
|
-
|
|
98
|
-
- scope expanded beyond issue/brief
|
|
99
|
-
- acceptance criteria changed after implementation
|
|
100
|
-
- tests removed or weakened
|
|
101
|
-
- verification config altered
|
|
102
|
-
- CI config altered
|
|
103
|
-
- required CI bypassed
|
|
104
|
-
- sensitive files touched without review
|
|
105
|
-
|
|
106
|
-
Sensitive areas include auth, security middleware, data migrations, CI config, deployment scripts, feature flags, test helpers, lint/type config, payment, crypto, and filesystem/network operations.
|
|
107
|
-
|
|
108
|
-
For multi-repo or cross-product changes, require an explicit coverage matrix
|
|
109
|
-
before a clean pass. The matrix must list every affected product/repo root and
|
|
110
|
-
the status of build/test evidence, dependency/security review, provider/CI
|
|
111
|
-
evidence, and any accepted gaps. A clean evidence verdict requires every
|
|
112
|
-
applicable root to be covered or a human-accepted gap recorded with the reason.
|
|
113
|
-
Do not infer cross-product coverage from a passing subset.
|
|
114
|
-
|
|
115
|
-
### 4. CI And Flake Assessment
|
|
116
|
-
|
|
117
|
-
Use `github-cli` / `gh` when available.
|
|
118
|
-
|
|
119
|
-
Record:
|
|
120
|
-
|
|
121
|
-
- check names
|
|
122
|
-
- pass/fail/skipped
|
|
123
|
-
- rerun count
|
|
124
|
-
- flake suspicion
|
|
125
|
-
- logs or artifact links
|
|
126
|
-
- failure class
|
|
127
|
-
- standard evidence refs when CI emits SARIF, JUnit, TAP, OpenTelemetry, Veritas, or another native proof format
|
|
128
|
-
|
|
129
|
-
For Flow Agents source changes, prefer the GitHub Actions `Flow Agents CI / Builder Kit Baseline` provider check when present. Its local equivalent is `bash evals/ci/run-baseline.sh`, which writes `evals/results/ci-baseline/summary.md` and command logs. Treat skipped live GitHub mutation checks, LLM acceptance, or unavailable Veritas/governance evidence as explicit skip or `NOT_VERIFIED` entries based on the work's risk class; do not convert the baseline summary into proof that those live lanes ran.
|
|
130
|
-
|
|
131
|
-
Treat passed-after-rerun as degraded confidence unless explained.
|
|
132
|
-
|
|
133
|
-
### 5. Evidence Tiers
|
|
134
|
-
|
|
135
|
-
Classify evidence:
|
|
136
|
-
|
|
137
|
-
- Tier 0: claim only, no artifact.
|
|
138
|
-
- Tier 1: local command output.
|
|
139
|
-
- Tier 2: automated test tied to acceptance criterion.
|
|
140
|
-
- Tier 3: CI-confirmed test on a clean environment.
|
|
141
|
-
- Tier 4: runtime/browser/production-like verification with trace or log artifact.
|
|
142
|
-
- Tier 5: post-deploy telemetry confirms expected behavior.
|
|
143
|
-
|
|
144
|
-
Higher-risk work requires stronger tiers.
|
|
145
|
-
|
|
146
|
-
When an evidence source already has a standard format, keep that format as the native artifact and reference it from `evidence.json`:
|
|
147
|
-
|
|
148
|
-
- SARIF: static analysis, security, code review, and policy findings.
|
|
149
|
-
- OpenTelemetry logs/traces: runtime behavior, tool/model calls, workflow telemetry, and post-deploy events.
|
|
150
|
-
- JUnit/TAP: test results.
|
|
151
|
-
- Veritas: optional evidence checks, repo standards, and authority settings. Flow Agents records the Veritas reference and verdict but does not own Veritas policy semantics.
|
|
152
|
-
|
|
153
|
-
Use `context/contracts/governance-adapter-contract.md` before invoking Veritas or any similar governance provider. If the adapter is unavailable, record `NOT_VERIFIED` unless the user explicitly accepts skipping that governance evidence.
|
|
154
|
-
|
|
155
|
-
### 6. Verdict
|
|
156
|
-
|
|
157
|
-
Produce:
|
|
158
|
-
|
|
159
|
-
- `PASS`: evidence satisfies risk and acceptance criteria.
|
|
160
|
-
- `FAIL`: evidence shows the work is wrong or unsafe.
|
|
161
|
-
- `NOT_VERIFIED`: evidence is missing, indirect, blocked, or inconclusive.
|
|
162
|
-
|
|
163
|
-
For failures, classify:
|
|
164
|
-
|
|
165
|
-
- implementation defect
|
|
166
|
-
- bad plan
|
|
167
|
-
- bad acceptance criteria
|
|
168
|
-
- flaky infrastructure
|
|
169
|
-
- missing environment
|
|
170
|
-
- security concern
|
|
171
|
-
- product ambiguity
|
|
172
|
-
- scope drift
|
|
173
|
-
|
|
174
|
-
Include required next evidence and whether to return to `plan-work`, `execute-plan`, `verify-work`, `remediate-ci`, or human decision.
|
|
175
|
-
|
|
176
|
-
### 7. Publish Change Gate
|
|
177
|
-
|
|
178
|
-
If the evidence verdict is otherwise `PASS` but the verified diff is not committed, pushed, and represented by a provider change record or an explicit no-provider-change reason, set `next_step` to `publish-change` instead of `release-readiness`.
|
|
179
|
-
|
|
180
|
-
Use `git` and the active `ChangeProvider` adapter when available to:
|
|
181
|
-
|
|
182
|
-
- confirm the working tree contains only the verified scope
|
|
183
|
-
- commit the verified diff with a clear message
|
|
184
|
-
- push the branch
|
|
185
|
-
- open or update the provider change record linked to the issue/brief, closing refs, and evidence artifact, or record why no provider change is required
|
|
186
|
-
- include or update the provider-facing `Acceptance Evidence` table, upgrading local source refs to immutable GitHub blob permalinks when the commit SHA and repository URL are known
|
|
187
|
-
- collect provider check/CI links and statuses, or record why provider checks are unavailable
|
|
188
|
-
- keep GitHub PRs as the first `ChangeProvider` adapter example: for GitHub, open or update a PR and collect PR checks
|
|
189
|
-
|
|
190
|
-
If commit, push, provider change publication, or provider checks are blocked, keep the release path at `NOT_VERIFIED` or `HOLD` until the blocker is resolved or explicitly accepted by the user.
|
|
191
|
-
|
|
192
|
-
### 8. Dependency And External-Audit Coverage
|
|
193
|
-
|
|
194
|
-
When dependency review is in scope, evidence-gate must preserve both local
|
|
195
|
-
inventory evidence and external advisory/audit evidence separately. External
|
|
196
|
-
audit commands and registry/advisory lookups may disclose private dependency
|
|
197
|
-
metadata; if the execution policy rejects the command or the user has not
|
|
198
|
-
explicitly approved that disclosure, record the affected roots as
|
|
199
|
-
`NOT_VERIFIED` for external audit and name the privacy/approval blocker.
|
|
200
|
-
|
|
201
|
-
For cross-product work, a dependency/security lane can pass only when every
|
|
202
|
-
applicable package-manager root has one of these recorded outcomes: `pass`,
|
|
203
|
-
`fail`, `skip_no_manifest`, or an accepted `not_verified` gap. Existing
|
|
204
|
-
vulnerabilities are still `FAIL` for the dependency lane unless the user
|
|
205
|
-
explicitly accepts them as unrelated residual risk.
|
|
206
|
-
|
|
207
|
-
## Gate
|
|
208
|
-
|
|
209
|
-
Evidence passes only when acceptance criteria, scope integrity, CI/runtime evidence, and residual risk are sufficient for the risk class.
|
|
210
|
-
|
|
211
|
-
For an active Builder Flow run, record merge readiness only after this skill reaches `PASS`:
|
|
212
|
-
|
|
213
|
-
```bash
|
|
214
|
-
npm run workflow:sidecar -- record-gate-claim .kontourai/flow-agents/<slug> \
|
|
215
|
-
--expectation merge-readiness \
|
|
216
|
-
--status pass \
|
|
217
|
-
--summary "Evidence gate passed: verified scope, acceptance evidence, review findings, and unresolved risks support provider review." \
|
|
218
|
-
--evidence-ref-json '{"kind":"artifact","file":".kontourai/flow-agents/<slug>/evidence.json","summary":"Structured evidence verdict and acceptance coverage."}'
|
|
219
|
-
```
|
|
220
|
-
|
|
221
|
-
Record `fail` or `not_verified` when the verdict is not `PASS`. The resulting trust bundle is evaluated by Flow and may route back to verification, execution, planning, or Probe according to the canonical definition.
|
|
222
|
-
|
|
223
|
-
After `PASS`, hand off to `publish-change` when the work is still local, or to `release-readiness` when the verified commit, pushed branch, provider change record or no-provider-change reason, provider checks, closing refs, structured evidence refs, and `Acceptance Evidence` table are available. After `FAIL` or `NOT_VERIFIED`, stop and name the missing work or evidence.
|
|
80
|
+
For a matching active run, publish that report as `merge-readiness`. This is a
|
|
81
|
+
merge-review confidence decision, not release or deployment authorization.
|