@manhquy/flow-skill 0.1.0-rc.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +60 -0
- package/LICENSE +21 -0
- package/README.md +152 -0
- package/README_VN.md +134 -0
- package/SECURITY.md +69 -0
- package/bin/cli.mjs +382 -0
- package/package.json +52 -0
- package/skills/flow/SKILL.md +234 -0
- package/skills/flow/_templates/00-idea.md +16 -0
- package/skills/flow/_templates/00-inspect.md +42 -0
- package/skills/flow/_templates/01-research.md +59 -0
- package/skills/flow/_templates/02-scope.md +71 -0
- package/skills/flow/_templates/03-prd.md +57 -0
- package/skills/flow/_templates/04-adr.md +21 -0
- package/skills/flow/_templates/05-contract.md +59 -0
- package/skills/flow/_templates/card.md +26 -0
- package/skills/flow/_templates/constitution.md +22 -0
- package/skills/flow/eval/fixtures/f01a/flow/01-research.md +77 -0
- package/skills/flow/eval/fixtures/f01b/flow/01-research.md +75 -0
- package/skills/flow/eval/fixtures/f02a/flow/02-scope.md +56 -0
- package/skills/flow/eval/fixtures/f02b/flow/02-scope.md +65 -0
- package/skills/flow/eval/fixtures/fcda/.flow/.gitkeep +0 -0
- package/skills/flow/eval/fixtures/fcda/cards/C-001.md +46 -0
- package/skills/flow/eval/fixtures/fcdb/.flow/.gitkeep +0 -0
- package/skills/flow/eval/fixtures/fcdb/cards/C-001.md +35 -0
- package/skills/flow/eval/manifest.tsv +7 -0
- package/skills/flow/harness/README.md +117 -0
- package/skills/flow/harness/_db.py +228 -0
- package/skills/flow/harness/_domain.py +161 -0
- package/skills/flow/harness/_presence.py +109 -0
- package/skills/flow/harness/flow_harness.py +1103 -0
- package/skills/flow/harness/repo_map.py +93 -0
- package/skills/flow/harness/schema/001-init.sql +129 -0
- package/skills/flow/harness/schema/002-story-verify.sql +9 -0
- package/skills/flow/harness/schema/003-tool-registry.sql +15 -0
- package/skills/flow/harness/schema/004-intervention.sql +15 -0
- package/skills/flow/harness/schema/005-tool-extensions.sql +20 -0
- package/skills/flow/harness/schema/009-accessed-count.sql +11 -0
- package/skills/flow/harness/schema/010-usage-event.sql +42 -0
- package/skills/flow/harness/schema/011-usage-gate-reason.sql +8 -0
- package/skills/flow/harness/schema/012-usage-ephemeral.sql +10 -0
- package/skills/flow/law/CLAUDE.md +140 -0
- package/skills/flow/law/DESIGN.md +141 -0
- package/skills/flow/law/RETRO.md +13 -0
- package/skills/flow/playbooks/README.md +28 -0
- package/skills/flow/playbooks/cloudflare-ai-gateway-qwen.md +346 -0
- package/skills/flow/playbooks/docker-deploy-stale-cache.md +32 -0
- package/skills/flow/playbooks/rag-lite-postgres-cloudflare.md +192 -0
- package/skills/flow/playbooks/windows-gitbash-jsonl-telemetry.md +57 -0
- package/skills/flow/references/adversarial-review.md +197 -0
- package/skills/flow/references/agent-detection.md +88 -0
- package/skills/flow/references/agent-stage-mapping.md +114 -0
- package/skills/flow/references/antigravity-integration.md +92 -0
- package/skills/flow/references/auto-run.md +81 -0
- package/skills/flow/references/claudekit-skills.md +128 -0
- package/skills/flow/references/codex-integration.md +104 -0
- package/skills/flow/references/command-dispatch.md +54 -0
- package/skills/flow/references/debt-and-halts.md +41 -0
- package/skills/flow/references/design-review-checklist.md +42 -0
- package/skills/flow/references/gate-eval.md +186 -0
- package/skills/flow/references/gate-rules.md +171 -0
- package/skills/flow/references/ground-truth-gates.md +49 -0
- package/skills/flow/references/loop-harness-2026-principles.md +52 -0
- package/skills/flow/references/mode-work.md +37 -0
- package/skills/flow/references/project-types.md +42 -0
- package/skills/flow/references/stage-state-machine.md +47 -0
- package/skills/flow/references/ui-patterns-tcr.md +49 -0
- package/skills/flow/runner/flow.cmd +18 -0
- package/skills/flow/runner/flow.sh +3297 -0
- package/skills-manifest.json +70 -0
- package/src/constants.mjs +65 -0
- package/src/detect.mjs +36 -0
- package/src/help.mjs +42 -0
- package/src/installer.mjs +231 -0
- package/src/prompt.mjs +60 -0
- package/src/tty.mjs +11 -0
|
@@ -0,0 +1,128 @@
|
|
|
1
|
+
# claudekit skills — the per-stage capability map (the seam)
|
|
2
|
+
|
|
3
|
+
`/flow` already orchestrates claudekit at the **agent layer** (`agent-detection.md` +
|
|
4
|
+
`agent-stage-mapping.md`: 13 ck: agents, ck:→bmad→built-in degrade). This file extends that
|
|
5
|
+
same seam to the **skill layer** — the curated set of ck *skills* worth reaching for at each
|
|
6
|
+
stage, beyond the wired agents. It is a **whitelist, not a dump**: the kit has ~87 skills and
|
|
7
|
+
~60% are marketing/content noise for a build harness. Only the skills below ever get surfaced.
|
|
8
|
+
|
|
9
|
+
This is the single source of truth for the skill map; `agent-stage-mapping.md` points here.
|
|
10
|
+
|
|
11
|
+
## The two rules that bind every skill suggestion
|
|
12
|
+
|
|
13
|
+
1. **A skill INFORMS a stage; the gate JUDGES it.** Same as the Codex/Antigravity rule
|
|
14
|
+
(`agent-detection.md`, `codex-integration.md`): a skill's output feeds your triage — it
|
|
15
|
+
**never auto-passes and never auto-fails** a gate. The `flow.sh` exit code + the
|
|
16
|
+
`gate-rules.md` challenge remain the only deciders. A skill that "looks good" is not a PASS.
|
|
17
|
+
2. **Detection is Claude-side, and degrades silently.** You already know the host Skills list
|
|
18
|
+
(the available-skills registry) — check it; do not assume. `flow.sh` **cannot** see which
|
|
19
|
+
skills are loaded (bash has no view of the registry, and the 5 install homes — Claude,
|
|
20
|
+
Codex, Antigravity ×2, project — differ), so skill detection is never put in the runner. A
|
|
21
|
+
missing skill **never lowers a gate** — it only changes whether the enrichment is offered.
|
|
22
|
+
Rich where the skill exists, unbroken where it doesn't.
|
|
23
|
+
|
|
24
|
+
## Cost gate — offer, don't impose
|
|
25
|
+
|
|
26
|
+
The high-value skills below are **opt-in-with-prompt**: at the matching gate you *offer* the
|
|
27
|
+
skill and the operator confirms — you do not auto-fire it on every stage (the constitution /
|
|
28
|
+
Codex cost-gate discipline: advisory, off the hot path, no token-tax on trivial work). Skip the
|
|
29
|
+
offer on a trivial decision/card. Never wire a skill into `cmd_next`/`cmd_check`.
|
|
30
|
+
|
|
31
|
+
## The whitelist — what to use when
|
|
32
|
+
|
|
33
|
+
Each skill is pinned to the stage where it pays off and the **distinct verb** it adds beyond the
|
|
34
|
+
already-wired agent. Pure skill/agent twins are deliberately absent (see "Don't surface").
|
|
35
|
+
|
|
36
|
+
| Skill | Stage | Distinct verb it adds (why it's not a twin) |
|
|
37
|
+
|---|---|---|
|
|
38
|
+
| **ck-predict** | ADR (04) | 5-persona pre-decision debate — no agent equivalent |
|
|
39
|
+
| **ck-scenario** | Scope/PRD/**Contract** (02–05) | 12-dimension edge-case → acceptance criteria + contract tests |
|
|
40
|
+
| **repomix** | Assess | packed single-file repo snapshot for LLM context |
|
|
41
|
+
| **ck-graphify** | Assess/Build | queryable knowledge graph (code+docs) for impact analysis |
|
|
42
|
+
| **review-pr** | Review/Ship | PR-context review: AI-slop, CI-blocker, breaking-change, `--fix` |
|
|
43
|
+
| **ck-security** | Review (security-class cards) | STRIDE+OWASP threat-model with attacker personas |
|
|
44
|
+
| **security-scan** | Review/Deploy | mechanical secrets/deps/OWASP scan (tip-level, cheap) |
|
|
45
|
+
| **retro** | Retro | git-history numeric retrospective (vs narrative journal) |
|
|
46
|
+
| **xia** | Research/Build | port/compare/adapt a feature from another repo |
|
|
47
|
+
| **ghpm** | Verify/Deploy | bind work to GitHub Issues/Projects/CI status |
|
|
48
|
+
| **deploy** | Deploy | platform auto-detect publish |
|
|
49
|
+
| **web-testing** | Verify-live | Playwright/k6 e2e/load/a11y (distinct from `tester` unit verb) |
|
|
50
|
+
| **docs-seeker** | Research/Build | fetch current library docs (llms.txt/context7) — version-drift guard |
|
|
51
|
+
| **scout** | Assess/any | fast parallel file discovery (distinct from inline read) |
|
|
52
|
+
| **ck-loop** | Build/Verify (Implement→Test→Audit→Fix) | mechanical iterate-to-numeric-target with per-iteration git commit/revert — the loop-engineering primitive flow lacked natively |
|
|
53
|
+
|
|
54
|
+
> Graph tool: **ck-graphify** is the chosen impact-analysis tool (operator decision, 2026-06-23
|
|
55
|
+
> — polyglot/doc-heavy brownfield fit, e.g. CMC Odoo). `gkg` is NOT wired; do not surface both.
|
|
56
|
+
|
|
57
|
+
## The 6 deep-wired skills (offered inside the gate ritual)
|
|
58
|
+
|
|
59
|
+
These are wired into the per-stage prose because each sits at a gate where a miss is most
|
|
60
|
+
expensive and adds a verb no wired agent provides. **Wired ≠ required**: each is offered
|
|
61
|
+
(opt-in-with-prompt), degrades silently, and only informs the gate. Wiring locations:
|
|
62
|
+
ck-predict/ck-scenario → `gate-rules.md` (stages 04/05); review-pr + ck-security →
|
|
63
|
+
`adversarial-review.md` (the Review gate); retro → `law/RETRO.md`; ck-loop → the Build/Verify
|
|
64
|
+
tail, plumbing via `flow.sh loop-prep`/`loop-log` (this file).
|
|
65
|
+
|
|
66
|
+
1. **ck-predict @ ADR** — before locking a non-trivial architecture decision, offer a 5-persona
|
|
67
|
+
debate. Catches arch/security/perf/UX defects when reversal is cheapest. Output informs the
|
|
68
|
+
ADR challenge; it does not pass it.
|
|
69
|
+
2. **ck-scenario @ Contract** (and Scope/PRD) — offer 12-dimension edge-case decomposition to
|
|
70
|
+
harden the seam (`flow/05-contract.md`): the cases become acceptance criteria + the per-type
|
|
71
|
+
no-drift checks. Complements `/flow consistency` (it *generates* cases; consistency checks
|
|
72
|
+
*coherence*).
|
|
73
|
+
3. **review-pr @ Review/Ship** — when the change lives as a GitHub PR, offer PR-context review
|
|
74
|
+
(AI-slop, CI-blocker, breaking-change) on top of the wired `code-reviewer` diff review.
|
|
75
|
+
4. **ck-security @ security-class cards** — on the exact card class flow already halts at
|
|
76
|
+
(auth, authz, tenancy, payments, data migration, removing validation), offer a STRIDE+OWASP
|
|
77
|
+
threat-model. It informs; the **Tier-C operator HALT is never auto-passed** by a clean scan.
|
|
78
|
+
5. **retro @ Retro** — offer a git-history numeric retrospective so the Retro line is backed by
|
|
79
|
+
real commit/velocity numbers, not prose.
|
|
80
|
+
6. **ck-loop @ Build/Verify (Implement→Test→Audit→Fix tail)** — when a fix needs more than one
|
|
81
|
+
experimental attempt against a single numeric target (failing-test count, lint errors, etc.),
|
|
82
|
+
offer `flow.sh loop-prep <card>` to set up an isolated worktree + Verify/Guard commands, then
|
|
83
|
+
invoke the `ck-loop` skill with the printed block (ck-loop stays the untouched execution
|
|
84
|
+
engine — flow supplies plumbing only). See "Loop vs two-strikes" below for when to reach for
|
|
85
|
+
this instead of the default repair path. The finished run is recorded via `flow.sh loop-log`
|
|
86
|
+
(NOT the `intervention` channel below — recording it twice would double-count the same event).
|
|
87
|
+
|
|
88
|
+
## Loop vs two-strikes — the one "fix it" decision tree
|
|
89
|
+
|
|
90
|
+
flow already has a bounded repair mechanism (two-strikes: one fresh-subagent repair attempt on a
|
|
91
|
+
review deadlock, then escalate to a cross-model lens — `adversarial-review.md`). `ck-loop` is a
|
|
92
|
+
different tool for a different situation; do not let operators reach for the wrong one:
|
|
93
|
+
|
|
94
|
+
| Situation | Use | Why |
|
|
95
|
+
|---|---|---|
|
|
96
|
+
| Review BLOCKED twice, same model, no metric | two-strikes → cross-model lens | bounded disagreement, not a number |
|
|
97
|
+
| One numeric verify command, needs >1 experimental attempt | `flow.sh loop-prep` + ck-loop | open iterate-to-target, git-tracked/revertable |
|
|
98
|
+
| Drive failing-tests / lint / perf count to a threshold | ck-loop | Verify is a single number; Direction lower |
|
|
99
|
+
| Single obvious fix, one retry | default auto repair | cheaper than spinning up a worktree+loop |
|
|
100
|
+
|
|
101
|
+
## Don't surface (cut on purpose — these add noise, not signal)
|
|
102
|
+
|
|
103
|
+
- **Skill/agent twins** — the wired agent already covers these; surfacing the duplicate skill
|
|
104
|
+
doubles the "what do I use" confusion: research/researcher, ck-code-review/code-reviewer,
|
|
105
|
+
ck-debug/debugger, test/tester, git/git-manager, ck-plan/planner, docs/docs-manager,
|
|
106
|
+
journal/journal-writer, ask, brainstorm. Prefer the **agent** for in-stage execution; reach
|
|
107
|
+
for the skill only for a distinct verb the agent lacks (already captured above).
|
|
108
|
+
- **Competing orchestrators (cook, vibe, ship, bootstrap)** — each is its own end-to-end
|
|
109
|
+
pipeline with its own gates; invoking one inside a flow stage double-gates and fights flow's
|
|
110
|
+
stage authority. Cherry-pick a sub-step (e.g. ship's merge/PR) at the matching stage only.
|
|
111
|
+
- **`worktree` skill** — flow already ships `flow.sh workspace` (git-worktree-per-agent). Pure
|
|
112
|
+
duplicate; never wire it.
|
|
113
|
+
- **`bmad-spec` as a gate** — overlaps flow's own `/flow consistency` cross-artifact audit.
|
|
114
|
+
Keep it only as the optional Contract drafter already named in `agent-stage-mapping.md`.
|
|
115
|
+
- **Marketing/content/media skills** (ads, seo, social, email, video, branding, copywriting,
|
|
116
|
+
the `ckm:*`/`bmad-*` marketing families, cti-expert) — never relevant to a build gate.
|
|
117
|
+
- **Stack adapters** (backend/frontend/db/auth/payment/mobile/ui-* …) — surface only via
|
|
118
|
+
`playbooks/<stack>.md` by project-type, never as a flat global menu.
|
|
119
|
+
|
|
120
|
+
## Lazy capture (ON — only at the 5 wired gates)
|
|
121
|
+
|
|
122
|
+
Skill-invocation telemetry is **enabled, lazily**: after a deep-wired skill runs at its gate,
|
|
123
|
+
record the use via the existing `flow.sh harness intervention add` (the same durable-metric
|
|
124
|
+
channel the Codex/Antigravity lenses use — `adversarial-review.md` §S2). **Only at the 5 wired
|
|
125
|
+
gates** — never on every skill, never on the `cmd_next`/`cmd_check` hot path, no new runner verb.
|
|
126
|
+
Note whether the skill caught a class the wired agent missed; that signal lets a later phase rank
|
|
127
|
+
the whitelist by what this operator actually reaches for (the `accessed_count` signal flow already
|
|
128
|
+
uses). The whitelist stays hand-curated until then.
|
|
@@ -0,0 +1,104 @@
|
|
|
1
|
+
# Codex — cross-vendor second engine (the seam)
|
|
2
|
+
|
|
3
|
+
`/flow` reaches OpenAI **Codex (GPT-5.x)** through the `openai-codex` Claude Code plugin
|
|
4
|
+
(detected, never required). Codex is a *second engine*, not a replacement for the ck:→bmad→
|
|
5
|
+
built-in ladder: a genuinely different model used at the moments where that is worth more than
|
|
6
|
+
another Claude pass. This file is the single source of truth for the seam; `agent-detection.md`
|
|
7
|
+
(detection + priority), `agent-stage-mapping.md` (per-stage use), `adversarial-review.md`
|
|
8
|
+
(cross-model review), and `auto-run.md` (Tier-B escalation) point here for the shapes.
|
|
9
|
+
|
|
10
|
+
## Why a second vendor (the measured gap)
|
|
11
|
+
|
|
12
|
+
A single-vendor harness makes the builder and the reviewer share one model → correlated blind
|
|
13
|
+
spots pass green gates (this project's own review-pass-#4 caught an auth-skip a same-model pass
|
|
14
|
+
missed — `docs/quality-metrics.md`). Cross-model review empirically catches markedly more (a
|
|
15
|
+
gemini-cli study: single-model review 43% merge-ready vs cross-model iteration 91%). A different
|
|
16
|
+
engine is the cheapest way to close a same-vendor gate without weakening any gate.
|
|
17
|
+
|
|
18
|
+
## Detection (I1) — two states: INSTALLED vs USABLE (never route on "installed" alone)
|
|
19
|
+
|
|
20
|
+
Detection has **two states**, because "installed" does not mean "works" — a plugin can be present
|
|
21
|
+
with no valid auth (common in headless/CI — see Codex issue #9253) or a broken companion script.
|
|
22
|
+
Routing into Codex on mere presence would spend a billable repair/review attempt and then fail at
|
|
23
|
+
invocation — that violates "absence never breaks a run". So:
|
|
24
|
+
|
|
25
|
+
- **INSTALLED** — `codex:codex-rescue` is in the host agent registry, OR the plugin dir exists
|
|
26
|
+
(`~/.claude/plugins/cache/openai-codex` / `.../marketplaces/openai-codex`). Necessary, not sufficient.
|
|
27
|
+
- **USABLE** — INSTALLED **and** a cheap, non-billable liveness/auth check passes:
|
|
28
|
+
`codex-companion.mjs setup --json` reports `ready === true` AND `auth.loggedIn === true`
|
|
29
|
+
(or a confirmed prior successful Codex call this session). Run this check **before the first
|
|
30
|
+
time** you would route work to Codex. NOTE: use `setup --json`, NOT `status` — `status` reports
|
|
31
|
+
runtime/job state but carries **no auth field**, so it passes even when Codex is unauthenticated
|
|
32
|
+
(it would route then fail at the real call). `setup --json` is the auth-aware, non-billable probe.
|
|
33
|
+
|
|
34
|
+
**Selection rule:** only select the Codex tier when state = **USABLE**. If INSTALLED-but-not-USABLE
|
|
35
|
+
(or absent): **degrade** to ck:→bmad→built-in, announce `"codex tier unavailable (installed but
|
|
36
|
+
not authenticated / not reachable) — degraded to <path>"`, and record the reason (durable metric).
|
|
37
|
+
Never select Codex on INSTALLED alone. Degrading is silent-but-announced, never an error, never a
|
|
38
|
+
gate change. Downstream docs (`agent-detection.md`, `auto-run.md`, `adversarial-review.md`) all
|
|
39
|
+
mean **USABLE** wherever they say "eligible".
|
|
40
|
+
|
|
41
|
+
## Cost gate (F-D) — when Codex is allowed to fire
|
|
42
|
+
|
|
43
|
+
Codex calls are **billable** external GPT-5.x. Fire them ONLY at high-value moments:
|
|
44
|
+
- a **two-strikes deadlock** (a same-model agent BLOCKED twice),
|
|
45
|
+
- a **security-class card review** (auth/authorization/tenancy/payments/data-migration), or
|
|
46
|
+
- an **explicit operator opt-in** (e.g. "draft this stage on Codex", "/flow ... codex").
|
|
47
|
+
|
|
48
|
+
Never call Codex on every stage by default. Default engine stays ck:.
|
|
49
|
+
|
|
50
|
+
## Invocation surfaces (from flow/05-contract.md — do not improvise other shapes)
|
|
51
|
+
|
|
52
|
+
| Use | Invocation | Effects | Returns |
|
|
53
|
+
|---|---|---|---|
|
|
54
|
+
| **Handoff / rescue / primary-draft** | `Task(subagent_type="codex:codex-rescue", prompt=<ScopedBrief>)` | writes code (`--write` default), billable | stdout + status line |
|
|
55
|
+
| **Cross-model review** | `node "$CODEX/scripts/codex-companion.mjs" review\|adversarial-review [--wait\|--background] [--base <ref>] [--scope auto\|working-tree\|branch] [focus]` | read-only review, billable | `ReviewResult` JSON |
|
|
56
|
+
| **Task (non-interactive)** | `node "$CODEX/scripts/codex-companion.mjs" task [--background] [--write] [--model <m\|spark>] [--effort <none..xhigh>] <prompt>` | writes if `--write`, billable | stdout + job id |
|
|
57
|
+
| **Job control** | `... status [job-id] [--all] [--json]` · `result [job-id] [--json]` · `cancel [job-id]` | read-only / cancel | job/result JSON |
|
|
58
|
+
|
|
59
|
+
`$CODEX` = the plugin root (`~/.claude/plugins/cache/openai-codex/codex/<ver>`). Prefer the
|
|
60
|
+
`codex:codex-rescue` **subagent** for handoffs (it owns prompt-shaping + foreground/background
|
|
61
|
+
choice); use `codex-companion.mjs` directly only for `review`/`adversarial-review`/job-control.
|
|
62
|
+
|
|
63
|
+
### ScopedBrief (context isolation — orchestration-protocol)
|
|
64
|
+
Give Codex ONLY: `task` (one goal) · `read_for_context` (contract, law, the card, playbook) ·
|
|
65
|
+
`files_to_modify` (the card's Allowed files ONLY) · `acceptance` (the stage gate / card Verify) ·
|
|
66
|
+
`constraints` (contract is the seam; Vietnamese user-facing copy; touch only allowed files) ·
|
|
67
|
+
`return` (artifact + status). **Never** the session history.
|
|
68
|
+
|
|
69
|
+
### ReviewResult (verbatim from `schemas/review-output.schema.json`)
|
|
70
|
+
```
|
|
71
|
+
{ verdict: "approve"|"needs-attention", summary: string,
|
|
72
|
+
findings: [ { severity:"critical"|"high"|"medium"|"low", title, body,
|
|
73
|
+
file, line_start>=1, line_end>=1, confidence:0..1, recommendation } ],
|
|
74
|
+
next_steps: [ string ] }
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
### Status protocol (handoff return)
|
|
78
|
+
`DONE | DONE_WITH_CONCERNS | BLOCKED | NEEDS_CONTEXT` — handle BLOCKED/NEEDS_CONTEXT before retry.
|
|
79
|
+
|
|
80
|
+
## Gate parity (absolute) & auth (delegated)
|
|
81
|
+
|
|
82
|
+
- **Gate parity.** Codex DRAFTS or CRITIQUES; the identical stage gate (`flow.sh` + `gate-rules.md`)
|
|
83
|
+
still judges. A cross-model review INFORMS triage — it never auto-passes or auto-fails a card.
|
|
84
|
+
- **Auth.** Delegated entirely to the plugin (`codex login` / `OPENAI_API_KEY` / ChatGPT
|
|
85
|
+
subscription). `/flow` never reads, stores, or logs Codex credentials.
|
|
86
|
+
- **Data boundary (the other half of trust).** Selecting Codex SENDS the ScopedBrief — the source
|
|
87
|
+
diff plus the contract / PRD / law excerpts — to OpenAI's API, governed by the operator's OpenAI
|
|
88
|
+
plan and its data-retention/training terms. This is distinct from the credential clause above:
|
|
89
|
+
even with perfect secret handling, the *code and specs* leave the machine. For a sensitive,
|
|
90
|
+
regulated, or NDA'd codebase, the operator must opt in knowingly; the cost gate (high-value
|
|
91
|
+
moments only) keeps the exposure surface small by default.
|
|
92
|
+
|
|
93
|
+
## Durable metric (S2)
|
|
94
|
+
|
|
95
|
+
After a Codex tier use, log it so the quality loop can read the trend (no new schema):
|
|
96
|
+
`flow.sh harness intervention add` (e.g. cross-model review found a class same-model missed) or
|
|
97
|
+
`intake` / `decision add`. This is the "thông số" that drives skill upgrades — see
|
|
98
|
+
`docs/quality-metrics.md` (cross-model-catch rate, rescue deflection rate).
|
|
99
|
+
|
|
100
|
+
## Announce the path
|
|
101
|
+
|
|
102
|
+
Always tell the operator which engine ran: `"review via Codex cross-model lens (needs-attention,
|
|
103
|
+
2 findings)"` / `"rescue via codex:codex-rescue after two strikes"` / `"codex tier unavailable —
|
|
104
|
+
degraded to code-reviewer"`. A run must stay legible.
|
|
@@ -0,0 +1,54 @@
|
|
|
1
|
+
# Command dispatch
|
|
2
|
+
|
|
3
|
+
Exact mapping: user input -> runner call -> your (Claude) duties. Always run the runner
|
|
4
|
+
first, relay its output faithfully, then do the semantic part. `<skill>` = this skill's
|
|
5
|
+
install dir; run from the project root so `flow/` and `cards/` resolve.
|
|
6
|
+
|
|
7
|
+
| User input | Runner call | Your duties after |
|
|
8
|
+
|---|---|---|
|
|
9
|
+
| `/flow` | `bash <skill>/runner/flow.sh status` | Runner now prints its own `NEXT ->` line (same helper as `resume`) plus stage dwell and, past 10 cards, a compact done/in-flight/todo summary. Relay it; nothing to author. |
|
|
10
|
+
| `/flow resume` | `bash <skill>/runner/flow.sh resume` | Read-only, no lock. Relay all four sections verbatim (last session, in-flight + dwell, gate state, NEXT ->). Run this FIRST when entering a project mid-cycle with no prior context this session. Never expand on or guess at raw args it didn't print — it deliberately shows command names only. |
|
|
11
|
+
| `/flow next` | `bash <skill>/runner/flow.sh next` | If FAIL: relay the exact violations + line numbers; offer help, don't author/tick in teach mode. If PASS: run the stage's challenge in `gate-rules.md`; flag hollow content, let operator decide. |
|
|
12
|
+
| `/flow assess` | `bash <skill>/runner/flow.sh assess` | Brownfield only: fill `flow/00-inspect.md` from EVIDENCE (read the code) — functionality/UI/UX vs product, risks, test baseline. Reuse `scout`/`researcher`. Gate is operator-reviewed; then proceed to `/flow next`. |
|
|
13
|
+
| `/flow card` | `bash <skill>/runner/flow.sh card` | Confirm the new card id; remind: fill Scope (one thing) / Allowed files / Verify / Done-evidence per `law/CLAUDE.md`. |
|
|
14
|
+
| `/flow card start C-NNN` | `bash <skill>/runner/flow.sh card start C-NNN` | Optional: mark the card in flight (operator-visible in-progress, shown in `/flow status`). Portable `cards/.inflight` registry; does NOT touch the gated `status:` field. Coexists with hand-edit. |
|
|
15
|
+
| `/flow card done C-NNN` | `bash <skill>/runner/flow.sh card done C-NNN` | Optional convenience: CLI-owned flip to `done`, gated by the SAME done-rules as `check` (reverts on fail — never a hollow done). Then still apply the semantic card review (diff-vs-scope, contract shapes, real evidence) as with `check`. |
|
|
16
|
+
| `/flow check C-NNN` | `bash <skill>/runner/flow.sh check C-NNN` | If PASS mechanically: review diff-vs-scope, allowed-files drift, contract-shape match, DESIGN.md for UI, evidence = real world-state. |
|
|
17
|
+
| `/flow contract` | `bash <skill>/runner/flow.sh contract` | After the contract gate / before UI cards (web): flags client base-URL vs served-path prefix drift (double-`/api`, mixed-prefix) — the class spec-diff tools miss. Advisory; confirm on the running app. |
|
|
18
|
+
| `/flow tokens` | `bash <skill>/runner/flow.sh tokens` | On/after UI cards: flags DESIGN.md tokens the CSS never uses + VALUE mismatches (same name, drifted value) + orphan CSS vars (info). Advisory; if the swap is intentional, record a dated DESIGN.md amendment. |
|
|
19
|
+
| `/flow coherence` | `bash <skill>/runner/flow.sh coherence` | Flags version drift across declared version fields (package.json / pyproject / src app_version). The cheap doc-vs-code slice; semantic contradictions stay a human gate-challenge. |
|
|
20
|
+
| `/flow consistency` | `bash <skill>/runner/flow.sh consistency` | After the contract gate, before cards: every PRD `FRn` is claimed by a card (`implements:`) and served by a contract interface; numeric success metric; no leftover placeholders. Then apply the semantic cross-artifact passes in `gate-rules.md` (hollow coverage, conflicting requirements, terminology drift). Advisory. |
|
|
21
|
+
| `/flow constitution` | `bash <skill>/runner/flow.sh constitution` | At the scope/PRD/contract seam: checks `flow/constitution.md` is well-formed (no placeholder, every invariant has an ID) + scans declared grep-markers. **Advisory, NOT a `next` gate** — then apply the semantic challenge in `gate-rules.md`. |
|
|
22
|
+
| `/flow eval [--stage 01\|02\|card] [--fixture <id>] [--n 3] [--timeout <s>]` | `bash <skill>/runner/flow.sh eval [...]` | **Billable** (skips cleanly, zero calls, if `claude` CLI absent). Relay the scorecard verbatim. See `references/gate-eval.md` before running: what it measures (a fresh-judge lower bound, not the work-mode self-challenge), cost, and thresholds. Never wired into `next`/`check` gating — informative only. |
|
|
23
|
+
| `/flow eval --report` | `bash <skill>/runner/flow.sh eval --report` | Offline, zero calls. Relay the last complete batch's scorecard + drift vs the prior complete batch. |
|
|
24
|
+
| `/flow project-type <web\|cli\|library\|skill>` | `bash <skill>/runner/flow.sh project-type [t]` | Set/read the project type that selects the per-type gate lens + done-evidence definition (`references/project-types.md`). Confirm the type matches reality before planning. |
|
|
25
|
+
| `/flow usage [--global\|--prune]` | `bash <skill>/runner/flow.sh usage [...]` | Roll up the JSONL usage log and relay the analytics (cycle-time, gate fail-rate, per-stage + per-card dwell, command breakdown). Local-only; `--prune [--keep N]` caps the log. Read-only — nothing to author. |
|
|
26
|
+
| `/flow skip <stage> --reason ...` | `bash <skill>/runner/flow.sh skip <stage> --reason ...` | Advance past a gate that has a matching open `DEBT.md` line. **Security-class skips are operator-only and HALT** — never auto-skip; the operator accepts the exposure in writing first. |
|
|
27
|
+
| `/flow debt add\|list` | `bash <skill>/runner/flow.sh debt add\|list` | Record/list deliberate gate-skips in `DEBT.md`. Security-class debt is operator-authored only; surface it at retro and before any real user touches the surface. |
|
|
28
|
+
| `/flow design <file>` | `bash <skill>/runner/flow.sh design <file>` | Mechanical DESIGN.md check on a UI file (emoji / `{{}}` / engine-words / gradient). Advisory; pair with the semantic `design-review-checklist.md` for mock/frontend cards. |
|
|
29
|
+
| `/flow harness <args>` | `bash <skill>/runner/flow.sh harness <args>` | Passthrough to the durable-layer CLI (intake/story/trace/decision/backlog/query). Visible output + real exit code. Use to write the durable hook after a stage/card; read it back with `recall`. |
|
|
30
|
+
| `/flow doctor` | `bash <skill>/runner/flow.sh doctor` | Environment/install self-check (bash/python/grep/git across macOS/Linux/Windows, runner path, Git Bash). Relay any FAIL as the fix list. Read-only. |
|
|
31
|
+
| `/flow promote <file>` | `bash <skill>/runner/flow.sh promote <file>` | Copy a hard-won playbook into the cross-project KB (`~/.claude/flow/playbooks`) so its lesson is surfaced by `recall` in every project, not just this one. |
|
|
32
|
+
| `/flow mode teach` | `bash <skill>/runner/flow.sh mode teach` | Confirm; you only gatekeep, operator authors. |
|
|
33
|
+
| `/flow mode work` | `bash <skill>/runner/flow.sh mode work` | Interview once, draft 00-05, pause for scope sign-off, deliver card set; still pass every gate. |
|
|
34
|
+
| `/flow ready` | `bash <skill>/runner/flow.sh ready` | Relay buildable cards; confirm allowed-files truly don't overlap before suggesting parallel. Operator dispatches. |
|
|
35
|
+
| `/flow workspace <verb>` | `bash <skill>/runner/flow.sh workspace add\|list\|enter\|remove\|check\|doctor [...]` | Multi-agent worktree isolation (human-driven, cross-vendor). Relay the runner output faithfully: on `add`, hand the operator the printed cd/env block (one worktree per agent — Claude `--worktree`/`-w`, Codex CLI manual + `CODEX_HOME`, Antigravity = open the dir as a workspace); on `check`/`doctor` exit 1, surface the collision/drift, don't auto-`--force` a `remove`. git is the source of truth; the `.flow/workspaces.jsonl` side-file only adds vendor/card/port/task. Advisory — not a `next` gate. |
|
|
36
|
+
| `/flow auto` | `bash <skill>/runner/flow.sh auto` | On preflight PASS, drive the autonomous run per SKILL.md AUTO principles (subagent/card, planner review, worktree, Tier-C halt on security debt). |
|
|
37
|
+
| `/flow recall` | `bash <skill>/runner/flow.sh recall` | Read back prior knowledge (open debt, recent retro, previous-card scope, harness friction/backlog, playbooks). Run at the START of a stage/card; apply it, don't re-learn known pain. |
|
|
38
|
+
| `/flow unlock` | `bash <skill>/runner/flow.sh unlock` | Clear this project's concurrency lock after a crashed/abandoned session. Confirm the other session is really gone first. |
|
|
39
|
+
| `/flow retro` | `bash <skill>/runner/flow.sh retro` | Ask the 3 questions; the operator writes the RETRO.md line — never you. |
|
|
40
|
+
|
|
41
|
+
## Behavioral invariants (all commands)
|
|
42
|
+
1. The runner's exit code is ground truth. Don't override it with optimism.
|
|
43
|
+
2. In `teach` mode never tick a box or write an artifact for the operator.
|
|
44
|
+
3. Never set a card `done` without pasted world-state evidence.
|
|
45
|
+
4. Read any file the runner just created before editing it.
|
|
46
|
+
5. Never edit `_templates/` or `runner/flow.sh` during a project run.
|
|
47
|
+
6. Relay failures verbatim (line numbers included) — they are the operator's to-do list.
|
|
48
|
+
7. If the runner reports BLOCKED by another session's lock, STOP and coordinate — never `FLOW_FORCE` past a live session; concurrent runs corrupt the plan. Set `FLOW_SESSION_ID` per session for hard protection.
|
|
49
|
+
8. At the start of a stage or build card, run `/flow recall` first — its output (prior debt / retro / friction / previous-card scope) is context to apply, not noise. `status` shows a one-line memory summary; `card` injects the previous card's scope automatically.
|
|
50
|
+
|
|
51
|
+
## Install-path note
|
|
52
|
+
- Project install: `bash .claude/skills/flow/runner/flow.sh <cmd>`
|
|
53
|
+
- Global install: `bash ~/.claude/skills/flow/runner/flow.sh <cmd>`
|
|
54
|
+
- Override project root: `FLOW_PROJECT_ROOT=/path bash <skill>/runner/flow.sh <cmd>`
|
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
# Debt & halts
|
|
2
|
+
|
|
3
|
+
Reordering or skipping a gate is a legitimate OPERATOR call (demo-first, riskiest-first).
|
|
4
|
+
But a skipped gate is a loan, and loans get written down. (buildflow CLAUDE.md "Debt".)
|
|
5
|
+
|
|
6
|
+
## The DEBT.md ledger
|
|
7
|
+
Every deliberate skip opens one line (create `DEBT.md` on first use). Use the runner:
|
|
8
|
+
```
|
|
9
|
+
bash <skill>/runner/flow.sh debt add "<what was skipped>" "<the exposure, concretely>" "<close before: named condition>"
|
|
10
|
+
bash <skill>/runner/flow.sh debt list # open debts
|
|
11
|
+
```
|
|
12
|
+
Line shape:
|
|
13
|
+
```
|
|
14
|
+
- [ ] DEBT: <what was skipped> -- <exposure> -- close before: <named condition> -- opened <date> (cards: C-NNN...)
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
## Security-class skips (never silent, never planner-decided)
|
|
18
|
+
auth · authorization · admin-surface exposure · tenancy · payments · data loss/migration ·
|
|
19
|
+
removing/weakening validation.
|
|
20
|
+
- The OPERATOR explicitly accepts the exposure, in writing, in the DEBT line. You do not
|
|
21
|
+
decide it for them.
|
|
22
|
+
- In `/flow auto` this is a **Tier-C HALT** — stop and ask.
|
|
23
|
+
- Closing a run with open security debt requires explicit operator acknowledgment.
|
|
24
|
+
"Temporary" is one forgotten step from production.
|
|
25
|
+
|
|
26
|
+
## Close conditions
|
|
27
|
+
Checked at every `/flow retro` and before ANY real user touches the build. A card blocked
|
|
28
|
+
by a debt stays `todo` with PARTIAL evidence naming the debt — never half-done, never
|
|
29
|
+
rounded up to done.
|
|
30
|
+
|
|
31
|
+
## Halts (when the run stops itself)
|
|
32
|
+
A run HALTS and reports — never silently continues — on any of:
|
|
33
|
+
- A hard-stop cap exceeded (iterations/tokens/wall-clock). (`loop-harness-2026-principles.md`)
|
|
34
|
+
- A red ground-truth signal that can't be repaired in two strikes. (`adversarial-review.md`)
|
|
35
|
+
- A Tier-C security-class touch. (above)
|
|
36
|
+
- A merge conflict during parallel builds (the allowed-files overlap check was gamed —
|
|
37
|
+
stop and re-plan). (`auto-run.md`)
|
|
38
|
+
- `BLOCKED` / `NEEDS_CONTEXT` from a subagent that more context can't resolve.
|
|
39
|
+
|
|
40
|
+
On halt: state what stopped it, what's done so far, the open debt/blocker, and the 2-4
|
|
41
|
+
concrete options for the operator. Let them decide — don't patch around a regression.
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
# Design review checklist (UI cards)
|
|
2
|
+
|
|
3
|
+
Every mock/frontend card is reviewed against `law/DESIGN.md` the same way backend cards are
|
|
4
|
+
reviewed against the contract. Two parts: a **mechanical** part (`flow.sh design <file>`,
|
|
5
|
+
grep-able, deterministic) and a **semantic** part (you, against DESIGN.md's structure laws).
|
|
6
|
+
|
|
7
|
+
## Mechanical (run `flow.sh design <htmlfile>` — catches the never-do list)
|
|
8
|
+
Flags, with line numbers:
|
|
9
|
+
- **Emoji** anywhere (DESIGN.md: "No emojis. Anywhere. Ever.").
|
|
10
|
+
- **Raw `{{ }}` templates or visible JSON** outside a power surface.
|
|
11
|
+
- **Gradients on inputs / table rows / body bg** (gradients are hero-surface only).
|
|
12
|
+
- **Engine words** in user-facing copy: workflow, trigger, queue, webhook, agent, prompt,
|
|
13
|
+
cron, payload, endpoint (users think in THEIR objects, not engine concepts).
|
|
14
|
+
- **Design comments left in HTML.**
|
|
15
|
+
A flag is a strong signal, not always a verdict (an engine word inside a `<code>` sample is
|
|
16
|
+
fine) — you confirm. Exit 1 if any flag fires.
|
|
17
|
+
|
|
18
|
+
## Semantic (you review against DESIGN.md — the script can't judge these)
|
|
19
|
+
- **Object-first:** the home page of a thing IS the thing; tabs are lenses on one object,
|
|
20
|
+
not separate features.
|
|
21
|
+
- **WYSIWYG / edit-in-place:** 80% of edits inline on the object page; a separate edit page
|
|
22
|
+
only for structural 20%. No multi-step wizard for editing.
|
|
23
|
+
- **Defaults beat configuration:** creatable in <=6 visible fields; the rest behind "More options".
|
|
24
|
+
- **Plain language:** "4 days after it ends", a field-picker chip — never cron, never raw templates.
|
|
25
|
+
- **Power behind a door:** any power surface is a `Simple | Pro` toggle that loses no data,
|
|
26
|
+
with a visible path back to simple.
|
|
27
|
+
- **Affordance ladder:** the field uses the lightest rung its shape allows (inline text ->
|
|
28
|
+
inline control -> popover composite -> modal). Empty value renders as `+ Add {label}`.
|
|
29
|
+
- **Luma object page:** calm pulse strip (no stat-tile cards), <=3 gradient-tinted hero
|
|
30
|
+
action cards (one click, no kebab), tabs as lenses, modal-first sub-actions.
|
|
31
|
+
- **Tokens:** Editorial Minimal tokens (or a deliberate, documented, whole-cluster override).
|
|
32
|
+
Inter body / Fraunces only for h1+titles+prominent stats / JetBrains Mono only for
|
|
33
|
+
IDs/dates/counts. 1px borders, no stacked shadows.
|
|
34
|
+
- **VN conventions:** Vietnamese native copy, `₫750,000` (symbol leading, comma groups),
|
|
35
|
+
VietQR as default pay, Zalo as a first-class support channel.
|
|
36
|
+
|
|
37
|
+
## Process
|
|
38
|
+
1. The UI **mock card** (static HTML, real copy, no logic) IS the design review — render it,
|
|
39
|
+
operator approves in a browser. Iterate here (mock retries cost seconds; framework retries
|
|
40
|
+
cost deploys). No frontend code before the mock is approved.
|
|
41
|
+
2. Run `flow.sh design <file>` (mechanical) + this semantic pass on the mock and on every
|
|
42
|
+
frontend card. A violation blocks the card the same as a failing `## Verify`.
|
|
@@ -0,0 +1,186 @@
|
|
|
1
|
+
# Gate eval — behavioral proof for the semantic layer (`flow.sh eval`)
|
|
2
|
+
|
|
3
|
+
`gate-rules.md` is the semantic layer: after the mechanical gate passes, an LLM applies a
|
|
4
|
+
per-stage challenge and can flag a mechanically-clean artifact as substantively hollow. Until
|
|
5
|
+
this verb existed, nothing measured whether that actually happens — a hollow-but-well-formed
|
|
6
|
+
artifact passes the mechanical gate BY DESIGN (`flow.sh:scan_gate`/`cmd_check`), and the
|
|
7
|
+
semantic layer's only proof was operator trust. `flow.sh eval` runs the real per-stage ritual
|
|
8
|
+
text against curated hollow/sound fixture pairs and majority-votes a nonce-protected verdict,
|
|
9
|
+
so gate erosion (CLI update, model swap, prose edit) shows up as a number, not a vibe.
|
|
10
|
+
|
|
11
|
+
## What it measures — and the honest limit on what it does NOT
|
|
12
|
+
|
|
13
|
+
The eval judge is a **fresh** LLM call: it receives the gate-rules.md challenge text plus one
|
|
14
|
+
fixture's content, cold, with no memory of authoring anything. This proves a **necessary, not
|
|
15
|
+
sufficient**, lower bound: if a fresh judge can't catch an obvious hollow artifact when
|
|
16
|
+
explicitly asked to, the semantic layer has failed outright. A green scorecard proves that
|
|
17
|
+
floor holds — it does **not** prove the semantic layer works during a real build.
|
|
18
|
+
|
|
19
|
+
The gap: in a real `/flow` session, the SAME model that just authored a stage artifact is the
|
|
20
|
+
one applying the challenge to it (`gate-rules.md:11` — "in `work` mode you authored it, so you
|
|
21
|
+
self-challenge before presenting"). That is a self-review, with the sunk-cost/authorship bias a
|
|
22
|
+
fresh judge never has. This eval does not reproduce that condition, and closing that gap is out
|
|
23
|
+
of scope for v1 (recorded design boundary, not an oversight). Read a green scorecard as "the
|
|
24
|
+
rule text is enforceable by *a* judge," not "this build's self-review caught everything."
|
|
25
|
+
|
|
26
|
+
## Fixtures and thresholds
|
|
27
|
+
|
|
28
|
+
Six shipped fixtures (`skills/flow/eval/fixtures/`, manifest `skills/flow/eval/manifest.tsv`):
|
|
29
|
+
sound/hollow pairs for Stage 01 (Research — fabricated quotes, vague "users"), Stage 02 (Scope —
|
|
30
|
+
grade laundering, a real C quietly graded B), and the card gate ("merge ≈ shipped" evidence).
|
|
31
|
+
Fixture content maps 1:1 to the failure modes already documented in `gate-rules.md`, not an
|
|
32
|
+
invented taxonomy. The v1 input set is **manifest-listed shipped fixtures only** — `eval` never
|
|
33
|
+
accepts a caller-supplied artifact path; widening that is a v2 change that reopens security
|
|
34
|
+
review (an unvetted artifact reaching an LLM judge is a different trust boundary).
|
|
35
|
+
|
|
36
|
+
Per fixture, N=3 judge calls (`--n`), majority vote: **healthy = hollow fixtures flagged at
|
|
37
|
+
≥2/3 majority, sound fixtures passed at majority**. More than 1/3 of a fixture's runs coming
|
|
38
|
+
back unparseable/timed-out makes that fixture's result `UNRELIABLE` (an infra failure, reported
|
|
39
|
+
separately from a real mismatch — never silently counted as a gate pass or fail).
|
|
40
|
+
|
|
41
|
+
## Cost
|
|
42
|
+
|
|
43
|
+
`eval` is **opt-in and billable** — it makes real `claude -p` API calls. Zero cost when the
|
|
44
|
+
`claude` CLI isn't on `PATH` (clean skip, exit 0). When present, exactly one minimal probe call
|
|
45
|
+
is made to confirm the CLI runs headless before any real judging starts; a probe failure means
|
|
46
|
+
one billable call was made, then a clean skip. A full default batch is 6 fixtures × N=3 = 18
|
|
47
|
+
judge calls + the 1 probe = 19 calls.
|
|
48
|
+
|
|
49
|
+
**Measured real cost** (this machine, 2026-07-10, default/unforced model, no `--bare` available
|
|
50
|
+
under an OAuth/subscription session — see the build's spike notes): roughly **$0.30–0.37 per
|
|
51
|
+
call**, driven almost entirely by this repo's large global `CLAUDE.md` + skill/agent/MCP
|
|
52
|
+
declarations loading as system-prompt context on every call (~50–60K cache-creation tokens),
|
|
53
|
+
not by the judging prompt itself. A full 19-call baseline batch is real money, order **$6–7**
|
|
54
|
+
on this setup — not the trivial "a token or two" a casual reader might assume from "opt-in".
|
|
55
|
+
Forcing a cheaper model (`--model haiku` on the underlying CLI, not currently plumbed as a flow
|
|
56
|
+
flag) cuts this roughly 5x, but changes what the eval measures (a cheaper, possibly weaker judge)
|
|
57
|
+
and was deliberately left as an operator choice rather than a default, since the point of a
|
|
58
|
+
fresh-judge eval is to reflect a realistic judge, not the cheapest one that passes.
|
|
59
|
+
|
|
60
|
+
`--report` (below) is the free, offline way to re-read a prior batch's numbers without spending
|
|
61
|
+
anything.
|
|
62
|
+
|
|
63
|
+
## Running it
|
|
64
|
+
|
|
65
|
+
```
|
|
66
|
+
flow.sh eval # full manifest, N=3 per fixture
|
|
67
|
+
flow.sh eval --stage card # only the card-gate fixtures
|
|
68
|
+
flow.sh eval --fixture fcdb --n 1 # one fixture, one run (cheap smoke check)
|
|
69
|
+
flow.sh eval --timeout 180 # override the per-call timeout (default 120s)
|
|
70
|
+
flow.sh eval --report # OFFLINE: last complete batch's scorecard + drift
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
Exit code: `0` = every evaluated fixture majority-matched its expected verdict (or a clean
|
|
74
|
+
skip); `1` = any mismatch, any `UNRELIABLE` fixture, or nothing matched the given filters. This
|
|
75
|
+
exit code is **informative only** — it is never wired into `next`/`check` gating. A red `eval`
|
|
76
|
+
run does not block a build; it tells the operator the semantic layer needs attention.
|
|
77
|
+
|
|
78
|
+
## Results, scorecard, and drift
|
|
79
|
+
|
|
80
|
+
Every batch appends to `$FLOW_PROJECT_ROOT/.flow/eval-results.jsonl` (per-fixture rows plus a
|
|
81
|
+
`batch:"start"`/`batch:"done"` header/trailer pair). A batch with no trailer — Ctrl-C mid-run,
|
|
82
|
+
the only signal handled is best-effort (see Limitations) — is simply excluded from `--report`
|
|
83
|
+
and drift comparisons; that absence of a trailer IS the completeness signal, not a positive
|
|
84
|
+
"torn" flag. Reruns append a new `run_id`; there is no in-place dedup or pruning (personal
|
|
85
|
+
volume, same stance as the existing global usage-log rotation backlog — documented, not built).
|
|
86
|
+
|
|
87
|
+
Each result row carries `cli_version` (from `claude --version` — the CLI's own version string,
|
|
88
|
+
**not** a model id) and `model` (the served model id, parsed from the JSON response's own
|
|
89
|
+
`model` field — falls back to `"unknown"` if that source is ever unavailable; the two are kept
|
|
90
|
+
strictly separate so a CLI update is never mistaken for a model swap or vice versa), plus
|
|
91
|
+
`flow_version` and a CRLF-normalized `gate_rules_sha` (so a checkout's line-ending convention
|
|
92
|
+
never produces a false "the rules changed" alarm).
|
|
93
|
+
|
|
94
|
+
`--report` prints the last complete batch's scorecard (per-fixture MATCH/MISMATCH/UNRELIABLE,
|
|
95
|
+
per-stage hollow-flag-rate/sound-pass-rate) and an advisory drift line against the prior
|
|
96
|
+
complete batch: changes in `cli_version`/`model`/`gate_rules_sha`, plus the per-stage flag-rate
|
|
97
|
+
delta. Drift is advisory text only, never an exit-code signal. If the two compared batches
|
|
98
|
+
evaluated different fixture sets for a stage (e.g. a quick `--fixture`-filtered check against an
|
|
99
|
+
earlier full baseline), the delta is flagged as "not directly comparable" rather than presented
|
|
100
|
+
as a clean number — a shrinking/growing denominator is not the same signal as a judge's answer
|
|
101
|
+
changing. With `model:"unknown"` on either side, drift is explicitly narrowed to the
|
|
102
|
+
`cli_version`/prose axes.
|
|
103
|
+
|
|
104
|
+
## Why `eval` is not `read_only` in telemetry
|
|
105
|
+
|
|
106
|
+
`eval` is the only verb that makes billable network calls AND writes a results file
|
|
107
|
+
(`eval-results.jsonl`); every other read-only verb (`status`, `usage`, `doctor`, ...) only reads
|
|
108
|
+
existing state. Marking it `read_only:true` in the standard usage-log would be a telemetry lie,
|
|
109
|
+
so it is deliberately excluded from `_log_is_readonly`'s whitelist. `--report` alone genuinely
|
|
110
|
+
is read-only (zero calls, offline) but is not special-cased in telemetry, since the standard
|
|
111
|
+
event still correctly reflects "a build-affecting command ran" at the `eval`-verb level.
|
|
112
|
+
|
|
113
|
+
## v2 engine seam (documented, not built)
|
|
114
|
+
|
|
115
|
+
`_eval_engine_run()` is the single call site that invokes the judge CLI. A second/third engine
|
|
116
|
+
(Codex, Antigravity) would implement the same nonce-in/verdict-out contract behind that one
|
|
117
|
+
function, without touching the fixture loop, majority-vote, or scorecard logic. Not built in v1
|
|
118
|
+
— Claude-only was the approved scope, and no other engine's headless contract has been spiked
|
|
119
|
+
yet (the same "measure, don't assume" discipline that gated this build in the first place).
|
|
120
|
+
|
|
121
|
+
## Limitations (recorded, not hidden)
|
|
122
|
+
|
|
123
|
+
- **Fixture authorship bias**: the same people who wrote `gate-rules.md`'s challenges also wrote
|
|
124
|
+
the fixtures meant to test it. Mitigated by sourcing every hollow fixture's failure mode
|
|
125
|
+
verbatim from the rule text it's tested against (never an invented taxonomy); residual risk is
|
|
126
|
+
accepted for v1 and disclosed here.
|
|
127
|
+
- **Interrupt cleanup is best-effort, not guaranteed**, on at least this build's development
|
|
128
|
+
platform: a signal arriving while the runner is blocked on a foreground judge call was not
|
|
129
|
+
observed to preempt that call promptly. Cleanup between fixtures/calls fires reliably; a
|
|
130
|
+
normal (uninterrupted) run always cleans up its temp files immediately.
|
|
131
|
+
- **Small fixture corpus (six)**: enough to prove the mechanism and give a real number, not
|
|
132
|
+
enough for statistical confidence across every stage/failure-mode combination. Widening the
|
|
133
|
+
corpus is a natural v1.x follow-up, not a v1 blocker.
|
|
134
|
+
|
|
135
|
+
## Failure modes and postmortem (v0.21)
|
|
136
|
+
|
|
137
|
+
### INVALID storms — the 260710 case
|
|
138
|
+
|
|
139
|
+
The first real batch after v0.19 shipped came back **17/18 INVALID** (run `…-1783695631-…`). The
|
|
140
|
+
retry doubled cost without recovery — attempt 2 landed in the same window — and the CLI's
|
|
141
|
+
`session-lifecycle-hook cancelled` stderr line, the only diagnostic the storm actually produced,
|
|
142
|
+
was thrown away because the pre-v0.21 seam captured stdout only. A single-call retry cleared the
|
|
143
|
+
window; the mechanism was never confirmed at the time (rate-limit vs hook contention).
|
|
144
|
+
|
|
145
|
+
**v0.21 changes that make the next storm postmortemable and cost-safe:**
|
|
146
|
+
|
|
147
|
+
1. **Raw capture on final-INVALID.** Both attempts' stdout, stderr, and rc are persisted to
|
|
148
|
+
`.flow/eval-raw/<run_id>/<fixture>-v<vote>-a<attempt>.{out,err,rc}` (git-ignored via
|
|
149
|
+
`_ignore_run_state`; envelope stripped down to `result`/`assistant`/`rate_limit_event`
|
|
150
|
+
records — cwd, session_id, plugin/memory paths are removed).
|
|
151
|
+
2. **Injectable backoff.** `FLOW_EVAL_RETRY_BACKOFF` env (default 5s) delays the retry so it
|
|
152
|
+
doesn't hammer the same window. Tests set `0` to keep CI fast.
|
|
153
|
+
3. **Retry skip on rate-limit.** If attempt 1 carried a non-`allowed` `rate_limit_info.status`,
|
|
154
|
+
the retry is skipped (retry is documented for a *formatting slip*, not infra; retrying into a
|
|
155
|
+
live rate-limit window just doubles spend).
|
|
156
|
+
4. **Circuit breaker.** The FIRST evaluated fixture coming back UNRELIABLE (`invalid_count*3 > n`)
|
|
157
|
+
aborts the batch with a distinct nonzero exit code (2) and NO `done` trailer written — the
|
|
158
|
+
junk batch is invisible to `--report`/drift, so it cannot poison the baseline. `--keep-going`
|
|
159
|
+
forces the full batch; document worst case (≤ N_fixtures × n × 2 + 1 probe ≈ 37 calls at
|
|
160
|
+
default 6 × 3).
|
|
161
|
+
|
|
162
|
+
### Rate-limit visibility (advisory, best-effort)
|
|
163
|
+
|
|
164
|
+
The rows persisted after v0.21 gain `retries` (0/1 per fixture) and `rate_limited`
|
|
165
|
+
(`true`/`false` per fixture). `rate_limited` is anchored to the `rate_limit_info` record's own
|
|
166
|
+
`status` value — this is important because on cli 2.1.201 a **healthy** `allowed` event carries
|
|
167
|
+
`"overageStatus":"rejected"` as a separate field in the same envelope, and a naive
|
|
168
|
+
`grep '"status":"' | grep -v allowed` false-positives on it. A genuinely-throttled shape has not
|
|
169
|
+
yet been captured; treat `rate_limited` as best-effort advisory (absence ≠ not-throttled) until a
|
|
170
|
+
real throttled sample is added to the corpus.
|
|
171
|
+
|
|
172
|
+
### Concurrency caveat
|
|
173
|
+
|
|
174
|
+
`cmd_eval` takes **no** flow lock (it is a read-mostly verb, and the raw/prune writes are the
|
|
175
|
+
only cross-run writes it does). The pre-batch raw-prune keys off the epoch embedded in `run_id`
|
|
176
|
+
(deterministic, mount-independent, unforgeable by `touch`) and honors a `FLOW_LOCK_TTL`-second
|
|
177
|
+
guard so an in-flight or in-postmortem run's dir cannot be pruned under the operator's feet.
|
|
178
|
+
|
|
179
|
+
### INVALID-storm playbook
|
|
180
|
+
|
|
181
|
+
1. If `flow.sh eval` exits `2` and prints `ABORT after first fixture UNRELIABLE`, do NOT rerun
|
|
182
|
+
without `--keep-going` — read the named `raw capture` dir first.
|
|
183
|
+
2. Attempt-1 `.err` is usually the diagnostic (hook-cancelled / rate-limit line lands there).
|
|
184
|
+
3. If the storm was rate-limit: wait past the window (typically minutes), then rerun.
|
|
185
|
+
4. If persistent: capture the `.err` line in a comment, escalate — this is when a throttled
|
|
186
|
+
`rate_limit_info` sample would land in the corpus and let `rate_limited` become authoritative.
|