orchestrix-skills 0.2.0 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +7 -3
- package/package.json +1 -1
- package/skills/deploy/SKILL.md +83 -0
- package/skills/investigate/SKILL.md +80 -0
- package/skills/map-codebase/SKILL.md +76 -0
- package/skills/orchestrate/SKILL.md +62 -10
- package/skills/smoke-test/SKILL.md +88 -0
package/README.md
CHANGED
|
@@ -15,11 +15,15 @@ hosting, and team features are the premium layer — see below.
|
|
|
15
15
|
intent
|
|
16
16
|
└─ orchestrate (root: warm context, wires skills by output→input, enforces gates)
|
|
17
17
|
├─ brainstorm ──(needs facts?)─→ research
|
|
18
|
+
├─ (existing repo?) ──→ map-codebase (brownfield entry: evidence-based map → registry)
|
|
18
19
|
├─ (has UI?) ──→ design-system (once) → design-ui
|
|
19
20
|
├─ (arch decision?) ──→ design-architecture
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
21
|
+
├─ draft-story → implement → run-tests → review-code → commit
|
|
22
|
+
│ ↑ verify ↑ design-review (UI only)
|
|
23
|
+
│ (objective) ↑ accept (batched)
|
|
24
|
+
├─ (verify failing, cause unknown?) ──→ investigate (root cause → rework)
|
|
25
|
+
├─ (runnable app?) ──→ smoke-test (drive real flows, evidence captured)
|
|
26
|
+
└─ (accepted + ship it?) ──→ deploy (inline gate, rollback-first)
|
|
23
27
|
```
|
|
24
28
|
|
|
25
29
|
Human gates are front-loaded (planning = direction) and at the end (acceptance);
|
package/package.json
CHANGED
|
@@ -0,0 +1,83 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: deploy
|
|
3
|
+
description: Use when an ACCEPTED deliverable must be shipped to a live environment. Never mid-run, never on unaccepted work.
|
|
4
|
+
license: MIT
|
|
5
|
+
allowed-tools: [Read, Bash]
|
|
6
|
+
metadata:
|
|
7
|
+
contract:
|
|
8
|
+
inputs: [accepted_deliverable, target]
|
|
9
|
+
reads: [registry/deploy]
|
|
10
|
+
outputs: [deploy_report]
|
|
11
|
+
authority: "Run the project's DOCUMENTED deploy command for the named target, and its rollback. No infra provisioning, no DNS/billing/secret changes, no undocumented deploy paths."
|
|
12
|
+
verify: "A real request against the LIVE target succeeds post-deploy (the deploy tool's own 'success' does not count), and one core flow responds correctly."
|
|
13
|
+
accept:
|
|
14
|
+
when: "always — deploying is irreversible and user-facing."
|
|
15
|
+
timing: inline
|
|
16
|
+
---
|
|
17
|
+
|
|
18
|
+
# Deploy (Ship Accepted Work, With a Way Back)
|
|
19
|
+
|
|
20
|
+
The most dangerous skill in the graph. Everything here is designed around two
|
|
21
|
+
facts: deploys are user-facing, and the only safe deploy is one you can undo.
|
|
22
|
+
|
|
23
|
+
**Core principle:** No rollback plan, no deploy. If you cannot state the exact
|
|
24
|
+
command that undoes this deploy, you are not ready to run the one that does it.
|
|
25
|
+
|
|
26
|
+
## Preconditions — ALL must hold, verify each now
|
|
27
|
+
|
|
28
|
+
1. **The work is accepted.** Final acceptance happened; you are not deploying
|
|
29
|
+
to "see if it works" (that is `smoke-test`'s job, locally).
|
|
30
|
+
2. **The tree is clean and tested.** `git status` clean; `run-tests` green on
|
|
31
|
+
the exact commit being shipped (fresh run, not remembered).
|
|
32
|
+
3. **The deploy method is documented** in `registry/deploy` (or the project's
|
|
33
|
+
own deploy config/scripts). Not documented → STOP and ask the human how this
|
|
34
|
+
project ships. NEVER guess a deploy path — a guessed deploy against the
|
|
35
|
+
wrong target is the worst failure this graph can produce.
|
|
36
|
+
4. **The target is unambiguous.** "Deploy" without a named target defaults to
|
|
37
|
+
asking, not to production.
|
|
38
|
+
|
|
39
|
+
## Process
|
|
40
|
+
|
|
41
|
+
1. **Write the rollback plan first.** Record: the currently-live version
|
|
42
|
+
identifier (commit/tag/deployment id) and the exact rollback command.
|
|
43
|
+
Put both in the report BEFORE deploying.
|
|
44
|
+
2. **Inline human gate (this skill's accept).** Present: target, version to
|
|
45
|
+
ship, one-line change summary, rollback command. Deploy only on explicit
|
|
46
|
+
sign-off. This gate cannot be batched or deferred.
|
|
47
|
+
3. **Deploy** with the documented command. Capture the full output to
|
|
48
|
+
`.orchestrate/verify/deploy-<target>.log`.
|
|
49
|
+
4. **Post-verify against the LIVE target.** A real HTTP request (or the
|
|
50
|
+
platform equivalent) to the deployed URL: correct status AND expected
|
|
51
|
+
content, plus one core flow. The deploy tool saying "success" is step 3,
|
|
52
|
+
not step 4.
|
|
53
|
+
5. **On post-verify failure: roll back immediately** with the recorded
|
|
54
|
+
command, re-verify the old version is live again, and report honestly.
|
|
55
|
+
A failed deploy cleanly rolled back is a good outcome; a broken prod
|
|
56
|
+
left up while you debug is not.
|
|
57
|
+
|
|
58
|
+
## Output: `deploy_report`
|
|
59
|
+
|
|
60
|
+
```markdown
|
|
61
|
+
# Deploy — <target>
|
|
62
|
+
|
|
63
|
+
Shipped: <version/commit> (previous live: <version>)
|
|
64
|
+
Command: <documented command used>
|
|
65
|
+
Post-verify: <request + result> — passed/failed
|
|
66
|
+
Rollback: <command> — standing by | EXECUTED at <time>, old version re-verified live
|
|
67
|
+
Log: .orchestrate/verify/deploy-<target>.log
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
## Red flags — stop
|
|
71
|
+
|
|
72
|
+
- Deploying work that has not passed final acceptance
|
|
73
|
+
- Guessing the deploy method because `registry/deploy` is empty
|
|
74
|
+
- No recorded rollback command before deploying
|
|
75
|
+
- Treating the deploy tool's success message as proof the site works
|
|
76
|
+
- An ambiguous target resolved to production by default
|
|
77
|
+
- Debugging a broken deploy IN production instead of rolling back first
|
|
78
|
+
|
|
79
|
+
## Done
|
|
80
|
+
|
|
81
|
+
Live target verified serving the new version (or cleanly rolled back), report
|
|
82
|
+
written. Durable facts learned (the deploy command, the health URL) go back to
|
|
83
|
+
`registry/deploy` via the metabolism rules.
|
|
@@ -0,0 +1,80 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: investigate
|
|
3
|
+
description: Use when something is broken and the CAUSE is unknown — a repeatedly failing verify, a bug report, a regression — before any fix is attempted.
|
|
4
|
+
license: MIT
|
|
5
|
+
allowed-tools: [Read, Bash, Grep, Glob]
|
|
6
|
+
metadata:
|
|
7
|
+
contract:
|
|
8
|
+
inputs: [symptom, context?, prior_attempts?]
|
|
9
|
+
reads: [registry/architecture, taste/coding-standards]
|
|
10
|
+
outputs: [root_cause_report]
|
|
11
|
+
authority: "Read code and run diagnostics/reproductions. Temporary instrumentation is allowed but MUST be reverted before finishing. No fixes — the fix belongs to a re-dispatched implement."
|
|
12
|
+
verify: "The report contains a reproduction command that fails, and a mechanism that explains every observed symptom (not just the loudest one)."
|
|
13
|
+
accept:
|
|
14
|
+
when: "never — informational; it feeds the rework loop."
|
|
15
|
+
timing: deferred
|
|
16
|
+
---
|
|
17
|
+
|
|
18
|
+
# Investigate (Root Cause Before Any Fix)
|
|
19
|
+
|
|
20
|
+
Debugging is not "try changes until it passes". It is locating the mechanism.
|
|
21
|
+
|
|
22
|
+
**Core principle:** A fix you cannot explain mechanistically is not a fix — it
|
|
23
|
+
is a coincidence that will regress.
|
|
24
|
+
|
|
25
|
+
## The Iron Law
|
|
26
|
+
|
|
27
|
+
```
|
|
28
|
+
NO FIX WITHOUT A ROOT CAUSE STATED AS:
|
|
29
|
+
"X happens BECAUSE Y — evidenced by Z (something I observed this session)"
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
## Process
|
|
33
|
+
|
|
34
|
+
1. **Reproduce.** Find the exact command/steps that show the symptom, run them,
|
|
35
|
+
capture the output verbatim. Read the error LITERALLY — the message usually
|
|
36
|
+
says what is wrong, not what you assume is wrong. Cannot reproduce → that IS
|
|
37
|
+
the finding (report the conditions tried; do not "fix" what you cannot see).
|
|
38
|
+
2. **Localize.** Shrink the search space with evidence, not intuition:
|
|
39
|
+
- `git log`/`git bisect` when it used to work — what changed?
|
|
40
|
+
- Binary-search the pipeline: add temporary instrumentation (prints/asserts)
|
|
41
|
+
at midpoints to find where good state becomes bad state.
|
|
42
|
+
- Minimize the reproduction — smallest input that still fails.
|
|
43
|
+
3. **Hypothesize and FALSIFY.** For each hypothesis, design the observation
|
|
44
|
+
that would DISPROVE it, then run it. A hypothesis you only sought
|
|
45
|
+
confirmation for is not tested. Two failed rounds on the same theory →
|
|
46
|
+
the bug is in an assumption one level deeper (the config, the framework
|
|
47
|
+
contract, the data, the environment) — widen, don't re-try harder.
|
|
48
|
+
4. **State the root cause** in the iron-law form, plus:
|
|
49
|
+
- the minimal fix scope (which file/function, what change class),
|
|
50
|
+
- the regression test that would have caught it,
|
|
51
|
+
- any OTHER symptoms this mechanism predicts (check them — a mechanism
|
|
52
|
+
that doesn't explain all symptoms is incomplete).
|
|
53
|
+
5. **Revert your instrumentation.** `git diff` must be clean when you finish.
|
|
54
|
+
|
|
55
|
+
## Output: `root_cause_report`
|
|
56
|
+
|
|
57
|
+
```markdown
|
|
58
|
+
# Root cause — <symptom, one line>
|
|
59
|
+
|
|
60
|
+
## Reproduction — exact command; fails with <output tail>.
|
|
61
|
+
## Mechanism — X happens because Y. Evidence: Z (file:line / captured output).
|
|
62
|
+
## Fix scope — <file(s)>, <change class>. Explicitly NOT needed: <what a shotgun fix would have touched>.
|
|
63
|
+
## Regression test — <the test to add so this cannot silently return>.
|
|
64
|
+
## Ruled out — <hypotheses falsified and how> (saves the next person re-walking them).
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
## Red flags — stop and re-localize
|
|
68
|
+
|
|
69
|
+
- "Probably" / "might be" in the mechanism line
|
|
70
|
+
- A symptom fix: retry loops, sleeps, broad try/catch, widened types — with no
|
|
71
|
+
mechanism behind it
|
|
72
|
+
- Changing several things at once and observing "it passes now"
|
|
73
|
+
- A mechanism that explains one symptom but not the others
|
|
74
|
+
- Finishing with instrumentation still in the diff
|
|
75
|
+
|
|
76
|
+
## Done
|
|
77
|
+
|
|
78
|
+
Report written; reproduction demonstrably fails; diff clean. The orchestrator
|
|
79
|
+
re-dispatches `implement` with this report as `qa_feedback` — same capability,
|
|
80
|
+
now with a target instead of a guess.
|
|
@@ -0,0 +1,76 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: map-codebase
|
|
3
|
+
description: Use when entering an EXISTING codebase (brownfield) before designing or changing anything — build an evidence-based map of its architecture, conventions, and hazards.
|
|
4
|
+
license: MIT
|
|
5
|
+
allowed-tools: [Read, Bash, Grep, Glob]
|
|
6
|
+
metadata:
|
|
7
|
+
contract:
|
|
8
|
+
inputs: [repo_path, focus?]
|
|
9
|
+
reads: []
|
|
10
|
+
outputs: [codebase_map, registry_updates]
|
|
11
|
+
authority: "Read-only on source; non-mutating commands only (ls, grep, git log, test discovery). Writes go ONLY to knowledge/registry/*."
|
|
12
|
+
verify: "Every architectural claim in the map cites a file path you actually read; the traced flows exist in code, not in the README."
|
|
13
|
+
accept:
|
|
14
|
+
when: "never — informational; it feeds design and implement."
|
|
15
|
+
timing: deferred
|
|
16
|
+
---
|
|
17
|
+
|
|
18
|
+
# Map Codebase (Brownfield Entry)
|
|
19
|
+
|
|
20
|
+
Greenfield runs start from intent; brownfield runs start from SOMEONE ELSE'S
|
|
21
|
+
decisions. Changing code you haven't mapped produces changes that fight the
|
|
22
|
+
codebase — wrong layer, wrong convention, duplicated machinery.
|
|
23
|
+
|
|
24
|
+
**Core principle:** The map comes from reading code, not from directory names
|
|
25
|
+
or the README. A README describes what the project wishes it were; `git log`
|
|
26
|
+
and the source describe what it is.
|
|
27
|
+
|
|
28
|
+
## Process
|
|
29
|
+
|
|
30
|
+
1. **Inventory.** Layout, manifests (`package.json`/`pyproject`/`go.mod`…),
|
|
31
|
+
scripts, CI config, generated/vendored dirs (mark them DO-NOT-EDIT).
|
|
32
|
+
2. **Trace one or two REAL flows end to end** (guided by `focus` if given):
|
|
33
|
+
entry point → routing/dispatch → business logic → persistence/IO. Read the
|
|
34
|
+
actual files; record the chain as `file:symbol → file:symbol`. This step is
|
|
35
|
+
what separates a map from a guess.
|
|
36
|
+
3. **Extract conventions from evidence** — for each, cite the example file you
|
|
37
|
+
derived it from: naming, module boundaries, error handling style, test
|
|
38
|
+
location and framework, how config/env is read, commit message style.
|
|
39
|
+
4. **Hazards.** Migrations and how they run; generated code and what generates
|
|
40
|
+
it; global state; areas with no test coverage; anything `git log` shows as
|
|
41
|
+
churn-heavy (bug-prone) or untouched-for-years (fragile assumptions).
|
|
42
|
+
5. **Write back to `knowledge/registry/`** following the metabolism governance
|
|
43
|
+
in `orchestrate` (read-before-write, update-don't-append, facts only):
|
|
44
|
+
- `registry/architecture.md` — the traced structure
|
|
45
|
+
- `registry/conventions.md` — the evidenced conventions
|
|
46
|
+
Registry holds FACTS about this codebase. Opinions and preferences belong
|
|
47
|
+
in `taste/*`, and only via a human correction — not from this skill.
|
|
48
|
+
|
|
49
|
+
## Output: `codebase_map`
|
|
50
|
+
|
|
51
|
+
```markdown
|
|
52
|
+
# Codebase map — <repo> @ <commit>
|
|
53
|
+
|
|
54
|
+
## Shape — <stack, top-level layout, one paragraph>
|
|
55
|
+
## Traced flows
|
|
56
|
+
- <flow>: entry `a.ts:handler` → `b.ts:service` → `c.ts:repo` → <storage>
|
|
57
|
+
## Conventions (evidence-cited)
|
|
58
|
+
- <convention> — see <file>
|
|
59
|
+
## Hazards
|
|
60
|
+
- <hazard> — <why it bites> — <file/dir>
|
|
61
|
+
## Where a change like "<focus>" belongs — <layer/files>, following <convention>
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
## Red flags — stop
|
|
65
|
+
|
|
66
|
+
- Architecture described from folder names without opening the files
|
|
67
|
+
- Repeating a README/document claim without spot-checking it in code
|
|
68
|
+
- A "convention" cited from zero examples
|
|
69
|
+
- Writing preferences/judgments into `registry/` (facts only)
|
|
70
|
+
- Editing anything outside `knowledge/`
|
|
71
|
+
|
|
72
|
+
## Done
|
|
73
|
+
|
|
74
|
+
Map written with citations; registry updated. Downstream `design-architecture`
|
|
75
|
+
and `implement` read the registry and MUST follow the documented conventions —
|
|
76
|
+
that is the point of having mapped them.
|
|
@@ -4,7 +4,7 @@ description: Use when a goal must be delivered end-to-end by composing skills, w
|
|
|
4
4
|
license: MIT
|
|
5
5
|
allowed-tools: [Read, Write, Edit, Bash, Grep, Glob, Task]
|
|
6
6
|
metadata:
|
|
7
|
-
version:
|
|
7
|
+
version: 4
|
|
8
8
|
contract:
|
|
9
9
|
inputs: [intent, constraints?]
|
|
10
10
|
reads: [skill-registry, taste/*]
|
|
@@ -46,9 +46,20 @@ no step above intent.
|
|
|
46
46
|
the same turn, for speed. Keep dependent steps sequential. NEVER fire-and-forget
|
|
47
47
|
a background subagent and end the turn waiting to be woken — run foreground and
|
|
48
48
|
await; there is no reliable async wake.
|
|
49
|
-
5. **Verify (gate).**
|
|
50
|
-
|
|
51
|
-
|
|
49
|
+
5. **Verify (gate) — executable, not prose.** Prove the skill's `verify` with a
|
|
50
|
+
REAL command you run yourself via Bash, and capture the proof:
|
|
51
|
+
|
|
52
|
+
```
|
|
53
|
+
.orchestrate/verify/step-<n>-attempt-<k>.log ← exact command + exit code + output tail
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
The command comes from the step's nature (test runner, build, `git fsck`,
|
|
57
|
+
`test -f`, a curl against the running app …) — pick the one that would FAIL
|
|
58
|
+
if the claim were false. The subagent's own success report is NOT
|
|
59
|
+
verification. A prose-only verify is acceptable only when no command can
|
|
60
|
+
prove the claim (e.g. judging tone) — say so in the log file. If verify
|
|
61
|
+
fails, re-dispatch the **same** skill with the failure as feedback (see
|
|
62
|
+
Rework). If it passes, continue.
|
|
52
63
|
6. **Accept (gate).** Apply the rule below. Then continue — do not pause to ask
|
|
53
64
|
"should I keep going?" mid-run.
|
|
54
65
|
7. **Repeat** 3–6 until the intent is fulfilled.
|
|
@@ -66,17 +77,53 @@ no step above intent.
|
|
|
66
77
|
|
|
67
78
|
You are the teeth. The fields are only data; you enforce them.
|
|
68
79
|
|
|
69
|
-
## Rework is a loop, not a skill
|
|
80
|
+
## Rework is a loop, not a skill — and the loop is BOUNDED
|
|
70
81
|
|
|
71
82
|
A failed `verify` or a `changes_requested` review is not a separate "fix" step.
|
|
72
83
|
Re-dispatch the same skill with the feedback as an input (e.g. `qa_feedback`).
|
|
73
84
|
Same capability, new input.
|
|
74
85
|
|
|
75
|
-
|
|
86
|
+
**Rework without understanding is a coin flip.** If a verify failure's CAUSE
|
|
87
|
+
is not understood after the first failed attempt, dispatch `investigate` before
|
|
88
|
+
spending the next attempt — its `root_cause_report` becomes the re-dispatch's
|
|
89
|
+
`qa_feedback`. An attempt aimed at a stated mechanism converges; an attempt
|
|
90
|
+
aimed at a symptom re-rolls the dice.
|
|
91
|
+
|
|
92
|
+
**Hard cap: 3 attempts per step.** If a step's verify still fails on attempt 3,
|
|
93
|
+
STOP the run — do not burn a 4th attempt. Write a `gate` event to the ledger
|
|
94
|
+
(`{"e":"gate","kind":"rework_exhausted","question":"step <n> (<skill>) failed 3
|
|
95
|
+
attempts: <one-line why>"}`), summarize the three failures for the human, and
|
|
96
|
+
report AWAIT. A step that cannot pass its own verify after three tries needs a
|
|
97
|
+
human decision (wrong approach, wrong spec, or wrong verify), not more tokens.
|
|
98
|
+
|
|
99
|
+
## Metabolism — governed writeback
|
|
76
100
|
|
|
77
101
|
When the human corrects something at final acceptance ("not on-brand", "wrong
|
|
78
|
-
tone"),
|
|
79
|
-
the next run
|
|
102
|
+
tone"), fold it back into `taste/*` (preferences) or `registry/*` (facts about
|
|
103
|
+
this codebase), so the next run starts smarter. The run teaches the
|
|
104
|
+
organization — but memory rots without curation, so writeback is GOVERNED:
|
|
105
|
+
|
|
106
|
+
1. **Read before write.** Open the target file first. An existing entry on the
|
|
107
|
+
same topic gets UPDATED in place — never append a near-duplicate.
|
|
108
|
+
2. **Contradiction = replacement.** A correction that contradicts an existing
|
|
109
|
+
entry REPLACES it (add a short `supersedes: <old rule> (<date>)` note).
|
|
110
|
+
Never leave both standing — two contradictory rules poison every later run
|
|
111
|
+
that reads them.
|
|
112
|
+
3. **Durable preferences only.** Taste holds style/architecture/process
|
|
113
|
+
preferences that apply to FUTURE runs. One-off task facts, transient state,
|
|
114
|
+
and anything the repo or ledger already records do not belong there.
|
|
115
|
+
4. **One lesson per entry** — imperative phrasing, a one-line why, and a date.
|
|
116
|
+
5. **Size bound: ~100 lines per file.** If a write would push past it,
|
|
117
|
+
consolidate in the same edit (merge near-duplicates, drop obsolete entries)
|
|
118
|
+
— never blind-append to a bloated file.
|
|
119
|
+
6. **The human sees the diff.** taste/registry changes made during a run are
|
|
120
|
+
part of final acceptance: present what changed and why, so a bad lesson can
|
|
121
|
+
be vetoed before it contaminates future runs.
|
|
122
|
+
|
|
123
|
+
`taste/*` vs `registry/*`: taste is HOW we prefer things done (opinions,
|
|
124
|
+
overridable); registry is WHAT is true of this project (facts, verifiable).
|
|
125
|
+
A correction usually lands in taste; a discovered fact (the deploy command,
|
|
126
|
+
the test runner) lands in registry.
|
|
80
127
|
|
|
81
128
|
## The ledger (`.orchestrate/ledger.jsonl`)
|
|
82
129
|
|
|
@@ -97,8 +144,9 @@ Events and when to write them:
|
|
|
97
144
|
| `run_end` | at delivery or abandonment | `{"e":"run_end","run":"...","result":"delivered\|paused\|abandoned","ts":"..."}` |
|
|
98
145
|
|
|
99
146
|
A step recorded `done` is done — do not re-dispatch it. `evidence` on a `done`
|
|
100
|
-
step is required
|
|
101
|
-
|
|
147
|
+
step is required and should be the step's verify log path
|
|
148
|
+
(`.orchestrate/verify/step-<n>-attempt-<k>.log`); a `done` with no evidence is
|
|
149
|
+
a false claim.
|
|
102
150
|
|
|
103
151
|
## Context discipline (stay lean)
|
|
104
152
|
|
|
@@ -117,4 +165,8 @@ evidence is a false claim.
|
|
|
117
165
|
- Re-dispatching a step the ledger already marks done
|
|
118
166
|
- Dispatching a step without first writing its `dispatched` ledger line
|
|
119
167
|
- Ending a run without a `run_end` ledger line
|
|
168
|
+
- Marking a step done on the subagent's say-so, without your own verify command
|
|
169
|
+
- A 4th rework attempt on the same step (cap is 3 — stop and gate)
|
|
170
|
+
- A second rework attempt with no `investigate` when the failure isn't understood
|
|
171
|
+
- Appending to `taste/*` without reading it first (duplicate/contradiction risk)
|
|
120
172
|
- Marking the run complete without every step's `verify` evidence
|
|
@@ -0,0 +1,88 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: smoke-test
|
|
3
|
+
description: Use when the deliverable is a runnable app or service and its real user flows must be proven working from the OUTSIDE (browser/HTTP/CLI) before acceptance — unit tests passing is not that proof.
|
|
4
|
+
license: MIT
|
|
5
|
+
allowed-tools: [Read, Bash, Grep, Glob]
|
|
6
|
+
metadata:
|
|
7
|
+
contract:
|
|
8
|
+
inputs: [run_instructions, flows, qa_feedback?]
|
|
9
|
+
reads: [registry/app]
|
|
10
|
+
outputs: [smoke_report, verify_evidence]
|
|
11
|
+
authority: "Start and stop the app locally; drive it via browser automation, HTTP, or CLI. Read-only on source. No deploy, no external spend, no mutations outside the app's own local state."
|
|
12
|
+
verify: "self — this skill PRODUCES the acceptance evidence; every flow verdict points at a captured artifact."
|
|
13
|
+
accept:
|
|
14
|
+
when: "never — the evidence feeds final acceptance; it does not replace it."
|
|
15
|
+
timing: deferred
|
|
16
|
+
---
|
|
17
|
+
|
|
18
|
+
# Smoke Test (Prove It Works From the Outside)
|
|
19
|
+
|
|
20
|
+
Unit tests prove functions; a smoke test proves the PRODUCT. Launch the real
|
|
21
|
+
app and walk the real flows the user will walk.
|
|
22
|
+
|
|
23
|
+
**Core principle:** A flow is "passed" only when you drove it yourself and
|
|
24
|
+
captured the evidence. "The tests are green so it probably works" is a claim,
|
|
25
|
+
not a smoke test.
|
|
26
|
+
|
|
27
|
+
## The Iron Law
|
|
28
|
+
|
|
29
|
+
```
|
|
30
|
+
UNTESTED IS NOT PASSED. Every flow ends in exactly one of:
|
|
31
|
+
passed (with evidence) | failed (with evidence) | untested (with the reason)
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
## Process
|
|
35
|
+
|
|
36
|
+
1. **Discover how to run it.** `registry/app` first; else the project's
|
|
37
|
+
manifest (`package.json` scripts, `Makefile`, `README`). If the launch
|
|
38
|
+
method is genuinely undocumented and unguessable, report `untested:
|
|
39
|
+
cannot launch` — do not invent a server.
|
|
40
|
+
2. **Launch in the background, capture logs.** Redirect stdout/stderr to
|
|
41
|
+
`.orchestrate/verify/smoke-server.log`. Record the PID. Pick a free port if
|
|
42
|
+
configurable (avoid colliding with anything already running).
|
|
43
|
+
3. **Wait for readiness, bounded.** Poll the health endpoint / port / ready
|
|
44
|
+
line for up to ~60s. Not ready → flow verdicts are `failed: app did not
|
|
45
|
+
start`, attach the server log, skip to cleanup.
|
|
46
|
+
4. **Exercise EVERY listed flow** with the best driver available, in order of
|
|
47
|
+
fidelity:
|
|
48
|
+
- **Browser automation** (a Playwright/Chrome MCP tool, if available in
|
|
49
|
+
this session) — for UI flows: navigate, interact, assert on rendered
|
|
50
|
+
state, screenshot.
|
|
51
|
+
- **HTTP** (`curl`) — assert status code AND response content (a 200
|
|
52
|
+
serving an error page is a fail; check for a string the flow implies).
|
|
53
|
+
- **CLI** — invoke the command, assert exit code and output.
|
|
54
|
+
Use the highest-fidelity driver the flow needs: an interactive UI flow
|
|
55
|
+
"verified" by curling `/` is `untested`, not `passed`.
|
|
56
|
+
5. **Capture evidence per flow** to `.orchestrate/verify/smoke-<flow-slug>.log`
|
|
57
|
+
(the exact command/steps + relevant output tail, or the screenshot path).
|
|
58
|
+
6. **ALWAYS clean up** — kill the processes you started (and only those),
|
|
59
|
+
remove temp state you created. Cleanup runs even when flows fail.
|
|
60
|
+
|
|
61
|
+
## Output: `smoke_report`
|
|
62
|
+
|
|
63
|
+
```markdown
|
|
64
|
+
# Smoke Report — <app> @ <commit>
|
|
65
|
+
|
|
66
|
+
| Flow | Verdict | Evidence |
|
|
67
|
+
|---|---|---|
|
|
68
|
+
| signup happy path | passed | .orchestrate/verify/smoke-signup.log |
|
|
69
|
+
| checkout | failed — 500 on POST /pay | .orchestrate/verify/smoke-checkout.log |
|
|
70
|
+
| admin export | untested — needs OAuth I can't complete | (reason) |
|
|
71
|
+
|
|
72
|
+
Server log: .orchestrate/verify/smoke-server.log
|
|
73
|
+
Started/stopped: <pid(s)>, cleaned up: yes
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
## Red flags — stop
|
|
77
|
+
|
|
78
|
+
- Reporting `passed` for a flow you did not drive
|
|
79
|
+
- A UI flow "verified" with a single `curl /` status check
|
|
80
|
+
- Conflating "the app is broken" with "my harness is broken" — say which
|
|
81
|
+
- Leaving the server (or any process you spawned) running after the report
|
|
82
|
+
- Only testing the happy path when `flows` lists error/edge flows
|
|
83
|
+
|
|
84
|
+
## Done
|
|
85
|
+
|
|
86
|
+
Every flow has a verdict + evidence artifact; processes cleaned up. `failed`
|
|
87
|
+
and `untested` are honest, valid results — report them verbatim, never rounded
|
|
88
|
+
up. The report feeds the orchestrator's verify gate and final acceptance.
|