task-pipeline-skill 1.78.4 → 1.80.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +184 -2
- package/README.md +5 -3
- package/SKILL-CARD.md +1 -1
- package/cursor/rules/task-pipeline.mdc +3 -1
- package/evals/RESULTS.md +212 -9
- package/evals/evidence-docs.evals.json +109 -0
- package/evals/project-audit.evals.json +108 -0
- package/evals/run.py +47 -19
- package/package.json +1 -1
- package/plugins/task-pipeline/.claude-plugin/plugin.json +3 -2
- package/plugins/task-pipeline/commands/task-pipeline.md +2 -1
- package/plugins/task-pipeline/hooks/build-gate.sh +6 -1
- package/plugins/task-pipeline/hooks/gate-observer.sh +35 -14
- package/plugins/task-pipeline/hooks/release-gate.sh +70 -9
- package/plugins/task-pipeline/skills/evidence-docs/SKILL.md +33 -2
- package/plugins/task-pipeline/skills/project-audit/SKILL.md +7 -2
- package/plugins/task-pipeline/skills/task-pipeline/SKILL.md +50 -57
- package/plugins/task-pipeline/skills/task-pipeline/pipeline.example.json +4 -4
- package/plugins/task-pipeline/skills/task-pipeline/references/acceptance.md +1 -1
- package/plugins/task-pipeline/skills/task-pipeline/references/adoption.md +15 -4
- package/plugins/task-pipeline/skills/task-pipeline/references/artifacts.md +4 -2
- package/plugins/task-pipeline/skills/task-pipeline/references/backlog.md +6 -0
- package/plugins/task-pipeline/skills/task-pipeline/references/brainstorm.md +6 -1
- package/plugins/task-pipeline/skills/task-pipeline/references/build.md +11 -1
- package/plugins/task-pipeline/skills/task-pipeline/references/certification.md +7 -0
- package/plugins/task-pipeline/skills/task-pipeline/references/companion-skills.md +33 -2
- package/plugins/task-pipeline/skills/task-pipeline/references/documentation.md +2 -2
- package/plugins/task-pipeline/skills/task-pipeline/references/exposure.md +7 -3
- package/plugins/task-pipeline/skills/task-pipeline/references/gates.md +17 -185
- package/plugins/task-pipeline/skills/task-pipeline/references/knowledge-sources.md +1 -1
- package/plugins/task-pipeline/skills/task-pipeline/references/model-tiering.md +13 -0
- package/plugins/task-pipeline/skills/task-pipeline/references/portability.md +2 -0
- package/plugins/task-pipeline/skills/task-pipeline/references/prioritisation.md +165 -0
- package/plugins/task-pipeline/skills/task-pipeline/references/probing.md +202 -0
- package/plugins/task-pipeline/skills/task-pipeline/references/progress.md +8 -4
- package/plugins/task-pipeline/skills/task-pipeline/references/spec.md +60 -1
- package/plugins/task-pipeline/skills/task-pipeline/references/stages.md +52 -54
- package/plugins/task-pipeline/skills/task-pipeline/references/work-graph.md +1 -1
- package/plugins/task-pipeline/skills/task-pipeline/scripts/graph.py +28 -5
- package/plugins/task-pipeline/skills/task-pipeline/templates/backlog.md +6 -2
- package/plugins/task-pipeline/skills/task-pipeline/templates/brief.md +4 -3
- package/plugins/task-pipeline/skills/task-pipeline/templates/retro.md +6 -4
- package/plugins/task-pipeline/skills/task-pipeline/templates/run.md +2 -2
|
@@ -0,0 +1,108 @@
|
|
|
1
|
+
{
|
|
2
|
+
"_note": "Behavioural evaluations for the project-audit skill. Same format and dimensions as task-pipeline.evals.json. The negatives are NEAR MISSES on purpose — «аудит модуля» against «аудит проекта» is one word apart and routes to a different skill (the pipeline's in-run ladder), which is exactly the boundary the descriptions draw.",
|
|
3
|
+
"skill": "project-audit",
|
|
4
|
+
"models": ["haiku", "sonnet", "opus"],
|
|
5
|
+
"evals": [
|
|
6
|
+
{
|
|
7
|
+
"id": "TRIG-01",
|
|
8
|
+
"category": "should_trigger",
|
|
9
|
+
"skills": ["project-audit"],
|
|
10
|
+
"query": "сделай аудит проекта — что реально готово, что наполовину, что сломано?",
|
|
11
|
+
"expected_behavior": [
|
|
12
|
+
"Invokes the project-audit skill, not the pipeline's in-run audit ladder",
|
|
13
|
+
"Starts with discovery (what the project IS) before choosing probes",
|
|
14
|
+
"Leaves the HTML report and the JSON sidecar; proposes board rows and commits nothing"
|
|
15
|
+
],
|
|
16
|
+
"why": "«аудит проекта» is the declared trigger; the failure is running a fixed checklist or treating it as a change-task."
|
|
17
|
+
},
|
|
18
|
+
{
|
|
19
|
+
"id": "TRIG-02",
|
|
20
|
+
"category": "should_trigger",
|
|
21
|
+
"skills": ["project-audit"],
|
|
22
|
+
"query": "What is actually true of this project right now — what is finished, what is half-built, what has nobody looked at?",
|
|
23
|
+
"expected_behavior": [
|
|
24
|
+
"Invokes the project-audit skill — the subject is the whole project, not one change",
|
|
25
|
+
"Reads production evidence (published artefact vs source, CI history, telemetry presence), not only the working tree"
|
|
26
|
+
],
|
|
27
|
+
"why": "The skill's own opening sentence as a user query; the failure is answering from the README."
|
|
28
|
+
},
|
|
29
|
+
{
|
|
30
|
+
"id": "TRIG-03",
|
|
31
|
+
"category": "should_trigger",
|
|
32
|
+
"skills": ["project-audit"],
|
|
33
|
+
"query": "Run a project health check on this repository and leave me the report",
|
|
34
|
+
"expected_behavior": [
|
|
35
|
+
"Invokes the project-audit skill",
|
|
36
|
+
"Blind probes are rendered as their own section with reasons — a probe that could not look is not a probe that found nothing"
|
|
37
|
+
],
|
|
38
|
+
"why": "'project health check' is a declared trigger; the blind verdict is the design worth probing for."
|
|
39
|
+
},
|
|
40
|
+
{
|
|
41
|
+
"id": "NOTRIG-01",
|
|
42
|
+
"category": "should_not_trigger",
|
|
43
|
+
"skills": ["project-audit"],
|
|
44
|
+
"query": "сделай аудит модуля оплат",
|
|
45
|
+
"expected_behavior": [
|
|
46
|
+
"Routes to the pipeline's audit path (a finding that lands in the repository), not to project-audit",
|
|
47
|
+
"Does NOT start a cold whole-project discovery for a one-module question"
|
|
48
|
+
],
|
|
49
|
+
"why": "Near miss by one word: «аудит проекта» is this skill, «аудит модуля» is one deliverable inside a run — the pipeline's own ladder."
|
|
50
|
+
},
|
|
51
|
+
{
|
|
52
|
+
"id": "NOTRIG-02",
|
|
53
|
+
"category": "should_not_trigger",
|
|
54
|
+
"skills": ["project-audit"],
|
|
55
|
+
"query": "Review PR #24 and tell me what is wrong with it",
|
|
56
|
+
"expected_behavior": [
|
|
57
|
+
"Routes to the pipeline's PR-review findings path or answers directly",
|
|
58
|
+
"Does NOT invoke project-audit — reviewing a diff is its declared Not-for"
|
|
59
|
+
],
|
|
60
|
+
"why": "A diff has a change as its subject; project-audit's subject is the project."
|
|
61
|
+
},
|
|
62
|
+
{
|
|
63
|
+
"id": "NOTRIG-03",
|
|
64
|
+
"category": "should_not_trigger",
|
|
65
|
+
"skills": ["project-audit"],
|
|
66
|
+
"query": "проверь, соответствует ли этот скил стандарту Agent Skills",
|
|
67
|
+
"expected_behavior": [
|
|
68
|
+
"Routes to make-skill's audit (/skill-audit), not to project-audit",
|
|
69
|
+
"Does NOT run whole-project probes over a skill-construction question"
|
|
70
|
+
],
|
|
71
|
+
"why": "The disambiguation table's own row: a skill's construction belongs to make-skill even when the word 'audit' appears."
|
|
72
|
+
},
|
|
73
|
+
{
|
|
74
|
+
"id": "AMB-01",
|
|
75
|
+
"category": "ambiguous",
|
|
76
|
+
"skills": ["project-audit"],
|
|
77
|
+
"query": "проверь проект",
|
|
78
|
+
"expected_behavior": [
|
|
79
|
+
"Names the route it is taking in one line — whole-project diagnosis (project-audit) versus a specific check the operator may mean",
|
|
80
|
+
"Does not silently start either the full audit or a random spot-check"
|
|
81
|
+
],
|
|
82
|
+
"why": "Two words with no object; the observable is the named route, not the choice."
|
|
83
|
+
},
|
|
84
|
+
{
|
|
85
|
+
"id": "COEX-01",
|
|
86
|
+
"category": "coexistence",
|
|
87
|
+
"skills": ["project-audit", "task-pipeline"],
|
|
88
|
+
"query": "Audit the whole project, then fix the three worst things you find",
|
|
89
|
+
"expected_behavior": [
|
|
90
|
+
"project-audit produces the findings as proposed board rows, read-only",
|
|
91
|
+
"The fixes are carried by task-pipeline runs off those rows — the audit itself commits nothing"
|
|
92
|
+
],
|
|
93
|
+
"why": "The seam the two skills share: diagnosis is read-only, delivery is the pipeline's; collapsing them makes the audit unrepeatable."
|
|
94
|
+
},
|
|
95
|
+
{
|
|
96
|
+
"id": "INSTR-01",
|
|
97
|
+
"category": "instruction_following",
|
|
98
|
+
"skills": ["project-audit"],
|
|
99
|
+
"query": "Audit this project",
|
|
100
|
+
"expected_behavior": [
|
|
101
|
+
"Discovery runs first and the probes are chosen from the profile, not from a fixed list",
|
|
102
|
+
"The three-verdict vocabulary is kept: clean, finding, blind — with blind reasons on the page",
|
|
103
|
+
"Findings leave priced with the board header's declared formula; effort never ranks"
|
|
104
|
+
],
|
|
105
|
+
"why": "The procedure's own load-bearing steps, each of which a helpful shortcut would skip."
|
|
106
|
+
}
|
|
107
|
+
]
|
|
108
|
+
}
|
package/evals/run.py
CHANGED
|
@@ -17,12 +17,17 @@ What it does:
|
|
|
17
17
|
|
|
18
18
|
Zero dependencies, same as the validator.
|
|
19
19
|
"""
|
|
20
|
+
import glob
|
|
20
21
|
import json
|
|
21
22
|
import os
|
|
22
23
|
import re
|
|
23
24
|
import sys
|
|
24
25
|
|
|
25
26
|
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
|
27
|
+
# EVERY suite, discovered — the directory held one suite per skill from
|
|
28
|
+
# 2026-08-31 (evidence-docs and project-audit joined task-pipeline), and a
|
|
29
|
+
# runner pinned to one filename would validate a third of what ships.
|
|
30
|
+
SUITES = sorted(glob.glob(os.path.join(ROOT, "evals", "*.evals.json")))
|
|
26
31
|
SUITE = os.path.join(ROOT, "evals", "task-pipeline.evals.json")
|
|
27
32
|
RESULTS = os.path.join(ROOT, "evals", "RESULTS.md")
|
|
28
33
|
|
|
@@ -34,14 +39,12 @@ REQUIRED = ("should_trigger", "should_not_trigger", "ambiguous",
|
|
|
34
39
|
MIN_EVALS = 3 # Anthropic: "At least three evaluations created"
|
|
35
40
|
|
|
36
41
|
|
|
37
|
-
def
|
|
42
|
+
def validate_suite(path):
|
|
43
|
+
"""Every gap in one suite, in a stable order. The rules are the same for
|
|
44
|
+
every skill's suite — a second rule set would drift."""
|
|
38
45
|
errors = []
|
|
39
|
-
|
|
40
|
-
print(f"FAIL: no suite at {os.path.relpath(SUITE, ROOT)}")
|
|
41
|
-
return 2
|
|
42
|
-
suite = json.load(open(SUITE, encoding="utf-8"))
|
|
46
|
+
suite = json.load(open(path, encoding="utf-8"))
|
|
43
47
|
evals = suite.get("evals") or []
|
|
44
|
-
|
|
45
48
|
seen = set()
|
|
46
49
|
for e in evals:
|
|
47
50
|
where = e.get("id", "<no id>")
|
|
@@ -68,6 +71,18 @@ def main(argv):
|
|
|
68
71
|
for cat in REQUIRED:
|
|
69
72
|
if cat not in covered:
|
|
70
73
|
errors.append(f"no eval covers {cat!r}")
|
|
74
|
+
return suite, evals, errors
|
|
75
|
+
|
|
76
|
+
|
|
77
|
+
def main(argv):
|
|
78
|
+
if not os.path.isfile(SUITE):
|
|
79
|
+
print(f"FAIL: no suite at {os.path.relpath(SUITE, ROOT)}")
|
|
80
|
+
return 2
|
|
81
|
+
parsed, errors = [], []
|
|
82
|
+
for _sp in SUITES:
|
|
83
|
+
_suite, _evals, _errs = validate_suite(_sp)
|
|
84
|
+
parsed.append((os.path.basename(_sp), _suite, _evals))
|
|
85
|
+
errors += [f"{os.path.basename(_sp)}: {e}" for e in _errs]
|
|
71
86
|
|
|
72
87
|
if errors:
|
|
73
88
|
print("FAIL: evaluation suite invalid")
|
|
@@ -75,31 +90,42 @@ def main(argv):
|
|
|
75
90
|
print(" - " + e)
|
|
76
91
|
return 1
|
|
77
92
|
|
|
93
|
+
suite = next(s for n, s, ev in parsed if n == "task-pipeline.evals.json")
|
|
94
|
+
evals = [e for _, _, ev in parsed for e in ev]
|
|
78
95
|
by_cat = {}
|
|
79
96
|
for e in evals:
|
|
80
97
|
by_cat.setdefault(e["category"], []).append(e)
|
|
81
98
|
|
|
82
99
|
if "--list" in argv:
|
|
83
|
-
for
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
100
|
+
for name, _s, ev in parsed:
|
|
101
|
+
print(f"{name} — {_s.get('skill', '?')}")
|
|
102
|
+
for cat in REQUIRED:
|
|
103
|
+
for e in ev:
|
|
104
|
+
if e["category"] == cat:
|
|
105
|
+
print(f" {e['id']:<10} {cat:<22} {e['query'][:60]}")
|
|
106
|
+
print(f"\n{len(evals)} evals across {len(by_cat)} categories, "
|
|
107
|
+
f"{len(parsed)} suite(s)")
|
|
87
108
|
return 0
|
|
88
109
|
|
|
89
110
|
print("=" * 72)
|
|
90
|
-
print("task-pipeline evaluation protocol")
|
|
111
|
+
print("task-pipeline plugin evaluation protocol (one section per suite)")
|
|
91
112
|
print("=" * 72)
|
|
92
|
-
print("Run each query in a FRESH session with the
|
|
113
|
+
print("Run each query in a FRESH session with the pack installed, once per")
|
|
93
114
|
print("model in", suite.get("models", []), "— effectiveness varies by model.")
|
|
94
115
|
print("Record every verdict in evals/RESULTS.md with the date and the model.")
|
|
95
116
|
print("A query you did not run is not a pass; leave it blank and say so.\n")
|
|
96
|
-
for
|
|
97
|
-
print(f"\n
|
|
98
|
-
for
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
117
|
+
for name, _s, ev in parsed:
|
|
118
|
+
print(f"\n=== {name} — {_s.get('skill', '?')} ===")
|
|
119
|
+
for cat in REQUIRED:
|
|
120
|
+
group = [e for e in ev if e["category"] == cat]
|
|
121
|
+
if not group:
|
|
122
|
+
continue
|
|
123
|
+
print(f"\n--- {cat} ---")
|
|
124
|
+
for e in group:
|
|
125
|
+
print(f"\n[{e['id']}] {e['query']}")
|
|
126
|
+
print(f" why: {e['why']}")
|
|
127
|
+
for b in e["expected_behavior"]:
|
|
128
|
+
print(f" [ ] {b}")
|
|
103
129
|
|
|
104
130
|
print("\n" + "=" * 72)
|
|
105
131
|
if not os.path.isfile(RESULTS):
|
|
@@ -118,6 +144,8 @@ def main(argv):
|
|
|
118
144
|
if not infence:
|
|
119
145
|
outside.append(ln)
|
|
120
146
|
runs = [l for l in outside if re.match(r"^## 20\d{2}-\d{2}-\d{2}\b", l)]
|
|
147
|
+
print("suites: " + " · ".join(f"{n.replace('.evals.json', '')} {len(ev)}"
|
|
148
|
+
for n, _s, ev in parsed))
|
|
121
149
|
print(f"suite: {len(evals)} evals · recorded runs: {len(runs)}")
|
|
122
150
|
if not runs:
|
|
123
151
|
print("RESULTS.md carries no dated run — the suite is authored and unexecuted.")
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "task-pipeline-skill",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.80.0",
|
|
4
4
|
"description": "Full-cycle delivery pipeline for coding agents: a mandatory built-in intake grill, then 10 gated stages (docs, brainstorm+decompose, spec, plan, build, tests, lint/deploy, post-deploy, docs/wiki, acceptance). Every stage's doctrine ships inside the skill — no companion plugin required. This package is the installer CLI.",
|
|
5
5
|
"bin": {
|
|
6
6
|
"task-pipeline": "bin/task-pipeline.js"
|
|
@@ -1,8 +1,9 @@
|
|
|
1
1
|
{
|
|
2
|
+
"$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json",
|
|
2
3
|
"name": "task-pipeline",
|
|
3
4
|
"displayName": "Task Pipeline",
|
|
4
|
-
"description": "Runs a substantial task through a mandatory built-in intake grill, then 10 gated stages (docs, brainstorm+decompose, spec, plan, subagent build, tests, lint/deploy, post-deploy, docs/wiki, acceptance). Every stage's doctrine is built into the skill — no companion plugin required — with typed auto/manual gates, a frozen requirement spine that closes with evidence, a work board and a verification ledger that outlive a run, an exposure line naming what shipped unconfirmed, a progress rail computed from the project's own config, a loop guard whose review ceiling measures rather than stops, and stage-3 tracks for what a product does, how it sounds and how it looks. Two modes need no task: `checkup` (what is unverified) and `setup` (audit existing docs). Retro insights can publish upstream as issues, opt-in and redacted.",
|
|
5
|
-
"version": "1.
|
|
5
|
+
"description": "Runs a substantial task through a mandatory built-in intake grill, then 10 gated stages (docs, brainstorm+decompose, spec, plan, subagent build, tests, lint/deploy, post-deploy, docs/wiki, acceptance). Every stage's doctrine is built into the skill — no companion plugin required — with typed auto/judgment/manual gates, a frozen requirement spine that closes with evidence, a work board and a verification ledger that outlive a run, an exposure line naming what shipped unconfirmed, a progress rail computed from the project's own config, a loop guard whose review ceiling measures rather than stops, and stage-3 tracks for what a product does, how it sounds and how it looks. Two modes need no task: `checkup` (what is unverified) and `setup` (audit existing docs). Retro insights can publish upstream as issues, opt-in and redacted.",
|
|
6
|
+
"version": "1.80.0",
|
|
6
7
|
"author": {
|
|
7
8
|
"name": "ssheleg",
|
|
8
9
|
"url": "https://x.com/sshlg93"
|
|
@@ -100,7 +100,8 @@ Anything deferred enters the carry-over ledger the moment it is said.
|
|
|
100
100
|
| 9 | Docs + wiki | **three artifacts, not two** — module docs, the wiki, **and the code graph** |
|
|
101
101
|
| 10 | Acceptance | the ladder walk first, then the table, then the retrospective |
|
|
102
102
|
|
|
103
|
-
**Honor every gate by its type**: `auto` — verify the check yourself; `
|
|
103
|
+
**Honor every gate by its type**: `auto` — verify the check yourself; `judgment` —
|
|
104
|
+
record the named judge's ruling as judgement, never as a measurement; `manual` — wait
|
|
104
105
|
for an explicit go.
|
|
105
106
|
|
|
106
107
|
## Cross-cutting — the three that fire at any stage
|
|
@@ -64,7 +64,12 @@ def build_stage_id():
|
|
|
64
64
|
try:
|
|
65
65
|
cfg = json.load(open(os.path.join(project, "pipeline.json"), encoding="utf-8"))
|
|
66
66
|
for s in cfg.get("stages") or []:
|
|
67
|
-
|
|
67
|
+
# `build` OR `dev` — mirroring this hook's own ledger fallback
|
|
68
|
+
# (`build|dev`) below. Matching only `build` meant the shipped
|
|
69
|
+
# pipeline.example.json, whose build stage is `state: "dev"`, never
|
|
70
|
+
# armed this gate: the canonical config disarmed the hook it ships
|
|
71
|
+
# beside.
|
|
72
|
+
if isinstance(s, dict) and s.get("state") in ("build", "dev"):
|
|
68
73
|
return str(s.get("id"))
|
|
69
74
|
except Exception:
|
|
70
75
|
pass
|
|
@@ -49,40 +49,61 @@ try:
|
|
|
49
49
|
except Exception:
|
|
50
50
|
raise SystemExit(0)
|
|
51
51
|
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
52
|
+
# EVERY stage that declares a command, not the first. A first-match scan meant
|
|
53
|
+
# a lint stage declared before the tests stage was the only one ever observed,
|
|
54
|
+
# so the real suite's runs left no trace for the release gate to corroborate —
|
|
55
|
+
# the same first-match break the release gate's own stage scan shipped with,
|
|
56
|
+
# fixed together (sweep the class, not the instance).
|
|
57
|
+
declared = [(str(s.get("id")), (s.get("gate") or {})["command"].strip())
|
|
58
|
+
for s in cfg.get("stages") or []
|
|
59
|
+
if isinstance(s, dict) and (s.get("gate") or {}).get("command")]
|
|
60
|
+
if not declared:
|
|
58
61
|
raise SystemExit(0)
|
|
59
62
|
|
|
60
|
-
stage_id, declared = target
|
|
61
|
-
|
|
62
63
|
# Compared on the normalised command line, not on a substring: `echo "npm test"`
|
|
63
64
|
# and `npm test --watch` are not the project's gate, and treating them as one puts
|
|
64
65
|
# a fabricated observation in the ledger the release gate trusts.
|
|
65
66
|
def norm(s):
|
|
66
67
|
return " ".join(s.split())
|
|
67
68
|
|
|
68
|
-
if norm(cmd)
|
|
69
|
+
matches = [(sid, d) for sid, d in declared if norm(cmd) == norm(d)]
|
|
70
|
+
if not matches:
|
|
69
71
|
raise SystemExit(0)
|
|
70
72
|
|
|
71
73
|
# PostToolUse fires on success; PostToolUseFailure carries the error. Both are
|
|
72
74
|
# wired to this script, and `error` present means the command did not exit 0.
|
|
73
75
|
failed = bool(data.get("error")) or data.get("hook_event_name") == "PostToolUseFailure"
|
|
74
|
-
|
|
75
|
-
|
|
76
|
+
# The harness documents the result field as `tool_response`; `tool_output` is the
|
|
77
|
+
# name this script shipped reading, so it stays as a fallback rather than a
|
|
78
|
+
# breaking change. Reading only the wrong name left the exit-code branch dead.
|
|
79
|
+
# The fallback is BY FIELD, not by object: the real Bash `tool_response` is
|
|
80
|
+
# {stdout, stderr, interrupted} with no exit_code, and `resp or legacy` made a
|
|
81
|
+
# legacy exit_code unreachable behind it — found by the R-005 reader, measured,
|
|
82
|
+
# before this shipped. The same reading found `interrupted`: a gate cut short
|
|
83
|
+
# is not a gate that passed, whatever a stale exit_code says, so a failure
|
|
84
|
+
# event or an interruption is never recorded as exit 0.
|
|
85
|
+
out = {}
|
|
86
|
+
interrupted = False
|
|
87
|
+
for cand in (data.get("tool_response"), data.get("tool_output")):
|
|
88
|
+
if isinstance(cand, dict):
|
|
89
|
+
interrupted = interrupted or bool(cand.get("interrupted"))
|
|
90
|
+
if not out and cand.get("exit_code") is not None:
|
|
91
|
+
out = cand
|
|
92
|
+
if out:
|
|
76
93
|
code = int(out["exit_code"])
|
|
77
94
|
else:
|
|
78
95
|
code = 1 if failed else 0
|
|
96
|
+
if (failed or interrupted) and code == 0:
|
|
97
|
+
code = 1
|
|
79
98
|
|
|
80
99
|
stamp = datetime.datetime.now(datetime.timezone.utc).replace(microsecond=0).isoformat().replace("+00:00", "Z")
|
|
81
|
-
line = 'gate: %s — command "%s" — exit %d — %s\n' % (stage_id, norm(declared), code, stamp)
|
|
82
100
|
|
|
83
101
|
# Append-only, like every other line in this file. A ledger that is rewritten is a
|
|
84
|
-
# ledger whose history can be edited to say the suite passed.
|
|
102
|
+
# ledger whose history can be edited to say the suite passed. One line per stage
|
|
103
|
+
# that declared this exact command: two stages declaring the same command is two
|
|
104
|
+
# observations, because each stage's gate is corroborated separately.
|
|
85
105
|
with open(ledger, "a", encoding="utf-8") as fh:
|
|
86
|
-
|
|
106
|
+
for stage_id, decl in matches:
|
|
107
|
+
fh.write('gate: %s — command "%s" — exit %d — %s\n' % (stage_id, norm(decl), code, stamp))
|
|
87
108
|
PY
|
|
88
109
|
exit 0
|
|
@@ -54,13 +54,36 @@ decide() {
|
|
|
54
54
|
# JSON from there. Watched failing — the gate allowed every release, silently,
|
|
55
55
|
# because `sys.stdin.read()` came back empty and an empty payload is a skip.
|
|
56
56
|
HOOK_INPUT="$input" python3 - "$ledger" "$project" <<'PY'
|
|
57
|
-
import json, shlex, sys, os, re
|
|
57
|
+
import datetime, json, shlex, sys, os, re
|
|
58
58
|
|
|
59
59
|
ledger, project = sys.argv[1], sys.argv[2]
|
|
60
60
|
raw = os.environ.get("HOOK_INPUT", "")
|
|
61
61
|
try:
|
|
62
62
|
data = json.loads(raw)
|
|
63
|
-
|
|
63
|
+
if not isinstance(data, dict):
|
|
64
|
+
raise ValueError("payload is not a JSON object")
|
|
65
|
+
except Exception as exc:
|
|
66
|
+
# An empty or unreadable payload is the gate NOT SEEING — it says nothing
|
|
67
|
+
# about whether a release is happening. With no run in flight that is
|
|
68
|
+
# nobody's business, so stay silent. With a run in flight, a silent skip
|
|
69
|
+
# here is the fail-open this hook exists to close: watched happening —
|
|
70
|
+
# `printf '' | release-gate.sh` exited 0 with no output, and the gate
|
|
71
|
+
# allowed every release for as long as its stdin plumbing was broken. A
|
|
72
|
+
# component that never receives its input is indistinguishable from
|
|
73
|
+
# approval, so it fails CLOSED, and the blindness is written into the
|
|
74
|
+
# ledger (append-only, the lifecycle grammar) so the run can see the gate
|
|
75
|
+
# went blind even after the refusal scrolls away.
|
|
76
|
+
if os.path.exists(ledger):
|
|
77
|
+
why = ("empty payload" if not raw.strip()
|
|
78
|
+
else "unparseable payload (%s)" % type(exc).__name__)
|
|
79
|
+
try:
|
|
80
|
+
stamp = (datetime.datetime.now(datetime.timezone.utc)
|
|
81
|
+
.replace(microsecond=0).isoformat().replace("+00:00", "Z"))
|
|
82
|
+
with open(ledger, "a", encoding="utf-8") as fh:
|
|
83
|
+
fh.write("event: gate-blind — %s — %s\n" % (why, stamp))
|
|
84
|
+
except Exception:
|
|
85
|
+
pass # recording failed; the block below still stands
|
|
86
|
+
print("blind\t%s" % why); raise SystemExit(0)
|
|
64
87
|
print("skip"); raise SystemExit(0)
|
|
65
88
|
|
|
66
89
|
cmd = ((data.get("tool_input") or {}).get("command") or "")
|
|
@@ -99,8 +122,22 @@ def outward(tokens):
|
|
|
99
122
|
return "git push <tag>"
|
|
100
123
|
if t.endswith("gh") and rest[:2] == ["release", "create"]:
|
|
101
124
|
return "gh release create"
|
|
102
|
-
if t.endswith("npm")
|
|
103
|
-
|
|
125
|
+
if t.endswith("npm"):
|
|
126
|
+
# `publish` must be npm's SUBCOMMAND — the first non-flag token after
|
|
127
|
+
# npm. `"publish" in rest` matched anywhere in the argument list, so a
|
|
128
|
+
# project's own `npm run publish` script was gated as the registry
|
|
129
|
+
# act; fail-closed overmatch is still overmatch, and a gate that
|
|
130
|
+
# fights an ordinary script daily is a gate that gets removed. Known
|
|
131
|
+
# narrowness, stated rather than discovered: a flag whose value is a
|
|
132
|
+
# separate token (`npm --loglevel silent publish`) hides the
|
|
133
|
+
# subcommand from this scan; the `--flag=value` spelling is read
|
|
134
|
+
# correctly.
|
|
135
|
+
for a in rest:
|
|
136
|
+
if a.startswith("-"):
|
|
137
|
+
continue
|
|
138
|
+
if a == "publish":
|
|
139
|
+
return "npm publish"
|
|
140
|
+
break
|
|
104
141
|
return None
|
|
105
142
|
|
|
106
143
|
|
|
@@ -122,16 +159,25 @@ stage_lines = [l.strip() for l in text.splitlines() if l.strip().startswith("sta
|
|
|
122
159
|
|
|
123
160
|
|
|
124
161
|
def declared_test_stage():
|
|
125
|
-
"""The tests stage, from the project's own flow. `(id, command)` or None.
|
|
162
|
+
"""The tests stage, from the project's own flow. `(id, command)` or None.
|
|
163
|
+
|
|
164
|
+
TWO PASSES, and the order is the fix. One pass took the first stage that was
|
|
165
|
+
declared `tests` OR merely carried a gate command — so a lint stage declared
|
|
166
|
+
before the tests stage became "the tests stage", and its green observation
|
|
167
|
+
released a tag with the suite never run. A stage the project DECLARED as
|
|
168
|
+
tests outranks any stage that happens to carry a command; the command-bearing
|
|
169
|
+
fallback remains for flows that declare no `tests` state at all."""
|
|
126
170
|
try:
|
|
127
171
|
cfg = json.load(open(os.path.join(project, "pipeline.json"), encoding="utf-8"))
|
|
128
172
|
except Exception:
|
|
129
173
|
return None
|
|
130
|
-
for s in cfg.get("stages") or []
|
|
131
|
-
|
|
132
|
-
|
|
174
|
+
stages = [s for s in cfg.get("stages") or [] if isinstance(s, dict)]
|
|
175
|
+
for s in stages:
|
|
176
|
+
if s.get("state") == "tests":
|
|
177
|
+
return (str(s.get("id")), (s.get("gate") or {}).get("command"))
|
|
178
|
+
for s in stages:
|
|
133
179
|
gate = s.get("gate") or {}
|
|
134
|
-
if
|
|
180
|
+
if gate.get("command"):
|
|
135
181
|
return (str(s.get("id")), gate.get("command"))
|
|
136
182
|
return None
|
|
137
183
|
|
|
@@ -207,6 +253,21 @@ why=$(printf '%s' "$verdict" | cut -f3)
|
|
|
207
253
|
|
|
208
254
|
case "$state" in
|
|
209
255
|
skip|ok) exit 0 ;;
|
|
256
|
+
blind)
|
|
257
|
+
# act holds the reason here — the payload never carried a command to name.
|
|
258
|
+
cat >&2 <<EOF
|
|
259
|
+
task-pipeline: this project has a run in flight ($ledger) and the release gate
|
|
260
|
+
received an unreadable hook payload ($act), so it cannot tell whether this
|
|
261
|
+
command is an outward act. A gate that cannot see fails CLOSED — a blind gate
|
|
262
|
+
that waves things through is indistinguishable from approval, and that is the
|
|
263
|
+
exact shape it once shipped with.
|
|
264
|
+
|
|
265
|
+
An \`event: gate-blind\` line was appended to the ledger. Check the hook wiring
|
|
266
|
+
(stdin must carry the PreToolUse JSON), then re-run the command. To work without
|
|
267
|
+
the pipeline, remove the ledger ($ledger) or say «без пайплайна» and take the
|
|
268
|
+
route by hand.
|
|
269
|
+
EOF
|
|
270
|
+
exit 2 ;;
|
|
210
271
|
block)
|
|
211
272
|
cat >&2 <<EOF
|
|
212
273
|
task-pipeline: \`$act\` is an outward, irreversible act and the tests gate has not
|
|
@@ -24,6 +24,7 @@ The full statement of each canon, its rationale and its enforcement live in
|
|
|
24
24
|
7. **Silence is not a pass** — ask what a mechanism prints when it did not look.
|
|
25
25
|
8. **An estimate is never announced as a measurement** — a rule states its evidence condition.
|
|
26
26
|
9. **What was not checked is printed beside what was.**
|
|
27
|
+
- **9a. A measured zero and an unmeasured quantity may not print the same** — canon 9 says carry the absence; 9a says refuse the number when nothing measured it.
|
|
27
28
|
10. **The document ships in the change that made it true** — and a correction is appended, never written over.
|
|
28
29
|
|
|
29
30
|
They are **epistemic**: what makes a claim documentation. The operational layer — what to
|
|
@@ -62,8 +63,38 @@ way to teach an agent to route around the rule where it actually protects someth
|
|
|
62
63
|
pass through here: do it directly and **say out loud** that the claim is unbacked, rather
|
|
63
64
|
than presenting an estimate as a measurement (canon 8).
|
|
64
65
|
|
|
66
|
+
## Whom the canons serve
|
|
67
|
+
|
|
68
|
+
**The operator's intent is the point. Evidence is how it survives contact with reality —
|
|
69
|
+
not a licence to refuse it.**
|
|
70
|
+
|
|
71
|
+
Everything above exists so the operator gets the result they actually wanted and nothing
|
|
72
|
+
breaks quietly on the way. It does not exist to decide *what they are allowed to say*. A
|
|
73
|
+
skill that reads canon 8 as "I may not write an unverified claim" and refuses the work has
|
|
74
|
+
inverted its own purpose: it protected a rule and lost the person the rule was for.
|
|
75
|
+
|
|
76
|
+
The line, and it is not the same line:
|
|
77
|
+
|
|
78
|
+
| The operator may | The operator may not, and this is refusal ground |
|
|
79
|
+
|---|---|
|
|
80
|
+
| assert something not yet true — a landing page describing the product they are building, a roadmap, a pitch | make a **measurement** say something it did not: a test that passed, a benchmark, a count, a citation |
|
|
81
|
+
| ship an unbacked claim knowingly, after being told once | have an estimate **presented as** a measurement (canon 8), which is the one thing no intent authorises |
|
|
82
|
+
|
|
83
|
+
**So the sequence on an unproven claim is: say it once, label it, do the work.** Not a
|
|
84
|
+
negotiation, not a second warning, and never a silent refusal dressed as a question. If
|
|
85
|
+
the operator confirms they know, the claim ships with a marker naming it as forward-looking
|
|
86
|
+
— that marker is the whole of what this skill owes here, and the task gets done.
|
|
87
|
+
|
|
88
|
+
**Then do the harder half.** Following intent is the floor, not the service. The service is
|
|
89
|
+
making that intent *better* — more structural, more predictable, easier to maintain and
|
|
90
|
+
extend than the operator asked for — while still being the thing they asked for. An agent
|
|
91
|
+
that only obeys is a slower keyboard.
|
|
92
|
+
|
|
65
93
|
## The one test
|
|
66
94
|
|
|
67
95
|
Before a document ships, read it for the sentence that would embarrass you if someone
|
|
68
|
-
asked *"how do you know?"* —
|
|
69
|
-
|
|
96
|
+
asked *"how do you know?"* — then do one of **three** things: give that sentence its
|
|
97
|
+
address, delete it, or **mark it as an unbacked claim the operator chose to make and ship
|
|
98
|
+
it**. The third option is not a loophole; it is the reason the other two are worth
|
|
99
|
+
anything. A rule with no way to proceed under it becomes a rule people route around, and
|
|
100
|
+
then nothing carries an address.
|
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: project-audit
|
|
3
3
|
description: "Use when someone asks what is actually true of a whole project right now — what is finished, what is half-built, what is broken, and what nobody has looked at. Walks a cold start: discover what the project is, run a registry of probes chosen from that, read production evidence (published artefact against source, CI history, telemetry present or absent), then leave a self-contained HTML report and a JSON sidecar so the next audit can say what moved. Read-only: it proposes board rows and commits nothing. Triggers - 'project audit', 'audit the project', 'codebase audit', 'state of the project', 'what is unfinished', 'project health check', 'аудит проекта', 'проаудируй проект', 'состояние проекта', 'что не доделано', 'аудит кодовой базы'. Not for: auditing one deliverable inside a run (that is the pipeline's own ladder), reviewing a diff, or checking a skill's construction — say 'без диагностики' to opt out."
|
|
4
|
+
compatibility: "The collector (scripts/audit.py) needs python3 and reads committed state, so it needs git. Probes needing gh, npm, network or a browser declare it and report blind when it is absent — degraded, never silent."
|
|
4
5
|
---
|
|
5
6
|
|
|
6
7
|
# Project audit — what is true of this project right now
|
|
@@ -33,6 +34,7 @@ next audit reads.
|
|
|
33
34
|
| this skill | the **procedure** — cold start, probes, production, the report | a whole project is the subject |
|
|
34
35
|
| `/skill-audit` (make-skill) | a skill's construction against the standard | the thing audited is a skill or plugin |
|
|
35
36
|
| `/ux-audit` (super-ux) | code against documented scenarios | the question is user-facing behaviour |
|
|
37
|
+
| `/seo-aeo-audit` (seo-aeo-audit) | a public surface's search and answer-engine visibility | the question is whether a machine will find it |
|
|
36
38
|
|
|
37
39
|
**The method is not restated here.** Phase 4 below hands off to `audit.md` and
|
|
38
40
|
comes back; a second copy of the ladder would be a second rule, and the two
|
|
@@ -96,8 +98,11 @@ the same object.
|
|
|
96
98
|
### 6. Propose — rows, not edits
|
|
97
99
|
|
|
98
100
|
**This skill commits nothing.** Findings leave as board rows in the project's
|
|
99
|
-
own vocabulary, priced with the
|
|
100
|
-
|
|
101
|
+
own vocabulary, priced with **the board header's declared formula** — the shipped
|
|
102
|
+
default is `Sev × Blast + age_bonus` (`references/backlog.md`, the pipeline's
|
|
103
|
+
board doctrine) — and the operator accepts them. Effort never ranks inside an
|
|
104
|
+
audit: what a fix costs is the fixer's decision, not the finder's
|
|
105
|
+
(`references/prioritisation.md`). An audit
|
|
101
106
|
that edits while it reads cannot be re-run to check itself.
|
|
102
107
|
|
|
103
108
|
## Three verdicts, and why the third one exists
|