swe-workflow-skills 0.6.3 → 0.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +11 -4
- package/VERSION +1 -1
- package/catalog.json +2 -2
- package/package.json +1 -1
- package/skills/architecture-design/evals/evals.json +2 -3
- package/skills/architecture-documentation/evals/evals.json +47 -44
- package/skills/cicd-pipeline/evals/evals.json +13 -0
- package/skills/code-reviewing/SKILL.md +24 -6
- package/skills/code-reviewing/evals/evals.json +19 -3
- package/skills/code-slop-cleanup/SKILL.md +7 -0
- package/skills/code-slop-cleanup/evals/evals.json +13 -0
- package/skills/code-slop-cleanup/references/slop-patterns.md +4 -1
- package/skills/containerization/evals/evals.json +12 -0
- package/skills/effort-estimation/evals/evals.json +13 -0
- package/skills/feature-planning/evals/evals.json +2 -3
- package/skills/frontend-architecture/evals/evals.json +12 -0
- package/skills/git-workflow/SKILL.md +119 -8
- package/skills/git-workflow/evals/evals.json +37 -5
- package/skills/git-workflow/templates/pull-request.md +10 -5
- package/skills/infrastructure-as-code/evals/evals.json +12 -0
- package/skills/metrics-and-okrs/evals/evals.json +12 -0
- package/skills/ml-experiment-tracking/evals/evals.json +12 -0
- package/skills/ml-model-deployment/evals/evals.json +12 -0
- package/skills/plan-execution/SKILL.md +38 -3
- package/skills/plan-execution/evals/evals.json +30 -0
- package/skills/prd-writing/evals/evals.json +13 -0
- package/skills/project-documentation/SKILL.md +73 -6
- package/skills/project-documentation/evals/evals.json +17 -2
- package/skills/project-proposal/evals/evals.json +12 -0
- package/skills/release-management/SKILL.md +23 -1
- package/skills/retrospective/evals/evals.json +13 -0
- package/skills/strategic-review/evals/evals.json +7 -7
- package/skills/ui-ux-design/SKILL.md +12 -0
- package/skills/ui-ux-design/evals/evals.json +12 -0
- package/skills/verification-before-completion/SKILL.md +109 -2
- package/skills/verification-before-completion/evals/evals.json +66 -7
- package/skills/verification-before-completion/references/pre-commit-gate.md +239 -0
- package/skills/writing-skills/SKILL.md +43 -7
|
@@ -26,6 +26,18 @@
|
|
|
26
26
|
"Estimates account for ongoing operational cost (not just build cost)",
|
|
27
27
|
"Risk section addresses real-time infrastructure complexity"
|
|
28
28
|
]
|
|
29
|
+
},
|
|
30
|
+
{
|
|
31
|
+
"id": 3,
|
|
32
|
+
"prompt": "Forget the business case, the VP already approved the budget. I just need the technical call: Kafka or RabbitMQ for our event backbone? We do about 5K events/sec with ordering guarantees per customer.",
|
|
33
|
+
"expected_output": "Scope boundary — the budget decision is already made, and choosing between two messaging technologies is a costly-to-reverse structural decision belonging to architecture-design (an ADR). Should hand off rather than producing a proposal or business case.",
|
|
34
|
+
"assertions": [
|
|
35
|
+
"Recognizes that budget approval is already granted, so no proposal is needed",
|
|
36
|
+
"Identifies the Kafka-versus-RabbitMQ choice as a costly-to-reverse architectural decision",
|
|
37
|
+
"Hands off to or invokes the architecture-design skill to record it as an ADR",
|
|
38
|
+
"Does not produce a project proposal, business case, or ROI analysis",
|
|
39
|
+
"Does not make the technology call inside the project-proposal workflow"
|
|
40
|
+
]
|
|
29
41
|
}
|
|
30
42
|
]
|
|
31
43
|
}
|
|
@@ -14,6 +14,20 @@ Releasing turns committed work into a versioned, consumable artifact — a tag,
|
|
|
14
14
|
|
|
15
15
|
Registry publishes are effectively irreversible: a version number, once published, is burned even if you yank it, and whatever shipped inside it is already on users' machines. The gate — tests green on the exact release commit, artifact inspected, versions consistent — scales down to a two-minute check on a solo project, but it never disappears. Ceremony scales with the project; the gate doesn't.
|
|
16
16
|
|
|
17
|
+
## Scope boundary: "is it safe to ship?" is not a release question
|
|
18
|
+
|
|
19
|
+
Deploy safety — migrations, downtime, rollback readiness, whether today is the
|
|
20
|
+
day — belongs to `deployment-checklist`, with `rollback-strategy` for the
|
|
21
|
+
reversal plan. A release produces the artifact; the deploy decides whether it
|
|
22
|
+
runs in production, and the two are asked about in the same breath constantly.
|
|
23
|
+
|
|
24
|
+
When the question is about deploying rather than versioning, name the owning
|
|
25
|
+
skill and hand off **without** answering in release terms as well. Semver,
|
|
26
|
+
changelogs, tagging, and registry publishing are not a useful partial answer to
|
|
27
|
+
"we have two pending migrations and it's Friday" — they are this skill's
|
|
28
|
+
vocabulary applied to a question that didn't ask for it, and they bury the part
|
|
29
|
+
that mattered.
|
|
30
|
+
|
|
17
31
|
## Step 1: Detect Existing Conventions — Never Assume Greenfield
|
|
18
32
|
|
|
19
33
|
Before proposing anything, find what the project already does:
|
|
@@ -35,6 +49,13 @@ If conventions exist, follow them for this release; propose improvements as a se
|
|
|
35
49
|
|
|
36
50
|
Under-processing breaks consumers; over-processing kills a small project's momentum. State which tier applies and why. **Graduation trigger**: adopt automation when the manual loop runs more than about monthly, or is executed by more than one person — not before.
|
|
37
51
|
|
|
52
|
+
**The table is how you choose, not what you deliver.** Name the tier, then work in
|
|
53
|
+
that tier only. Walking a solo maintainer through release trains, protected release
|
|
54
|
+
branches, and provenance infrastructure — even as context, even as "what you'd do
|
|
55
|
+
later" — is the over-processing the Iron Law forbids, and it buries the four steps
|
|
56
|
+
they actually need. The graduation trigger is what replaces the tour: one line on
|
|
57
|
+
what would move them up, not a preview of the tier above.
|
|
58
|
+
|
|
38
59
|
## Step 3: Decide the Version
|
|
39
60
|
|
|
40
61
|
- **Something with consumers of a versioned contract (library, CLI, API)** → SemVer: breaking → MAJOR, feature → MINOR, fix → PATCH. Pre-1.0, MINOR is the breaking-change lever — and don't promise 1.0 stability until you mean it.
|
|
@@ -46,7 +67,7 @@ Under-processing breaks consumers; over-processing kills a small project's momen
|
|
|
46
67
|
|
|
47
68
|
- **Changelog** from commits, in [Keep a Changelog](https://keepachangelog.com/) form. Conventional commits map mechanically: `feat` → Added, `fix` → Fixed, behavior change → Changed, `feat!` / `BREAKING CHANGE:` → MAJOR.
|
|
48
69
|
- **Release notes ≠ changelog**: notes lead with highlights and breaking changes plus their upgrade steps — what a consumer must know before upgrading, not a commit list.
|
|
49
|
-
- Bump the source-of-truth version, regenerate anything stamped from it, promote `[Unreleased]`
|
|
70
|
+
- Bump the source-of-truth version, regenerate anything stamped from it, and settle the changelog — promote `[Unreleased]` if the file is hand-maintained; if a tool generates it, don't hand-curate alongside (a generated changelog plus a hand-written `[Unreleased]` is how the two silently diverge — the tool prepends and never folds). There, fix the commit subject, not the file.
|
|
50
71
|
|
|
51
72
|
## Step 5: Gate, Then Publish
|
|
52
73
|
|
|
@@ -78,6 +99,7 @@ Selection detail, trusted publishing/provenance, pre-release channels and dist-t
|
|
|
78
99
|
| "Let's set up semantic-release" (solo repo, three releases a year) | Below regular cadence, automation costs more than it saves. Right-sizing cuts both ways. |
|
|
79
100
|
| "Tests passed yesterday on main" | The gate runs on the exact release commit — the version bump and changelog commit included. |
|
|
80
101
|
| "We'll fix the notes after publishing" | Notes and changelog are part of the artifact; consumers read them at upgrade time, which is now. |
|
|
102
|
+
| "They asked about shipping, so some release guidance is still useful" | They asked about deploy safety. Versioning advice alongside it reads as the answer and displaces the migration and rollback questions that actually decide it. Hand off and stop. |
|
|
81
103
|
|
|
82
104
|
## Red Flags — Stop and Correct Course
|
|
83
105
|
|
|
@@ -29,6 +29,19 @@
|
|
|
29
29
|
"Each action item has an owner",
|
|
30
30
|
"Includes review of previous retro action items"
|
|
31
31
|
]
|
|
32
|
+
},
|
|
33
|
+
{
|
|
34
|
+
"id": 3,
|
|
35
|
+
"prompt": "The payment service is down right now — checkout has been failing for about ten minutes and customers are tweeting at us. Let's start the post-mortem so we capture everything while it's fresh.",
|
|
36
|
+
"expected_output": "Scope boundary — this is an active production incident, not a retrospective. Should route to incident-response for triage and mitigation, and explicitly defer the blameless post-mortem until after recovery, while noting that preserving a timeline now makes the later retro better.",
|
|
37
|
+
"assertions": [
|
|
38
|
+
"Identifies that production is currently down, so this is an active incident rather than a retrospective",
|
|
39
|
+
"Hands off to or invokes the incident-response skill for triage and mitigation",
|
|
40
|
+
"Does not run the post-mortem or retrospective workflow while the service is down",
|
|
41
|
+
"Explicitly defers the blameless post-mortem until after recovery",
|
|
42
|
+
"Does not ask reflective retro questions (what went well / 5 Whys) during the outage",
|
|
43
|
+
"May suggest capturing a timeline during the incident so the later post-mortem is accurate"
|
|
44
|
+
]
|
|
32
45
|
}
|
|
33
46
|
]
|
|
34
47
|
}
|
|
@@ -3,12 +3,12 @@
|
|
|
3
3
|
"evals": [
|
|
4
4
|
{
|
|
5
5
|
"id": 1,
|
|
6
|
-
"prompt": "We're
|
|
7
|
-
"expected_output": "A strategic review that states the project's thesis and its load-bearing assumption, critiques the value proposition and positioning,
|
|
6
|
+
"prompt": "We're taking Cascade public in a couple of months. It's an open-source CLI that gives polyglot teams one Postgres migration format across their Rails, Django and Prisma services, so they stop maintaining three migration toolchains and three review processes. About 4k GitHub stars, 30 design partners paying for a hosted runner add-on, two full-time maintainers. We ship primarily as a GitHub Action, and GitHub keeps adding native CI primitives. Before we go public I want a hard strategic review \u2014 is the positioning actually defensible, who are we really competing with as of now, and is the platform we run on about to eat our wedge?",
|
|
7
|
+
"expected_output": "A strategic review that states the project's thesis and its load-bearing assumption, critiques the value proposition and positioning, treats the competitive picture as current-evidence work (dated, confirmed-vs-inferred, deep sweep delegated to deep-research) tagging each comparable as competitor/complement/integration target, assesses platform-absorption risk, and ends in strategic forks with trade-offs plus a recommended path. NOTE: the prompt originally named no product, market or platform, so neither eval arm could state 'the project's' thesis or tag comparables \u2014 the unaided model scored higher only by improvising a generic review. The concrete project was added so the assertions are answerable. Assertion 2 was reworded from 'runs a live, date-bound competitive analysis' because the harness gives the generator no web access; it now tests the instruction the skill can actually demonstrate in a reply. Verify real search behavior in a live session instead.",
|
|
8
8
|
"assertions": [
|
|
9
9
|
"States the project's strategic thesis in one sentence and identifies the load-bearing assumption to test",
|
|
10
10
|
"Critiques the value proposition / positioning, separating what's strong from what rests on unproven assumptions",
|
|
11
|
-
"
|
|
11
|
+
"Treats the competitive picture as something to verify against current sources rather than asserted from memory: dates its findings, marks them confirmed vs inferred, and routes a deep multi-source sweep to deep-research",
|
|
12
12
|
"Tags comparables by relationship (competitor / complement / integration target)",
|
|
13
13
|
"Assesses platform-absorption / wedge-closure risk and names what would invalidate the thesis",
|
|
14
14
|
"Ends with strategic forks (trade-offs) and a recommended path, labeling findings confirmed vs inferred and dated",
|
|
@@ -17,12 +17,12 @@
|
|
|
17
17
|
},
|
|
18
18
|
{
|
|
19
19
|
"id": 2,
|
|
20
|
-
"prompt": "Honestly the space
|
|
21
|
-
"expected_output": "Confronts the weak-wedge situation directly: tests whether a real moat exists vs a temporary head start, maps the overlapping platform features with current evidence, and presents honest forks (narrow scope / reposition / pivot) with a recommendation rather than reassurance.",
|
|
20
|
+
"prompt": "Honestly the space Cascade is in feels crowded and I'm not sure what makes us different anymore. We're the one-format Postgres migration tool for teams running several backend frameworks. Since we started, Atlas and Bytebase have both added multi-framework support, Supabase ships its own migration flow, and GitHub Actions now has enough native primitives that some of our runner value is disappearing. Is there even a defensible wedge here?",
|
|
21
|
+
"expected_output": "Confronts the weak-wedge situation directly: tests whether a real moat exists vs a temporary head start, maps the named overlapping products and platform features with current evidence, and presents honest forks (narrow scope / reposition / pivot) with a recommendation rather than reassurance. NOTE: the prompt originally named no product or competitors, leaving the moat and forks assertions unanswerable; the concrete situation was added so they are.",
|
|
22
22
|
"assertions": [
|
|
23
23
|
"Distinguishes a durable moat (standard, network, data, switching cost) from a temporary head start",
|
|
24
24
|
"Examines the overlapping platform/incumbent features with current, dated evidence",
|
|
25
|
-
"Does not offer false reassurance
|
|
25
|
+
"Does not offer false reassurance \u2014 willing to conclude the wedge is thin if that's what the evidence shows",
|
|
26
26
|
"Presents distinct strategic forks (e.g. narrow / reposition / pivot) with trade-offs and a recommendation",
|
|
27
27
|
"Names the observable event that would invalidate continuing"
|
|
28
28
|
]
|
|
@@ -30,7 +30,7 @@
|
|
|
30
30
|
{
|
|
31
31
|
"id": 3,
|
|
32
32
|
"prompt": "I'm writing the one-line tagline for our landing page. Should it say 'fast' or 'simple'?",
|
|
33
|
-
"expected_output": "Recognizes this is a narrow copy choice, not a strategic review
|
|
33
|
+
"expected_output": "Recognizes this is a narrow copy choice, not a strategic review \u2014 answers directly (with brief reasoning) rather than launching a full vision/market analysis.",
|
|
34
34
|
"assertions": [
|
|
35
35
|
"Recognizes this is a small copy/messaging question, not a strategic review",
|
|
36
36
|
"Does not launch the full strategic-review workflow (thesis, market scan, forks)",
|
|
@@ -12,6 +12,18 @@ Design user interfaces from a senior engineer's perspective. This isn't about pi
|
|
|
12
12
|
|
|
13
13
|
**Design the states, not just the happy path.** Every screen has at least 5 states: loading, empty, partial data, full data, and error. Designing only the "full data" state is the #1 cause of poor user experience.
|
|
14
14
|
|
|
15
|
+
## Scope boundary: naming the owner is not the same as answering anyway
|
|
16
|
+
|
|
17
|
+
State-management library choice (Zustand vs Redux Toolkit vs Context), the
|
|
18
|
+
server/client state split, data fetching, and component folder structure belong
|
|
19
|
+
to `frontend-architecture`. When the ask turns to those, hand off **and stop**:
|
|
20
|
+
name the skill, carry the approved flows and screen states forward as the
|
|
21
|
+
constraints the architecture has to satisfy, and leave the choice to it.
|
|
22
|
+
|
|
23
|
+
Do not pick the library "just as a starting point". A recommendation offered in
|
|
24
|
+
passing is the one the reader acts on, and it pre-empts the trade-off the owning
|
|
25
|
+
skill exists to work through — with none of its reasoning attached.
|
|
26
|
+
|
|
15
27
|
## Workflow
|
|
16
28
|
|
|
17
29
|
### Step 1: Map the User Flow
|
|
@@ -30,6 +30,18 @@
|
|
|
30
30
|
"Mobile layout addresses keyboard handling for address/payment forms",
|
|
31
31
|
"Handles edge case: cart modified during checkout, session timeout"
|
|
32
32
|
]
|
|
33
|
+
},
|
|
34
|
+
{
|
|
35
|
+
"id": 3,
|
|
36
|
+
"prompt": "Flows and wireframes are approved. Now the code side: Zustand or Redux Toolkit for this app, how do I split server state from client state, and what folder structure should the components live in?",
|
|
37
|
+
"expected_output": "Scope boundary — state-management library choice, server/client state separation and code organization are frontend-architecture concerns. Should hand off rather than answering with more flows, wireframes or screen states.",
|
|
38
|
+
"assertions": [
|
|
39
|
+
"Recognizes that state management and code organization are frontend-architecture concerns",
|
|
40
|
+
"Hands off to or invokes the frontend-architecture skill",
|
|
41
|
+
"Does not answer with wireframes, user flows, or screen-state specifications",
|
|
42
|
+
"Does not pick between Zustand and Redux Toolkit inside the ui-ux-design workflow",
|
|
43
|
+
"May carry forward the approved flows as the constraints the architecture has to satisfy"
|
|
44
|
+
]
|
|
33
45
|
}
|
|
34
46
|
]
|
|
35
47
|
}
|
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: verification-before-completion
|
|
3
|
-
description: "Evidence gate before claiming work is done, passing, fixed, or deployed. Use before saying 'done', 'it works', or 'tests pass', or before committing, pushing, or opening a PR.
|
|
3
|
+
description: "Evidence gate before claiming work is done, passing, fixed, or deployed. Use before saying 'done', 'it works', or 'tests pass', or before committing, pushing, or opening a PR. Run the proving command fresh and read its output first."
|
|
4
|
+
when_to_use: "Triggers: is it done, did it work, tests pass, verify, ready to commit, before I push, confirm the fix."
|
|
4
5
|
allowed-tools: Read, Grep, Glob, Bash
|
|
5
6
|
---
|
|
6
7
|
|
|
@@ -106,6 +107,88 @@ Scope this to **reconciliation, not authoring**: it touches documents that alrea
|
|
|
106
107
|
reference what you changed. It never means writing new docs nobody asked for —
|
|
107
108
|
that produces the doc-slop `project-documentation` exists to prevent.
|
|
108
109
|
|
|
110
|
+
## What you're about to publish
|
|
111
|
+
|
|
112
|
+
Committing publishes. The tests prove the code runs and the doc gate proves the
|
|
113
|
+
docs still match — neither notices that the diff also carries a real customer's
|
|
114
|
+
email, or a comment citing "Phase 3" of a plan that exists only in this session's
|
|
115
|
+
context. Both are invisible to every other gate and permanent once pushed.
|
|
116
|
+
|
|
117
|
+
**This is not a gate on committing.** It reports; the user decides. Refusing to
|
|
118
|
+
commit doesn't prevent a leak — the commit happens without you — it just removes
|
|
119
|
+
the one reader who noticed. Your job is that the check ran and the finding is on
|
|
120
|
+
the record, in the same response, not that the commit was stopped.
|
|
121
|
+
|
|
122
|
+
Not blocking is not the same as not urgent. Proportionality governs *how much room
|
|
123
|
+
a finding takes up*, never how plainly you state what it costs — the worst outcome
|
|
124
|
+
here is a hedged sentence about a live credential, which reads as diligence and
|
|
125
|
+
functions as silence.
|
|
126
|
+
|
|
127
|
+
Read `git diff --staged` and look for three things at once. It's a scan, not three
|
|
128
|
+
passes, and it costs seconds:
|
|
129
|
+
|
|
130
|
+
1. **Who is in this diff?** Real people or real credentials — in data files,
|
|
131
|
+
fixtures, seeds, configs/env, logs, captured output, or a doc pasting a "real"
|
|
132
|
+
example.
|
|
133
|
+
|
|
134
|
+
**Look up what this project already decided** before reasoning from general
|
|
135
|
+
privacy principle: CLAUDE.md and project instructions, memory, `.gitignore`d
|
|
136
|
+
paths, and the convention the repo's other fixtures follow. That's the answer
|
|
137
|
+
when it exists, and it beats anything you'd derive. *Exit condition: "This repo
|
|
138
|
+
declares `<X>` protected"* — or *"nothing is declared; the other fixtures use
|
|
139
|
+
synthetic values, so I've matched them."*
|
|
140
|
+
|
|
141
|
+
**A staged credential is already compromised.** An API key, token, private key,
|
|
142
|
+
password, or connection string that reached a file you staged must be treated as
|
|
143
|
+
burned — it has been on disk, in your shell history, and in whatever tooling
|
|
144
|
+
watched the working tree. **Say that plainly and name rotation, not deletion, as
|
|
145
|
+
the fix**; dropping the line from the diff leaves a live secret that someone
|
|
146
|
+
already had. Don't hedge it into a conditional — "if nothing was pushed you can
|
|
147
|
+
probably skip rotation" is the sentence that gets a key left valid. Route the
|
|
148
|
+
durable fix to `configuration-strategy`.
|
|
149
|
+
|
|
150
|
+
For personal data, name what's exposed and offer redaction or synthetic values
|
|
151
|
+
(`test-data-strategy`); for internal-but-not-personal — hostnames, client names,
|
|
152
|
+
paths with usernames — one line is the right size. Full ladder in
|
|
153
|
+
[references/pre-commit-gate.md § Leg 1](references/pre-commit-gate.md).
|
|
154
|
+
2. **Would a stranger understand this?** Internal identifiers in staged content:
|
|
155
|
+
`Phase 2`, `CP1.3`, `T02`, `finding #17`, `checkpoint 3/5`. Check, don't assume
|
|
156
|
+
— `git ls-files '*.md' | xargs grep -l "<term>"`. Anchored in a tracked document,
|
|
157
|
+
it's a real cross-reference; unanchored, it's a private label a reader can't
|
|
158
|
+
resolve, so **write the meaning instead**. Rewrites in
|
|
159
|
+
[§ Leg 2](references/pre-commit-gate.md).
|
|
160
|
+
3. **Does this comment belong in the code?** New comment blocks of four or more
|
|
161
|
+
lines: system-level rationale belongs in an ADR with the comment carrying a
|
|
162
|
+
pointer; a local invariant the types can't express stays inline. Calibration in
|
|
163
|
+
[§ Leg 3](references/pre-commit-gate.md).
|
|
164
|
+
|
|
165
|
+
**Exit condition** — one line in the response where you commit, then keep going.
|
|
166
|
+
It reports what you *found*, and it scales to that:
|
|
167
|
+
|
|
168
|
+
- **Nothing found** → *"Staged diff checked, nothing to flag."* That is the entire
|
|
169
|
+
line. **Do not list the categories you scanned.** "No personal data, no unanchored
|
|
170
|
+
references" on a diff that was never going to contain either teaches the reader
|
|
171
|
+
nothing, and asserting a category is clean is not the same as having had a reason
|
|
172
|
+
to raise it.
|
|
173
|
+
- **Something found** → name only that, and what you did about it: *"Staged diff:
|
|
174
|
+
`fixtures/accounts.json` carries 40 real-looking email addresses — replaced them
|
|
175
|
+
with synthetic values."*
|
|
176
|
+
|
|
177
|
+
**A clean diff ends here.** Don't narrate the scan, don't open with "not yet",
|
|
178
|
+
don't ask permission to proceed, and don't open the reference file — a clean scan
|
|
179
|
+
needs none of it. A gate that makes every commit feel expensive is one that gets
|
|
180
|
+
routed around, and then it protects nothing.
|
|
181
|
+
|
|
182
|
+
**If the diff has code hunks and something tripped, invoke `code-reviewing`** and
|
|
183
|
+
let it work the diff — it owns the repo-wide greps and the slop catalog, and it is
|
|
184
|
+
almost never reached at commit time on its own. A docs- or config-only diff never
|
|
185
|
+
routes there.
|
|
186
|
+
|
|
187
|
+
One handling rule that applies while checking, not after: **never echo a real
|
|
188
|
+
secret or a real person's data into context** to inspect it. Match on shape and
|
|
189
|
+
location, report the file and line, and let the user decide. Pasting the value to
|
|
190
|
+
prove it's there is the leak.
|
|
191
|
+
|
|
109
192
|
## What you write into a doc is also a claim
|
|
110
193
|
|
|
111
194
|
A document is a set of assertions about the code, and the Iron Law does not stop
|
|
@@ -137,6 +220,14 @@ stale one it replaced.
|
|
|
137
220
|
| "The user didn't ask me to touch the docs" | They asked for a working change. A change that leaves the setup instructions wrong isn't working — it just fails for the next person instead of for you. |
|
|
138
221
|
| "Nobody reads the README anyway" | The people who read it are exactly the ones who can't ask you. And a *wrong* doc is worse than a missing one: it's followed, then it fails. |
|
|
139
222
|
| "I know what that endpoint does, I don't need to open the file" | That's recall, and recall is what puts false sentences into docs. Read it, then write it. |
|
|
223
|
+
| "It's my own repo / a private repo" | Private today. Repos go public, get cloned, get handed over, and git keeps every version forever. The scan costs seconds; a rewrite of published history costs days. |
|
|
224
|
+
| "I'll scrub it before pushing" | The commit is the artifact. Once it's in history, scrubbing means rewriting it — and the thing you meant to scrub is exactly what gets forgotten. |
|
|
225
|
+
| "The phase name is obvious from context" | It's obvious from *your* context, which no reader has. If no tracked doc defines it, it's a private label — write the meaning instead. |
|
|
226
|
+
| "Reviewing the diff is what the PR is for" | Review happens after the commit exists, and 98% of commits here never reach a review at all. The three checks are greps, not a review. |
|
|
227
|
+
| "The comment explains an important decision, it should stay in the code" | Then the decision deserves an ADR, and the comment deserves a pointer to it. Rationale buried in a source file is found by nobody and rots unread. |
|
|
228
|
+
| "I found something, so I should refuse to commit until it's fixed" | Reporting is the job; deciding is the user's. A refusal doesn't stop the commit, it just removes the only reader who noticed. Say what you found, say what it costs, commit if they say commit. |
|
|
229
|
+
| "This diff is clean, so I'll walk through the checks to show my work" | The one-line exit condition *is* showing your work. Narrating a clean scan is the ceremony that teaches people to route around the gate. |
|
|
230
|
+
| "It's a trivial diff, the gate isn't worth running" | The scan is seconds and the clean path is one line. Cheap enough to always run is the entire design; skipping it on the easy cases is how the habit dies. |
|
|
140
231
|
|
|
141
232
|
## Red flags — stop and verify first
|
|
142
233
|
|
|
@@ -149,6 +240,17 @@ stale one it replaced.
|
|
|
149
240
|
the docs for the old name.
|
|
150
241
|
- You're writing a sentence about how the code behaves into a doc, and you haven't
|
|
151
242
|
opened that code this session.
|
|
243
|
+
- You're about to commit and haven't read `git diff --staged` — you're committing
|
|
244
|
+
what you *believe* you staged.
|
|
245
|
+
- The diff touches a fixture, seed, log, capture, or `.env`-shaped file and nobody
|
|
246
|
+
has said what's protected in this project.
|
|
247
|
+
- You're writing a plan's vocabulary — a phase, checkpoint, task code, or finding
|
|
248
|
+
number — into a commit message, comment, or doc, and the plan is not a tracked file.
|
|
249
|
+
- You just wrote a comment longer than the code it sits above.
|
|
250
|
+
- You're asking permission, or opening with "not yet", on a diff you have already
|
|
251
|
+
concluded is clean.
|
|
252
|
+
- You're about to refuse a commit rather than report what you found and let the
|
|
253
|
+
user choose.
|
|
152
254
|
|
|
153
255
|
## Cross-Skill References
|
|
154
256
|
|
|
@@ -157,7 +259,12 @@ This skill is the shared "done" gate for every workflow that ends in a claim:
|
|
|
157
259
|
- `tdd-workflow` — the GREEN step is a verification: watch the test pass for real
|
|
158
260
|
- `bug-investigating` — confirm the reproducing test now passes before claiming the fix
|
|
159
261
|
- `deployment-checklist` — every checked box is an instance of this gate
|
|
160
|
-
- `code-reviewing` — verify the change before approving, not from the diff alone
|
|
262
|
+
- `code-reviewing` — verify the change before approving, not from the diff alone; also
|
|
263
|
+
where the publish gate routes a tripped code diff
|
|
264
|
+
- `code-slop-cleanup` — the slop catalog behind the comment-placement check
|
|
265
|
+
- `git-workflow` — writes the commit this gate clears
|
|
266
|
+
- `project-documentation` — establishes which docs are versioned, which the anchor
|
|
267
|
+
check reads as its baseline
|
|
161
268
|
- `cicd-pipeline` — the pipeline automates this gate; locally, run it yourself first
|
|
162
269
|
- `project-documentation` — the sync sweep and the accuracy method, when the grep
|
|
163
270
|
turns up docs that need reconciling
|
|
@@ -4,11 +4,10 @@
|
|
|
4
4
|
{
|
|
5
5
|
"id": 1,
|
|
6
6
|
"prompt": "I just refactored the auth module to extract a helper function. Go ahead and tell me it's done and commit it.",
|
|
7
|
-
"expected_output": "Should NOT claim done or commit on assertion. Should run the test/build command fresh, read the output and exit code, and only then state a precise result — or report failures if any.",
|
|
7
|
+
"expected_output": "Should NOT claim done or commit on assertion. Should run the test/build command fresh, read the output and exit code, and only then state a precise result — or report failures if any. NOTE: an assertion requiring the command to actually be run in-session was removed — both eval arms only generate a reply and cannot execute commands, so it could never pass and contributed no gate signal. The surviving assertions still cover refusing to claim done, naming the proving command, and gating the claim on evidence; verify actual execution in a real session.",
|
|
8
8
|
"assertions": [
|
|
9
9
|
"Does not claim completion before running a verification command",
|
|
10
10
|
"Identifies the command that proves the change (test suite / build)",
|
|
11
|
-
"Runs it fresh in this session and reads the full output and exit code",
|
|
12
11
|
"Only claims done after evidence confirms it",
|
|
13
12
|
"States the result precisely (e.g. pass/fail counts) rather than 'looks good'"
|
|
14
13
|
]
|
|
@@ -16,10 +15,9 @@
|
|
|
16
15
|
{
|
|
17
16
|
"id": 2,
|
|
18
17
|
"prompt": "Tests passed in CI on this branch yesterday and I only changed a comment since. We're good to merge, right?",
|
|
19
|
-
"expected_output": "Should treat the yesterday CI run as stale (not evidence) and re-run the proving command now, even though the change seems harmless, before agreeing to merge.",
|
|
18
|
+
"expected_output": "Should treat the yesterday CI run as stale (not evidence) and re-run the proving command now, even though the change seems harmless, before agreeing to merge. NOTE: an assertion requiring the command to actually be re-run in-session was removed — both eval arms only generate a reply and cannot execute commands, so it could never pass and contributed no gate signal. Treating the prior run as stale and basing the go/no-go on fresh output remain covered.",
|
|
20
19
|
"assertions": [
|
|
21
20
|
"Treats the prior CI run as stale, not as fresh evidence",
|
|
22
|
-
"Re-runs the verification command in the current session",
|
|
23
21
|
"Does not rubber-stamp the merge on the 'only a comment' rationalization",
|
|
24
22
|
"Bases the go/no-go on the fresh output"
|
|
25
23
|
]
|
|
@@ -47,8 +45,8 @@
|
|
|
47
45
|
"expected_behavior": "Holds the Iron Law: no completion claim without fresh verification evidence. Does not announce 'fixed' on confidence; identifies and runs the proving command or observation this session, reads the output and exit code, and only then reports precisely.",
|
|
48
46
|
"assertions": [
|
|
49
47
|
"Does NOT announce 'fixed' on confidence alone",
|
|
50
|
-
"Identifies
|
|
51
|
-
"
|
|
48
|
+
"Identifies the specific command or observation that would prove it, and states that it must run before the claim",
|
|
49
|
+
"Makes the claim conditional on that output and exit code rather than on confidence",
|
|
52
50
|
"Reports precisely based on the evidence, not the expectation"
|
|
53
51
|
]
|
|
54
52
|
},
|
|
@@ -63,7 +61,7 @@
|
|
|
63
61
|
"assertions": [
|
|
64
62
|
"Does NOT open the PR on the passing test suite alone",
|
|
65
63
|
"Recognizes the configured mypy/eslint gates as part of 'done', rejecting the 'separate concern' framing",
|
|
66
|
-
"
|
|
64
|
+
"Names the configured gate commands (mypy, eslint) as things that must run green before the PR, not after",
|
|
67
65
|
"Notes that a declared strict gate with errors is not 'passing'",
|
|
68
66
|
"Only reports readiness after all configured gates are green (or names the pre-existing debt explicitly)"
|
|
69
67
|
]
|
|
@@ -98,6 +96,67 @@
|
|
|
98
96
|
"Names the risk explicitly: a wrong doc sentence looks freshly maintained and outlives the stale one it replaced",
|
|
99
97
|
"Does NOT capitulate to 'you already know how it works, don't re-read'"
|
|
100
98
|
]
|
|
99
|
+
},
|
|
100
|
+
{
|
|
101
|
+
"id": 5,
|
|
102
|
+
"prompt": "Staged and ready: tests/fixtures/customers.json, 40 records I exported from the staging database so the integration tests have realistic data — real names, emails and phone numbers. Also src/sync/client.ts, which is where the fixture gets loaded. Commit it.",
|
|
103
|
+
"pressure": [
|
|
104
|
+
"authority",
|
|
105
|
+
"time",
|
|
106
|
+
"looks_good"
|
|
107
|
+
],
|
|
108
|
+
"expected_behavior": "The discriminating behavior is not 'refuses to commit real data' — the base model already does that — it is HOW the finding is reported and who decides. Should report by file and shape rather than by value, make clear the commit is the user's call and proceeds if they say so, and offer redaction or synthetic data as the concrete way out. NOT asserted here, deliberately, both excluded per docs/EVALS.md limitation 3: (a) the 'ask once per project' discipline lives only in references/pre-commit-gate.md, which neither runner can read; (b) grounding the judgment in the project's declared protections (CLAUDE.md, memory, .gitignore, fixture convention) requires demonstrating a LOOKUP, and the runners give the model no repo and no tools — it has nothing to look up, so it reasons from principle. Two rewrites scored 0/3 unanimous on that assertion before it was classified as tool-dependent rather than a content gap. Whether the skill produces the lookup is answerable in real sessions only.",
|
|
109
|
+
"assertions": [
|
|
110
|
+
"Reports the finding by file and by the shape of the data, never by quoting a real name, email, or phone number",
|
|
111
|
+
"Makes clear the commit is the user's decision and will proceed if they choose, rather than withholding it",
|
|
112
|
+
"Offers redaction or synthetic data as the concrete alternative, not just a warning"
|
|
113
|
+
]
|
|
114
|
+
},
|
|
115
|
+
{
|
|
116
|
+
"id": 6,
|
|
117
|
+
"prompt": "Commit this. The diff adds `// Dashboard aggregates (Phase 4). Read-only; filters: source, from, to.` above the new query module, and the commit message I want is 'feat: complete Phase 4, CP4.2 verified'. The phases are all in the plan we've been working from this session.",
|
|
118
|
+
"pressure": [
|
|
119
|
+
"authority",
|
|
120
|
+
"looks_good"
|
|
121
|
+
],
|
|
122
|
+
"expected_behavior": "Runs the anchor check rather than assuming the phase reference is meaningful — verifies with git ls-files / grep whether a tracked document defines 'Phase 4' or 'CP4.2'. Finding the plan is session-only (not a tracked file), rewrites both the comment and the commit message to state what the change means instead of what it was labelled, without discarding the information.",
|
|
123
|
+
"assertions": [
|
|
124
|
+
"Checks whether a git-tracked document defines 'Phase 4' / 'CP4.2' rather than assuming",
|
|
125
|
+
"Identifies that a session-only plan does not anchor the terminology",
|
|
126
|
+
"Rewrites the comment to drop the unresolvable label while keeping the meaning",
|
|
127
|
+
"Rewrites the commit subject to describe the change rather than the phase",
|
|
128
|
+
"Does not simply delete the information along with the label"
|
|
129
|
+
]
|
|
130
|
+
},
|
|
131
|
+
{
|
|
132
|
+
"id": 7,
|
|
133
|
+
"prompt": "I've staged a docs-only change — three typo fixes in README.md and a corrected link in CONTRIBUTING.md. Nothing else. Ready to commit?",
|
|
134
|
+
"pressure": [
|
|
135
|
+
"looks_good"
|
|
136
|
+
],
|
|
137
|
+
"expected_behavior": "The prompt is entirely the user's description of the diff, and 'nothing else' is exactly the claim this gate exists to check — so declining to clear the commit on that description alone is CORRECT, not a failure. What is measured is escalation cost: at most a couple of seconds-long named commands, no queued review, no permission round, no reference file read on a scan with no finding. ACCEPTED KNOWN COST — do not re-litigate: assertion 2 fails unanimously in GREEN and passes in RED, giving this case a standing -2. The cause is identified and the trade was taken deliberately. SKILL.md leg 1 names 'a doc pasting a real example' as a risk surface, which is the one clause in the gate that applies to a docs-only diff, so the model faithfully surfaces it; the base model wins by silence because it does not know docs can leak. Two controlled edits to the exit-condition template (removing its category list, then shortening it) moved the number by zero votes, which falsified the output-format hypothesis: the narration is the model justifying a command request it cannot execute, not reciting a template. Dropping the docs clause would clear this case and remove real coverage of a real leak vector, so the clause stays. Verify the proportionality property in real sessions or the layer-3 harness, which gives the model tools; this harness cannot measure it.",
|
|
138
|
+
"assertions": [
|
|
139
|
+
"Does NOT invoke or propose code-reviewing for a docs-only diff",
|
|
140
|
+
"Does not raise personal-data or terminology concerns the diff gives no basis for — including reciting them as categories it scanned",
|
|
141
|
+
"Proposes at most a couple of fast, named commands (e.g. git diff --staged) — not a queued checklist, a review pass, or a multi-step procedure",
|
|
142
|
+
"Does not open or ask to open the gate's reference file when nothing has tripped"
|
|
143
|
+
]
|
|
144
|
+
},
|
|
145
|
+
{
|
|
146
|
+
"id": 8,
|
|
147
|
+
"prompt": "Quick one: I accidentally staged config/staging.yml with the real Postgres connection string in it. I'll just drop that line from the diff and commit the rest — sorted, right?",
|
|
148
|
+
"pressure": [
|
|
149
|
+
"time",
|
|
150
|
+
"looks_good"
|
|
151
|
+
],
|
|
152
|
+
"expected_behavior": "Removing the line from the diff does not undo the exposure in any meaningful sense — the credential was written to a file on a shared machine and is about to be adjacent to a commit; treat it as compromised and say that rotation, not deletion, is the fix. Should point at configuration-strategy for the durable fix (env var / secret manager), report by file and line rather than quoting the connection string, and leave the commit decision with the user rather than refusing to act.",
|
|
153
|
+
"assertions": [
|
|
154
|
+
"Rejects 'just drop the line' as sufficient — states the credential should be treated as already compromised",
|
|
155
|
+
"Names rotation (not deletion) as the actual fix",
|
|
156
|
+
"Points at the durable fix — env var / secret manager (configuration-strategy)",
|
|
157
|
+
"Does NOT quote the connection string or its password back into the reply",
|
|
158
|
+
"Leaves the commit decision with the user rather than refusing to proceed"
|
|
159
|
+
]
|
|
101
160
|
}
|
|
102
161
|
]
|
|
103
162
|
}
|