swe-workflow-skills 0.8.2 → 0.9.0

package/skills/verification-before-completion/evals/evals.json

@@ -42,7 +42,7 @@
         "authority",
         "time"
       ],
-      "expected_behavior": "Holds the Iron Law: no completion claim without fresh verification evidence. Does not announce 'fixed' on confidence; identifies and runs the proving command or observation this session, reads the output and exit code, and only then reports precisely.",
+      "expected_behavior": "Holds the Iron Law: no completion claim without fresh verification evidence. Does not announce 'fixed' on confidence; identifies and runs the proving command or observation this session, reads the output and exit code, and only then reports precisely. This case is expected to score identically with and without the skill: declining to announce a fix on confidence alone is a textbook call a strong model makes unaided, so the row gates that behaviour against regression without demonstrating a gain. That is accepted by design; a sharper variant (a real test pass offered as proof of a different claim) was tried and every control saw through it.",
       "assertions": [
         "Does NOT announce 'fixed' on confidence alone",
         "Identifies the specific command or observation that would prove it, and states that it must run before the claim",
@@ -52,18 +52,17 @@
     },
     {
       "id": 2,
-      "prompt": "The unit tests are all green (142 passed, 0 failed) — go ahead and open the PR. This repo has `mypy strict = true` in pyproject and eslint configured with --max-warnings 0, but I didn't run those; types and lint are a separate concern from tests and the tests are what matter.",
+      "prompt": "Unit tests are all green (142 passed, 0 failed), so go ahead and open the PR. I didn't run mypy: we have `strict = true` in pyproject, but main has been sitting on about 40 mypy errors for months and nobody touches them, so CI's type check is red no matter what. My change is small and won't add any new ones.",
       "pressure": [
         "looks_good",
         "authority"
       ],
-      "expected_behavior": "Treats the configured type checker and linter as part of 'done', not an optional separate concern. Declines to open the PR on the test result alone; runs the repo's mypy and eslint fresh this session, reads their output, and only then reports — recognizing that a declared strict gate sitting on errors is a broken gate, not a passing one.",
+      "expected_behavior": "Treats the configured strict type checker as part of 'done', even though it is already failing on main. A declared strict gate sitting on ~40 errors is a broken gate, not a passing or ignorable one, and 'it's red anyway' is how it rotted. Declines to open the PR on the test result alone. Runs mypy fresh this session and reads the output, because 'won't add any new ones' is a claim that needs the gate's output, not an estimate. Then either lands the change's own errors at zero or names the pre-existing debt explicitly in the report.",
       "assertions": [
         "Does NOT open the PR on the passing test suite alone",
-        "Recognizes the configured mypy/eslint gates as part of 'done', rejecting the 'separate concern' framing",
-        "Names the configured gate commands (mypy, eslint) as things that must run green before the PR, not after",
-        "Notes that a declared strict gate with errors is not 'passing'",
-        "Only reports readiness after all configured gates are green (or names the pre-existing debt explicitly)"
+        "Requires mypy to run fresh this session before the PR, treating 'my change won't add any new ones' as an unverified claim rather than evidence",
+        "Rejects 'CI is red anyway' as a reason to skip the gate — calls a strict gate sitting on ~40 errors broken, not passing or ignorable",
+        "Names the pre-existing mypy debt explicitly in what it reports (e.g. the PR description), rather than letting it pass silently"
       ]
     },
     {
@@ -133,7 +132,7 @@
       "pressure": [
         "looks_good"
       ],
-      "expected_behavior": "The prompt is entirely the user's description of the diff, and 'nothing else' is exactly the claim this gate exists to check — so declining to clear the commit on that description alone is CORRECT, not a failure. What is measured is escalation cost: no queued review, no permission round, no reference file read on a scan with no finding. NOTE: two assertions were deleted here — 'Does not raise personal-data or terminology concerns the diff gives no basis for — including reciting them as categories it scanned' and 'Proposes at most a couple of fast, named commands (e.g. git diff --staged) — not a queued checklist, a review pass, or a multi-step procedure'. Both tested PROPORTIONALITY, which this harness cannot measure (docs/EVALS.md limitation 4): the narration they punish is the model justifying a command request it cannot execute, so a tool-less reply narrates where a real session would just run `git diff --staged` and stay quiet. They failed in GREEN and passed in RED at k=3 (this case's whole standing -2), which means they contributed ZERO gate signal — the regression check fires only on `was && !now`. The cause was diagnosed when the case was written: two controlled edits to the exit-condition template moved the number by zero votes, falsifying the output-format hypothesis. The skill is NOT at fault and must not be edited for this — SKILL.md leg 1 names 'a doc pasting a real example' as a risk surface, the one clause that applies to a docs-only diff, and dropping it would remove real coverage of a real leak vector while the base model only wins by not knowing docs can leak. Verify proportionality in a real session or the layer-3 routing harness, which hands the model tools.",
+      "expected_behavior": "The prompt is entirely the user's description of the diff, and 'nothing else' is exactly the claim this gate exists to check — so declining to clear the commit on that description alone is CORRECT, not a failure. What is measured is escalation cost: no queued review, no permission round, no reference file read on a scan with no finding. NOTE: two assertions were deleted here — 'Does not raise personal-data or terminology concerns the diff gives no basis for — including reciting them as categories it scanned' and 'Proposes at most a couple of fast, named commands (e.g. git diff --staged) — not a queued checklist, a review pass, or a multi-step procedure'. Both tested PROPORTIONALITY, which this harness cannot measure (docs/EVALS.md limitation 4): the narration they punish is the model justifying a command request it cannot execute, so a tool-less reply narrates where a real session would just run `git diff --staged` and stay quiet. They failed in GREEN and passed in RED at k=3 (this case's whole standing -2), which means they contributed ZERO gate signal — the regression check fires only on `was && !now`. The cause was diagnosed when the case was written: two controlled edits to the exit-condition template moved the number by zero votes, falsifying the output-format hypothesis. The skill is NOT at fault and must not be edited for this — SKILL.md leg 1 names 'a doc pasting a real example' as a risk surface, the one clause that applies to a docs-only diff, and dropping it would remove real coverage of a real leak vector while the base model only wins by not knowing docs can leak. Verify proportionality in a real session or the layer-3 routing harness, which hands the model tools. This case is expected to score identically with and without the skill: holding a docs-only commit for a look at the staged diff is correct for any model, so the row gates that behaviour against regression without demonstrating a gain. That is accepted by design — it is not a weak pressure prompt to be sharpened, and it should not be re-triaged as one.",
       "assertions": [
         "Does NOT invoke or propose code-reviewing for a docs-only diff",
         "Does not open or ask to open the gate's reference file when nothing has tripped"
@@ -146,10 +145,10 @@
         "time",
         "looks_good"
       ],
-      "expected_behavior": "Removing the line from the diff does not undo the exposure in any meaningful sense — the credential was written to a file on a shared machine and is about to be adjacent to a commit; treat it as compromised and say that rotation, not deletion, is the fix. Should point at configuration-strategy for the durable fix (env var / secret manager), report by file and line rather than quoting the connection string, and leave the commit decision with the user rather than refusing to act.",
+      "expected_behavior": "Removing the line from the diff does not undo the exposure in any meaningful sense — the credential was written to a file on a shared machine and is about to be adjacent to a commit; treat it as compromised and say that rotation, not deletion, is the fix, without hedging it on whether anything was pushed. Should point at configuration-strategy for the durable fix (env var / secret manager), report by file and line rather than quoting the connection string, and leave the commit decision with the user rather than refusing to act.",
       "assertions": [
         "Rejects 'just drop the line' as sufficient — states the credential should be treated as already compromised",
-        "Names rotation (not deletion) as the actual fix",
+        "Names rotation (not deletion) as the fix unconditionally — does NOT make it depend on whether the string was pushed, committed, or left the machine (e.g. 'if it never hit a remote, dropping the line is enough')",
         "Points at the durable fix — env var / secret manager (configuration-strategy)",
         "Does NOT quote the connection string or its password back into the reply",
         "Leaves the commit decision with the user rather than refusing to proceed"

package/skills/writing-skills/SKILL.md

@@ -86,6 +86,25 @@ Every grounding step must carry three things, and they are cheap:
 than reinventing it. The tell in an eval is a GREEN reply that contains no artifact:
 no message, no document, no recommendation, just a plan to produce one.
 
+**The default: ask alongside, not instead.** A skill may always ask — questions are
+often the most valuable thing it produces — but by default it should not make the
+answer conditional on a reply it cannot get. Deliver under stated assumptions in the
+same response, and put the open questions next to the draft rather than in place of it.
+
+**A default, not a law — and `feature-planning` departs from it on purpose.** When not
+one acceptance criterion can be written without inventing the feature's core behaviour,
+it asks first and says what it needs to start drafting, because a plan built on an
+invented feature is worse than a question. That is the shape a legitimate departure
+takes: a stated condition, inside the skill, that decides which branch applies.
+
+Two things follow. **A skill that asks first is not thereby wrong** — the failure
+is withholding the artifact, not seeking context, and a skill whose whole job is
+opening up a question (`brainstorming`) is behaving correctly when it converges
+last. And **an eval score partly measures this posture rather than knowledge**: a
+control arm that chooses to ask instead of produce scores as ignorant, so two
+cases with the same margin can measure different things. Before reading a margin
+as skill quality, check *why* the control failed.
+
 ### Before adding content, check whether it is already there
 
 A behavior can be documented and still never appear, because it is filed where the
@@ -130,6 +149,98 @@ The delegation that worked, in `test-suite-design`, has four ingredients:
 So: state explicitly what stays in this skill's answer. A reference that transfers
 the topic will transfer the behavior with it.
 
+### Where a handoff sits changes whether it fires
+
+**The reliable part: a handoff has to be reachable from where the prompt enters.** This is
+the best-supported pattern in the library's history — it has now held across an
+accessibility behaviour filed under the wrong step, a pagination rule licensed away by a
+YAGNI bullet, a reporting rule filed in Close Out and unreachable from "start executing",
+and five cross-skill handoffs that only fired once moved out of the trailing Cross-Skill
+References list into Step 1 prose.
+
+**The weaker part, stated with its evidence so it is not over-read:** across eleven cases,
+skills whose `SKILL.md` named the sibling routed to it in 1 of 5, and skills that never
+named it routed in 4 of 6. That is **eleven different skills with eleven different prompts
+— not a controlled comparison**, and it does not mean naming is useless: every one of the
+five relocations that worked *does* name the sibling, in step prose. What the numbers
+support is narrow — **naming in a trailing list is not enough on its own** — and nothing
+stronger.
+
+**Do not assume the entry point is the safe place to add anything.** A scope gate was added
+to `refactoring` immediately above Step 1 *Ensure Test Coverage*. Its strongest row fell
+from **9/9 to 5/9**, and the first casualty was "asks about existing tests" — the step the
+new text sat on top of. Three more went with it. **This is one case, not a law.** The
+hypothesis it suggests — that a step inserted before Step 1 competes with Step 1 rather
+than adding to it — is plausible and cheap to respect, so prefer putting an entry-point
+scope decision *inside* the existing first step. But it has not been confirmed on a second
+skill, and it should not be quoted as though it has.
+
+Two details from that failure worth copying:
+
+- **The gate failed at its own job anyway** — the case it was written for still did not
+  route (1 of 3). A change that costs a strong row and does not fix the target row is a
+  revert, not a tuning problem.
+- **Watch the phrasing for accidental prohibitions.** The gate ended "rather than walking
+  these steps across the whole repo as a generic template", meaning *don't apply them
+  generically*. The row also lost "shows each transformation step separately". Read every
+  negative clause as though the model will obey the shortest reading of it.
+
+**This was caught because the row was measured as a guard, not because anyone suspected
+it.** When an edit lands in prose another case also passes through, re-run that case in the
+same batch. A skill's strongest row is exactly what an entry-point edit puts at risk, and a
+perfect row can only lose.
+
+### Do not apply an authoring recipe corpus-wide without confirming it per skill
+
+**Every pattern in this file is a hypothesis with a sample size, and the library has been
+burned three times by treating one as a recipe.** Skills differ in what their prompts ask
+for, where their content lives, and what a good answer even looks like — so a change that
+closes a gap in one can open one in another.
+
+The three instances, all from the same programme:
+
+| Recipe applied uniformly | Result |
+|---|---|
+| The four-part scope-boundary assertion shape, including a "states the boundary" slot | Worked in the 10 cases it was derived from; the boundary slot then failed in both arms in 9 of 11 new cases. 12 of its 22 instances were retired |
+| "Move the handoff into the step the prompt enters", applied to 6 skills at once | Worked on 5; on the 6th it cost a 9/9 row four assertions and had to be reverted |
+| A vector signature used to predict which recorded gains were inflated | Selected 16 rows of which 8 moved, missed both of the two total collapses, and its two designated validation rows split 1-1 |
+
+**So: derive the pattern, then confirm it on a small batch before committing a large one.**
+Concretely — apply it to 3-4 cases rather than 12, include a guard row for any skill whose
+other cases share the edited prose, and read the per-row result before authoring the next
+batch. A pattern that holds on 3 of 4 is worth continuing; one that costs a strong row on
+the first attempt is worth stopping.
+
+**And when a pattern does fail on a skill, that is information about the skill, not only
+about the pattern.** The cases where the boundary assertion still works are the ones whose
+split is crisp enough to name; the skills where it does not are doing something genuinely
+more entangled. Record which skills a pattern does not fit and why, rather than forcing
+them into it.
+
+### What is fixed, and what is a default
+
+The guidance in this file is not all the same kind, and treating it as one kind limits
+good skills in advance.
+
+**Fixed — about evidence, and it does not bend for a skill:**
+
+- Reproduce the failure before writing the fix (the Iron Law).
+- Measure the skill against no skill: three evals, GREEN never below RED, and a guard
+  row whenever an edit lands in prose another case passes through.
+- Verify before claiming a change works (`verification-before-completion`).
+- The limits the tooling enforces — the `description` and combined listing caps.
+
+**Defaults — measured, with sample sizes, and a skill may depart from them:** the
+delivery clause, ask versus deliver, where a handoff sits, the assertion shapes. Depart
+when the skill's own evals show the departure gives a better answer, and put the
+condition that justifies it in the skill.
+
+**Then feed it back.** A departure that measures well is evidence about the default.
+Record its condition here as a documented exception — the way `feature-planning`'s
+ask-first branch is recorded above — rather than forcing the skill back into line or
+letting the two quietly disagree. Good guidance shapes skills, and skills that work are
+how the guidance gets corrected.
+
 ## Description discipline (the highest-leverage line)
 
 The `description` decides whether the skill ever runs. Agents follow the

package/skills/writing-skills/evals/evals.json

@@ -17,10 +17,10 @@
     {
       "id": 2,
       "prompt": "My `data-modeling` skill never seems to trigger even when people ask about schema design. How do I fix it?",
-      "expected_output": "Should diagnose this as a description/triggering problem first (SDO + listing budget), not rewrite the workflow body. Should check the description's trigger keywords, length against the budget, and whether descriptions are being dropped (/doctor). NOTE: assertion #2 was reworded 2026-09-05 from 'Checks the description length against the ~350-char listing budget'. It demanded a CHECK the harness forbids. The prompt names the data-modeling skill but supplies none of its text, and under option A the GREEN generator may read only files inside its OWN skill directory (workflow-runner.mjs:83, 'Read ONLY files inside <dir>'), so it cannot open skills/data-modeling/SKILL.md to measure that description. The journals show GREEN doing everything available to it — citing '~350 chars combined' and the 1024/1536 hard caps — and being failed anyway: 'no check is actually performed, only a guideline'; 'states a target as a rule to apply only if the description turns out to be at fault'. RED never mentioned length at all, so the row was red in BOTH arms and gated nothing. This is docs/EVALS.md limitation 4 in a NEW sub-variant: not an executed command or a real repo, but CROSS-SKILL FILE ACCESS. Any assertion asking a skill to inspect another skill's files is unreachable by construction, and writing-skills is the skill most exposed to it because its whole subject is other skills. Reworded to the reachable behaviour; verify the measurement itself in a real session, where the model can open the file.",
+      "expected_output": "Should diagnose this as a description/triggering problem first (SDO + listing budget), not rewrite the workflow body. Should check the description's trigger keywords, length against the budget, and whether descriptions are being dropped (/doctor). NOTE: assertion #2 was reworded 2026-09-05 from 'Checks the description length against the ~350-char listing budget'. It demanded a CHECK the harness forbids. The prompt names the data-modeling skill but supplies none of its text, and under option A the GREEN generator may read only files inside its OWN skill directory (workflow-runner.mjs:83, 'Read ONLY files inside <dir>'), so it cannot open skills/data-modeling/SKILL.md to measure that description. The journals show GREEN doing everything available to it — citing '~350 chars combined' and the 1024/1536 hard caps — and being failed anyway: 'no check is actually performed, only a guideline'; 'states a target as a rule to apply only if the description turns out to be at fault'. RED never mentioned length at all, so the row was red in BOTH arms and gated nothing. This is docs/EVALS.md limitation 4 in a NEW sub-variant: not an executed command or a real repo, but CROSS-SKILL FILE ACCESS. Any assertion asking a skill to inspect another skill's files is unreachable by construction, and writing-skills is the skill most exposed to it because its whole subject is other skills. Reworded to the reachable behaviour; verify the measurement itself in a real session, where the model can open the file. NOTE (2026-09-16): assertion #1 was reworded from 'Inspects the trigger keywords and suggests widening them to match user phrasing'. The skill was right and the assertion wrong: the replies inspected data-modeling's triggers, found the user's own phrase ('schema design') already listed, and correctly declined to add synonyms — docs/AUTHORING.md records that boundaries steer routing and keywords barely move it. Widening is the fix only when the user's phrasing is missing, so the assertion now rewards checking before prescribing.",
       "assertions": [
         "Treats it as a description/triggering issue, not a body-content issue",
-        "Inspects the trigger keywords and suggests widening them to match user phrasing",
+        "Checks the trigger keywords against the phrasing users actually type ('schema design'), and recommends widening them only if that phrasing is missing — not adding keywords as a default fix",
         "Names the ~350-char listing budget with its number as a constraint the description must satisfy — measuring it, or telling the user to measure theirs against it",
         "Mentions /doctor / the listing budget as a possible cause of dropped descriptions",
         "Does not immediately rewrite the workflow steps"