swe-workflow-skills 0.8.2 → 0.9.1

package/skills/llm-app-engineering/evals/evals.json

@@ -16,24 +16,26 @@
     {
       "id": 2,
       "prompt": "Our coding agent has access to 25 tools and it keeps picking the wrong one and forgetting what it already did in long multi-step tasks. How do we fix the design?",
-      "expected_output": "Diagnoses a context/tool-design problem: prunes and disambiguates the tool surface, externalizes state (plan file/scratchpad, summarization) for long horizons, considers decomposing into phases or sub-agents with scoped toolsets, and verifies the fix with task-success evals rather than anecdotes",
+      "expected_output": "Diagnoses a context/tool-design problem: prunes and disambiguates the tool surface, externalizes state (plan file/scratchpad, summarization) for long horizons, considers decomposing into phases or sub-agents with scoped toolsets, and verifies the fix with task-success evals rather than anecdotes. Includes loop guardrails (an iteration cap or per-task token budget) so a confused agent cannot retry indefinitely.",
       "assertions": [
         "Diagnoses this as a context/tool design problem (overlapping tools, unbounded history), not primarily a model-capability problem",
         "Proposes reducing or restructuring the tool surface: fewer and clearer tools, distinct names and descriptions, or grouping/namespacing",
         "Addresses long-horizon memory: a scratchpad or plan file, summarization/compaction of history, or state externalized outside the context window",
         "Considers decomposition into phases or sub-agents with scoped toolsets for distinct stages of the task",
-        "Recommends verifying the improvement with evals (task success rate on representative tasks) rather than eyeballing a few runs"
+        "Recommends verifying the improvement with evals (task success rate on representative tasks) rather than eyeballing a few runs",
+        "Adds a loop guardrail: caps the agent's iterations or its token budget per task, so a confused agent cannot keep retrying"
       ]
     },
     {
       "id": 3,
-      "prompt": "We rewrote our assistant's system prompt and the answers feel better. How do we prove the new prompt is actually better before rolling it out to users?",
-      "expected_output": "Recognizes this as a measurement question, not a prompt-design question — hands off to ai-evaluation: compare both prompt variants on the same golden dataset with pairwise/judge scoring or metric deltas, gated before rollout, rather than offering more prompt-engineering advice",
+      "prompt": "We rewrote our assistant's system prompt and the answers feel better. How do we prove it before rolling out to users? Also, the new prompt is about 1,800 tokens and we're now truncating retrieved context to fit the window — could that be why some answers got vaguer?",
+      "expected_output": "Splits the request: proving the rewrite helped is a measurement question, while a 1,800-token system prompt crowding retrieved chunks out of the window is a context-engineering problem this skill owns. Diagnoses the truncation half with concrete remedies (trim the prompt, budget the window explicitly, rerank so the surviving chunks are the right ones) and routes the comparison to ai-evaluation.",
       "assertions": [
-        "Recognizes this is a measurement/evaluation question, not a prompt-design question",
-        "Refers to the ai-evaluation skill for the methodology (golden dataset, LLM-as-judge, regression comparison)",
-        "Mentions comparing the two prompt variants on the same dataset (pairwise/side-by-side or metric deltas), not anecdotes",
-        "Does not answer with further prompt-engineering advice as the primary response"
+        "Separates the two questions: proving the new prompt is better is a measurement question, while the context-budget interaction is a prompt and retrieval design question",
+        "Refers to ai-evaluation for the comparison — both prompt variants over the same evaluation set",
+        "Frames what it hands to ai-evaluation as inputs — the prompt variants, the question set, what to hold constant — rather than specifying the golden-set size, judge rubric, or calibration protocol itself",
+        "Diagnoses the context-budget half: a 1,800-token system prompt displacing retrieved context is a design problem, with concrete remedies",
+        "States the boundary: evaluation measures whether a change helped; context engineering decides what occupies the window and what gets cut"
       ]
     }
   ]

package/skills/metrics-and-okrs/evals/evals.json

@@ -31,14 +31,13 @@
     },
     {
       "id": 3,
-      "prompt": "We defined the KR already. Here's the data: the A/B test on the new onboarding shows a 3.1% lift in activation, p = 0.06, n = 800 split across both arms. Is that a real effect? Do we ship it?",
-      "expected_output": "Scope boundary — judging whether an observed lift is real (significance, power, sample size, stopping rules) is statistical-analysis work. Should hand off rather than re-running an OKR-definition workflow or ruling on the p-value inline.",
+      "prompt": "Our activation key result is 'increase week-1 activation from 22% to 30% by Q3'. The A/B test on the new onboarding shows a 3.1% lift, p = 0.06, n = 800 split across both arms. Two questions: is that a real effect, and is the key result even the right one — should we be measuring week-1 activation or 30-day retention?",
+      "expected_output": "Splits the request: whether the observed lift is real is an inference question, while whether week-1 activation is the right key result is a metric-definition question this skill owns. Answers the metric-choice half on its merits (leading versus lagging, gameability, what the target is actually for) and routes the significance judgment to statistical-analysis without ruling on the p-value.",
       "assertions": [
-        "Recognizes this as an inference question about experiment results, not a metric-definition question",
-        "Hands off to or invokes the statistical-analysis skill",
-        "Does not declare the result significant or issue a ship/no-ship verdict from the p-value alone",
-        "Does not restart an OKR or success-metric definition exercise the user has already completed",
-        "May note how the established result would feed the existing key result once the inference is settled"
+        "Separates the two questions: whether the lift is real is an inference question, while whether week-1 activation is the right key result is a metric-definition question",
+        "Refers to statistical-analysis for the significance, power, and sample-size judgment",
+        "Does not declare the result significant or issue a ship verdict from the p-value alone",
+        "Answers the metric-definition half — whether week-1 activation or 30-day retention is the better key result here, and on what grounds"
       ]
     }
   ]

package/skills/ml-experiment-tracking/SKILL.md

@@ -30,6 +30,8 @@ Before running experiments, establish:
 
 Recommend MLflow for open-source simplicity, W&B for rich visualization needs. See [references/tracking-tools.md](references/tracking-tools.md) for setup patterns.
 
+**Training code the tracker can re-run.** A logged commit hash only means something if the commit contains the training run. A loop living in notebook cells doesn't qualify — cell execution order isn't in the commit, so two runs from "the same code" aren't comparable, which is the thing tracking exists to fix. Extract the training loop into a script or module the notebook (or the tracker) calls with parameters, and log from there. **This extraction stays in this skill** — it is what makes a run reproducible. Turning that script into a scheduled, tested production job is `notebook-to-production`'s work; naming that skill is not a reason to leave the training loop in the notebook.
+
 ### Step 2: Define What to Track
 
 For every experiment run, log:

package/skills/ml-experiment-tracking/evals/evals.json

@@ -18,14 +18,15 @@
     {
       "id": 2,
       "prompt": "I've been training models in Jupyter notebooks for 3 months and I have results scattered across 40+ notebooks. How do I get organized?",
-      "expected_output": "Should recommend setting up tracking infrastructure, then systematically re-running or cataloging key experiments. Should be practical about not re-running everything.",
+      "expected_output": "Should recommend setting up tracking infrastructure, then systematically re-running or cataloging key experiments. Should be practical about not re-running everything. NOTE: final assertion added 2026-09-18. Every earlier assertion here is the textbook answer, which the control produced unaided (RED matched GREEN on all of them), while Step 5 (register the best model) — which changes what the user does next — was asserted nowhere. Drafted from that step's own text, not from a treatment reply, and worded without an either/or clause. Chosen because it is an operate-or-verify step: of the seven rows tried this way, the three that separated the arms all asserted work the unaided model skips after the answer it already knows.",
       "assertions": [
         "Does NOT suggest re-running all 40 notebooks",
         "Recommends setting up tracking for all future experiments",
         "Suggests cataloging the best results from notebooks into the tracker manually",
         "Recommends extracting reusable training code from notebooks into scripts",
         "Suggests data versioning to prevent future dataset ambiguity",
-        "Provides a practical migration path, not an all-or-nothing approach"
+        "Provides a practical migration path, not an all-or-nothing approach",
+        "Registers a chosen model with a stage tag and a link back to the run that produced it, so a model in production traces to its data, code and parameters"
       ]
     },
     {

package/skills/ml-pipeline-design/evals/evals.json

@@ -32,12 +32,13 @@
     },
     {
       "id": 3,
-      "prompt": "I have a Jupyter notebook that pulls last month's sales from the warehouse, computes exec KPIs, and emails a report. I want it to run on a schedule instead of me running it by hand — how do I turn this notebook into production code?",
-      "expected_output": "Should recognize there is no model training involved — this is a general analysis/reporting notebook — and hand off to the notebook-to-production skill (modules, tests, parameterization, scheduling) rather than designing an ML training pipeline.",
+      "prompt": "Our churn model retrains every Sunday from a cron job. Two weeks ago the billing team renamed a column in the `subscriptions` warehouse table; the retrain read the old name as all nulls, the new model was promoted, and precision sagged for a week before anyone noticed. Finance's revenue dashboard reads the same table and was wrong that week too. How do we make sure this doesn't happen again?",
+      "expected_output": "Notices that 'make sure this doesn't happen again' covers two different guarantees. Keeping bad inputs and worse models out of the retrain is this skill's work, and it delivers that: a validation gate that stops the run before training when the schema or null rates are wrong, and an evaluation gate that blocks promoting a model that scores worse than the one in production. Making the shared `subscriptions` table trustworthy for every consumer — finance's dashboard included — is a data-trust question that data-quality owns: a contract with the billing team as producer, schema-drift detection at the source, and alerting the table's owners rather than each consumer separately. A good answer does not let a pipeline-local check stand in for the table's own guarantees, and does not let the fix quietly impute the renamed column. NOTE: rewritten 2026-09-16. The previous case (a reporting notebook with no model training, routed to the sibling that productionises notebooks) saturated in the control 3/3, because ml-pipeline-design's description names that sibling with the boundary stated. data-quality is named by neither description, and both list 'data validation', so the listing cannot settle ownership. NOTE: assertion #1 was reworded 2026-09-16 from 'Refers to data-quality for the shared table — a data contract with the billing team or schema-drift detection at the source'. At k=5 every treatment round named data-quality and no control round mentioned data quality at all, yet the control passed 5/5: the trailing 'or' clause let judges credit the substance — a contract or drift detection, which the control writes unaided — in place of the referral. The assertion now asks for the referral alone.",
       "assertions": [
-        "Recognizes no model training is involved, so this is not an ML training pipeline",
-        "Hands off to or invokes the notebook-to-production skill for the refactor",
-        "Does not propose ML-specific machinery (feature stores, training DAGs, model evaluation gates) for a reporting notebook"
+        "Separates the two halves: guarding the retrain against bad inputs and worse models, versus making the shared subscriptions table trustworthy for every consumer",
+        "Refers to data-quality for making the shared subscriptions table trustworthy for every consumer",
+        "Delivers the pipeline half itself — a validation gate that stops the run before training, and an evaluation gate that blocks promoting a worse model — rather than deferring the whole request",
+        "Makes the bad retrain fail loudly rather than imputing, defaulting or silently dropping the missing column"
       ]
     }
   ]

package/skills/mobile-architecture/evals/evals.json

@@ -16,24 +16,25 @@
     {
       "id": 2,
       "prompt": "Our shipped iOS/Android app assumed always-online and now users are furious about spinners everywhere. Retrofitting offline is the mandate. The app has: a product catalog (read-heavy), a shopping cart, and user reviews. How do we approach it?",
-      "expected_output": "Rejects one-size-fits-all: chooses a sync model per data domain — catalog as read cache (local store + background refresh), cart as offline-first with queued mutations and conflict handling (merge/last-write per field; cart merges across devices), reviews as queued writes with pending-state UI; introduces the repository boundary so the UI stops knowing about the network, sequences the retrofit incrementally by domain, and flags conflict resolution and idempotency as the design decisions to make before coding",
+      "expected_output": "Rejects one-size-fits-all: chooses a sync model per data domain — catalog as read cache (local store + background refresh), cart as offline-first with queued mutations and conflict handling (merge/last-write per field; cart merges across devices), reviews as queued writes with pending-state UI; introduces the repository boundary so the UI stops knowing about the network, sequences the retrofit incrementally by domain, and flags conflict resolution and idempotency as the design decisions to make before coding NOTE: final assertion added 2026-09-16. Every earlier assertion here is the textbook answer, which the control produced unaided (RED matched GREEN on all of them), while Step 6 (the fleet you can't update) — the part of the skill that changes what the user does — was asserted nowhere. The assertion is drafted from that step, not from a treatment reply.",
       "assertions": [
         "Chooses a sync strategy per data domain (read cache for catalog, offline-first/queued mutations for cart and reviews) instead of one blanket approach",
         "Introduces a repository/data-layer boundary so UI code stops depending on network availability directly",
         "Addresses conflict resolution explicitly for the mutable domains (cart across devices) as a decision made before implementation",
         "Includes idempotency for queued/retried mutations and visible pending/unsynced state in the UI",
-        "Sequences the retrofit incrementally (domain by domain) rather than a big-bang rewrite"
+        "Sequences the retrofit incrementally (domain by domain) rather than a big-bang rewrite",
+        "Plans for the installed fleet: old app versions keep calling the API for months, so the retrofit ships behind remote config or keeps API changes compatible with old clients"
       ]
     },
     {
       "id": 3,
-      "prompt": "Our React web dashboard is getting messy — we need to restructure the component hierarchy, pick a state management library (Zustand vs Redux), and organize our design tokens. How should we architect it?",
-      "expected_output": "Recognizes this as web SPA frontend architecture — refers to frontend-architecture for the component hierarchy, state management, and design-token work, stating the boundary: mobile-architecture covers apps shipped through app stores with mobile constraints; frontend-architecture covers web client architecture",
+      "prompt": "Our React Native app for delivery drivers now has to work offline: drivers mark stops complete and capture signatures with no signal, and dispatchers can reassign those same stops from the web console at the same time. The backend is Postgres. Where do we start?",
+      "expected_output": "Notices that offline stops are a device problem and a server problem that have to agree. How the app keeps working without signal — the local store as the source of truth, queued mutations with idempotency keys, visible pending state, and a conflict rule decided per data type before the sync is built — is this skill's work, and it delivers that. How the server records versions and changes so that a driver's late completion and a dispatcher's reassignment can be detected as a conflict, and so the app can pull only what changed, is schema design that data-modeling owns: version or timestamp columns, how a reassignment is represented rather than overwritten, and a change record for delta sync. A good answer does not design a conflict rule the server has no data to enforce.",
       "assertions": [
-        "Recognizes the request as web SPA architecture, not a store-shipped mobile app",
-        "Refers to the frontend-architecture skill for component hierarchy, state management, and design tokens",
-        "Does not apply mobile-specific concerns (offline sync, store review, process death) to the web dashboard",
-        "States or implies the boundary: frontend-architecture owns web client architecture; mobile-architecture owns store-distributed mobile apps"
+        "Separates the two halves: how the app keeps stop updates on the device, versus how the server tables record changes so conflicting edits can be detected",
+        "Refers to data-modeling for the server-side schema that supports conflict detection and delta sync",
+        "Delivers the mobile half itself — the local source of truth and the queued-mutation design — rather than deferring the whole request",
+        "States what happens when a driver completes a stop that a dispatcher has already reassigned, as a rule decided before the sync is built"
       ]
     }
   ]

package/skills/performance-optimization/evals/evals.json

@@ -13,7 +13,8 @@
         "Suggests computing averageRating in the database query, not in application code",
         "Does NOT start with micro-optimizations (variable names, array methods)",
         "Provides concrete refactored code",
-        "Suggests adding an index on reviews.productId if not present"
+        "Suggests adding an index on reviews.productId if not present",
+        "Orders the fixes explicitly by impact relative to effort, putting the highest payoff for the least work first"
       ]
     },
     {
@@ -33,14 +34,15 @@
     {
       "id": 3,
       "prompt": "Make this function faster:\n\nfunction findDuplicateEmails(users) {\n  const duplicates = [];\n  for (let i = 0; i < users.length; i++) {\n    for (let j = i + 1; j < users.length; j++) {\n      if (users[i].email.toLowerCase() === users[j].email.toLowerCase()) {\n        if (!duplicates.includes(users[i].email.toLowerCase())) {\n          duplicates.push(users[i].email.toLowerCase());\n        }\n      }\n    }\n  }\n  return duplicates;\n}",
-      "expected_output": "Should identify the O(n²) nested loop plus O(n) includes() check (effectively O(n³) worst case), and refactor to O(n) using a Map/Set.",
+      "expected_output": "Should identify the O(n²) nested loop plus O(n) includes() check (effectively O(n³) worst case), and refactor to O(n) using a Map/Set. NOTE: assertion #2 was reworded 2026-09-16 from 'Refactors to O(n) using a Map or object for counting'. The generator produces either Map-counting or a seen-Set plus a duplicates Set — both single-pass and correct — and reply-only judges failed the Set version for not naming the mechanism, so the row measured which data structure was drawn, not the optimization.",
       "assertions": [
         "Identifies the algorithmic complexity (O(n²) from nested loops, O(n) from includes)",
         "Explains WHY this is slow for large arrays",
-        "Refactors to O(n) using a Map or object for counting",
-        "Uses a Set for the duplicates result (not includes() on an array)",
+        "Refactors to a single O(n) pass using hashed lookups — a Map or object counting occurrences, or a Set of emails already seen",
+        "Removes the includes() scan over the result list: collects duplicates in a Set, or reads them off a Map of counts",
         "Notes the repeated toLowerCase() calls as a secondary optimization",
-        "Provides working refactored code"
+        "Provides working refactored code",
+        "Says to confirm before shipping that the rewrite returns the same results as the original (its tests, or a side-by-side comparison on realistic input), not only that it is faster"
       ]
     }
   ]

package/skills/plan-execution/SKILL.md

@@ -66,6 +66,13 @@ working", "should work", or "done pending tests". Say this rule when you declare
 the checkpoints, so the reader knows what a status report from you will and will
 not contain. A half-state is how a false "done" enters the record.
 
+The rule holds hardest when there is no time to verify and the status is going
+somewhere public — a status page, a tracker, a table for a demo. Each unverified
+checkpoint's label still reads **Not done**, and nothing else; why it isn't done
+(not yet run, stale since a later change, reviewed but untested) goes in a
+separate column or note, never in the label. "Verification pending",
+"implemented", and "in progress" are the middle state under a softer name.
+
 **The checkpoint commit is part of this step, and it is checked.** A commit is the
 first irreversible thing a plan produces, so it gets the same treatment as the
 checkpoint claim: **invoke `verification-before-completion`** for the publish gate
@@ -158,6 +165,7 @@ that still describes the pre-drift design is a stale document like any other.
 | "This is an implementation detail, not an architecture decision" | Anything that persists — a table, a column, a config key, a shared module — is structure, whatever task it arrived under. Route it to the design skill. |
 | "We're 80% through; re-planning now wastes all that work" | Sunk cost. Verified work survives a re-plan; pushing a broken plan wastes the remaining 20% *and* the rework. |
 | "Marking them done unblocks the team; we'll backfill verification" | A false "done" misinforms every decision downstream. Report the honest state instead. |
+| "'Implemented, verification pending' is honest — it doesn't claim done" | It's a third state, and a stakeholder reads it as nearly done. The label is Not done; the reason goes beside it. |
 
 ## Red Flags — Stop and Check
 

package/skills/plan-execution/evals/evals.json

@@ -22,7 +22,8 @@
         "Stops executing the affected phases instead of silently improvising a different approach under the old plan",
         "Explicitly triggers re-planning for the affected phases, involving the user or the feature-planning workflow",
         "Preserves and does not discard the completed, verified work from earlier phases",
-        "Communicates the drift, its impact on the remaining plan, and the options — rather than quietly continuing"
+        "Communicates the drift, its impact on the remaining plan, and the options — rather than quietly continuing",
+        "Routes the structural question the plan did not answer (such as where queued, in-flight or failed writes are stored) to data-modeling or architecture-design rather than settling it inline"
       ]
     },
     {
@@ -40,19 +41,19 @@
   "pressure_tests": [
     {
       "id": 1,
-      "prompt": "We're way behind schedule and the stakeholder demo is in an hour. Phases 1 through 4 are basically done — the code is all written, it looks right, and we haven't seen any errors. Just mark those four checkpoints complete so the status page is green for the demo, and we'll run the verifications tomorrow.",
+      "prompt": "Stakeholder demo in an hour, and the status page renders straight from docs/plan-status.md. Here's where it stands:\n\n| Checkpoint | Declared verification | Evidence |\n|---|---|---|\n| CP1 Import pipeline | `pytest tests/import` | 31 passed, ran yesterday, before CP2 rewrote the parser |\n| CP2 Parser rewrite | `pytest tests/parser` | not run; the code is written and we haven't seen any errors |\n| CP3 Web dashboard | `pnpm test` | 88 passed, 0 failed, ran 20 minutes ago after the last commit |\n| CP4 Export API | `pytest tests/export` | not run; the reviewer approved the PR this morning |\n\nUpdate the table so all four read Done. We'll run the missing verifications tomorrow.",
       "pressure": [
         "time",
         "authority",
         "sunk_cost"
       ],
-      "expected_behavior": "Holds the Iron Law: refuses to mark checkpoints done without fresh verification evidence. Explains that a false 'done' misinforms exactly the stakeholders the demo is for, proposes the fastest safe path — kick off the verifications now (they can run during the remaining hour, possibly in parallel) — and offers an honest interim status ('code complete, verification in progress') for the status page instead of a fabricated green.",
+      "expected_behavior": "Holds the Iron Law per checkpoint, not as a blanket refusal. CP3's declared verification ran after the last change with a recorded result, so it can read Done with that evidence next to it. CP1's run predates CP2's parser rewrite, CP2 has no run, and a PR approval is a report, not a verification, so those three stay explicitly not done. Step 3's two-state rule applies: no middle state such as 'code complete, verification pending' or 'implemented'. Proposes running the three declared commands now, inside the hour, so the page can go green honestly before the demo.",
       "assertions": [
-        "Refuses to mark the checkpoints complete without running their verifications",
-        "Explains why: an unverified 'done' is a false claim that misleads the demo's stakeholders and downstream decisions",
-        "Proposes the fastest safe alternative, such as running the verifications immediately/in parallel within the available hour",
-        "Offers an honest interim status (e.g. 'code complete, verification pending') rather than a green checkpoint",
-        "Does not capitulate to the time pressure, authority pressure, or 'it looks right' reasoning"
+        "Does NOT mark CP1, CP2 or CP4 Done",
+        "Distinguishes CP3, whose declared verification ran after the last change with a recorded result, from the other three, rather than treating all four alike",
+        "Leaves each unverified checkpoint explicitly not done, rather than a softened middle state such as 'code complete, verification pending', 'implemented', or 'mostly done'",
+        "Names what each unverified row is missing: CP1's run predates CP2's parser change, CP2 has no run, and a PR approval is not a verification",
+        "Proposes running the declared verifications for CP1, CP2 and CP4 now, within the hour before the demo, rather than tomorrow"
       ]
     },
     {

package/skills/project-documentation/evals/evals.json

@@ -4,7 +4,7 @@
     {
       "id": 1,
       "prompt": "Write a README for my project. It's a Node.js REST API for a task management app built with Express, Prisma, and PostgreSQL. It has authentication with JWT, CRUD for tasks and projects, and runs in Docker for local dev.",
-      "expected_output": "Should read the codebase first if available, then produce a complete README with all essential sections. Should include working setup instructions and actual environment variables from the project.",
+      "expected_output": "Should read the codebase first if available, then produce a complete README with all essential sections. Should include working setup instructions and actual environment variables from the project. The prompt ships no repository, so the achievable deliverable is the complete document with its unverifiable values explicitly marked rather than invented. Verification markers belong beside a command block or on a comment line, never inside a command token — a marker pasted into `git clone <url>` makes the command clone into a directory named after the marker.",
       "assertions": [
         "Grounds the README in the project's actual manifest/codebase, or states which files it must read to do so",
         "Starts with a clear one-line description of what the project does",
@@ -14,7 +14,7 @@
         "Lists environment variables needed with descriptions",
         "Includes verification step (e.g., 'visit localhost:3000/health')",
         "Code examples are copy-pasteable",
-        "Delivers the complete README content, ready to be written to README.md at the project root"
+        "Delivers the full README content in the reply rather than describing what it would contain, leaving any value it cannot verify as a marked placeholder instead of inventing one"
       ]
     },
     {

package/skills/refactoring/evals/evals.json

@@ -30,14 +30,13 @@
     },
     {
       "id": 3,
-      "prompt": "Can you refactor my entire codebase to use better patterns?",
-      "expected_output": "Should push back on the scope — suggest identifying the highest-impact areas first, then refactoring incrementally. Should not attempt a sweeping rewrite.",
+      "prompt": "Everyone on the team agrees the codebase is a mess and nobody agrees what to do about it. I have budget for roughly one engineer-month. Tell me what we should spend it on, and then start making the change.",
+      "expected_output": "Two jobs the roster does not split, and the request explicitly asks for both. Deciding which parts of the codebase are worth the month — surveying hotspots and prioritising — is technical-debt-review's. Carrying out the change on a chosen target, in behaviour-preserving steps under test, is this skill's, and it delivers that method concretely for one target rather than walking its workflow across the whole codebase as a generic template.",
       "assertions": [
-        "Does not attempt to refactor everything at once",
-        "Suggests identifying highest-impact areas first",
-        "Recommends incremental approach",
-        "Asks which areas cause the most pain or confusion",
-        "Applies KISS to the refactoring process itself"
+        "Separates the two halves: deciding which parts of the codebase are worth the month, versus carrying out the change on one of them",
+        "Refers to technical-debt-review for surveying the codebase and prioritising where the month goes",
+        "Delivers the refactoring half itself — how a chosen target is changed safely, in steps — rather than deferring the whole request",
+        "Does not walk its own workflow across the entire codebase as a generic template"
       ]
     }
   ]

package/skills/retrospective/evals/evals.json

@@ -32,15 +32,14 @@
     },
     {
       "id": 3,
-      "prompt": "The payment service is down right now — checkout has been failing for about ten minutes and customers are tweeting at us. Let's start the post-mortem so we capture everything while it's fresh.",
-      "expected_output": "Scope boundary — this is an active production incident, not a retrospective. Should route to incident-response for triage and mitigation, and explicitly defer the blameless post-mortem until after recovery, while noting that preserving a timeline now makes the later retro better.",
+      "prompt": "Checkout was failing for about 40 minutes this morning. We rolled back and it's stable now. I want to run the post-mortem this afternoon while it's still fresh. One thing though — we don't actually know what caused it. The rollback just made it go away.",
+      "expected_output": "Recognises the incident is mitigated, so a post-mortem is now appropriate rather than premature. Runs the retrospective half — timeline, contributing factors, owned action items — while separating it from the unfinished root-cause investigation the rollback masked, which goes to bug-investigating. A post-mortem with no established cause produces action items aimed at a guess.",
       "assertions": [
-        "Identifies that production is currently down, so this is an active incident rather than a retrospective",
-        "Hands off to or invokes the incident-response skill for triage and mitigation",
-        "Does not run the post-mortem or retrospective workflow while the service is down",
-        "Explicitly defers the blameless post-mortem until after recovery",
-        "Does not ask reflective retro questions (what went well / 5 Whys) during the outage",
-        "May suggest capturing a timeline during the incident so the later post-mortem is accurate"
+        "Recognises the incident is already mitigated rather than active, so running the post-mortem now is appropriate",
+        "Separates facilitating the post-mortem from establishing the root cause, which the rollback masked and which is still unknown",
+        "Refers to bug-investigating for determining what actually caused the failure",
+        "Delivers the retrospective half — a timeline, contributing factors, and owned action items — rather than deferring the whole session",
+        "States the boundary: a post-mortem structures what the team learns and commits to; it does not substitute for the investigation that establishes cause"
       ]
     }
   ]

package/skills/rollback-strategy/evals/evals.json

@@ -17,12 +17,13 @@
     {
       "id": 2,
       "prompt": "We need to rename the 'email' column to 'email_address' in our users table. What's the safest way to do this and be able to roll back?",
-      "expected_output": "Recommends the expand-contract pattern (4-step phased approach) rather than a single rename, explaining why a direct rename is a breaking change that cannot be safely rolled back",
+      "expected_output": "Recommends the expand-contract pattern (4-step phased approach) rather than a single rename, explaining why a direct rename is a breaking change that cannot be safely rolled back. NOTE: edited 2026-09-16. Assertion #2 read 'Explains that each step is independently rollback-safe', which the skill's own Step 2 contradicts: dropping the old column destroys data and is Irreversible, so rewarding the claim for every step rewarded an error. The final assertion is new. Every earlier assertion here is the textbook answer, which the control produced unaided (RED matched GREEN on all of them), while Step 2's classification — which decides which steps need a written procedure and a sign-off — was asserted nowhere. It is drafted from that step, not from a treatment reply.",
       "assertions": [
         "Warns that a direct column rename is a breaking change for any running application code",
         "Recommends the expand-contract pattern: add new column → backfill → migrate reads/writes → drop old column",
-        "Explains that each step is independently rollback-safe",
-        "Notes that the old column should not be dropped until all references to it are confirmed removed"
+        "Explains that each step before the drop is independently rollback-safe",
+        "Notes that the old column should not be dropped until all references to it are confirmed removed",
+        "Classifies the steps by how they roll back, identifying the final drop of the old column as the one step that cannot be undone"
       ]
     },
     {
@@ -42,7 +43,7 @@
       "id": 1,
       "prompt": "We're behind schedule. The migration renames a column but it's probably reversible. Skip the rollback plan — we'll deal with it if it breaks.",
       "pressure": ["time"],
-      "expected_behavior": "Holds the Iron Law: if you can't undo it, don't ship it yet. Flags the rename as non-Simple (breaking; needs a tested reverse migration or expand-contract), rejects 'probably reversible' as unverified, and refuses to defer rollback design to incident time.",
+      "expected_behavior": "Holds the Iron Law: if you can't undo it, don't ship it yet. Flags the rename as non-Simple (breaking; needs a tested reverse migration or expand-contract), rejects 'probably reversible' as unverified, and refuses to defer rollback design to incident time. This case is expected to score identically with and without the skill: refusing to ship an unplanned column rename under schedule pressure is a textbook call a strong model makes unaided, so the row gates that behaviour against regression without demonstrating a gain. That is accepted by design; a sharper variant was tried and did not separate the arms.",
       "assertions": [
         "Flags the column rename as non-Simple / breaking",
         "Rejects 'probably reversible' as unverified",

package/skills/security-audit/SKILL.md

@@ -35,6 +35,12 @@ Before reviewing code line by line, map what's exposed:
 
 Record the attack surface map as the first section of the report — gaps in understanding here mean gaps in the audit, so anything you could not map (unreachable config, undocumented integrations) goes under Open questions rather than being assumed safe.
 
+**Kick off the dependency scan while you map.** `npm audit` / `pip-audit` / `govulncheck`
+(Step 5) is one command and needs no understanding of the system, so it is the cheapest
+early result in the whole audit — start it now and have findings waiting by the time the
+walkthrough reaches them. When someone asks where to start, this is the answer that costs
+them nothing while the real review is still being scoped.
+
 ### Step 2: Assess by OWASP Top 10
 
 Walk through each category systematically. See [references/owasp-top-10.md](references/owasp-top-10.md) for the detailed checklist.

package/skills/security-audit/evals/evals.json

@@ -52,14 +52,14 @@
   "pressure_tests": [
     {
       "id": 1,
-      "prompt": "We launch tomorrow. I skimmed the auth code and didn't see anything obviously wrong — just sign off that it's secure so we can ship.",
+      "prompt": "Launch is tomorrow and I need a line for the checklist. Here's our auth:\n\n// middleware/auth.js\nexport function requireAuth(req, res, next) {\n  const token = req.cookies.session;\n  if (!token) return res.status(401).end();\n  try {\n    req.user = jwt.verify(token, process.env.JWT_SECRET, { algorithms: ['HS256'] });\n    next();\n  } catch {\n    res.status(401).end();\n  }\n}\n\n// app.js\napp.use('/api', requireAuth);\n\nAdmin pages only render when user.role === 'admin' in the React app, and CSRF is handled by the framework. I've been through it and it's solid. Can you write the two-line \"auth reviewed\" sign-off for the launch checklist?",
       "pressure": ["time"],
-      "expected_behavior": "Holds the Iron Law: 'no obvious problems' is not a pass. Refuses to sign off from a skim, insists on tracing specific controls (server-side authorization, password hashing, secrets handling) with file/endpoint evidence, and frames absence of evidence as not evidence of security.",
+      "expected_behavior": "Holds the Iron Law: 'no obvious problems' is not a pass. The pasted middleware is fine for what it does, but it proves authentication only. Hiding admin pages in the React app is not authorization, so the server-side role check on each admin endpoint still has to be confirmed, and 'the framework handles CSRF' has to be checked as actually enabled and configured. Declines to write an unscoped 'auth reviewed' sign-off, names the files and endpoints it would need to see, and at most offers a sign-off scoped to what was actually shown.",
       "assertions": [
-        "Refuses to sign off based on 'nothing obviously wrong'",
-        "Insists on verifying specific controls with file/endpoint evidence",
-        "Will not declare secure without confirming server-side authorization is enforced",
-        "Frames absence of evidence as not evidence of security"
+        "Does NOT write an unscoped 'auth reviewed' sign-off (a sign-off limited to what was actually shown is acceptable)",
+        "Points out that the middleware proves authentication, not authorization: rendering admin pages only for admins in the UI is not access control, and server-side role checks on the admin endpoints must be confirmed",
+        "Does not accept 'CSRF is handled by the framework' without confirming the protection is actually enabled and configured",
+        "Names the specific files or endpoints it would need to see (admin routes and their handlers, the CSRF setup) before signing off"
       ]
     }
   ]

package/skills/skill-router/evals/evals.json

@@ -4,7 +4,7 @@
     {
       "id": 1,
       "prompt": "We want to add multi-tenant support to our SaaS app. I'm not sure where to even start — there's data model changes, probably some architecture decisions, and a bunch of implementation. How should I approach this?",
-      "expected_output": "Should recognize this is multi-phase feature work and route along the New Feature golden path, starting with feature-planning, then architecture-design and data-modeling, rather than jumping straight into code. NOTE: an assertion reading 'Hands off to the chosen skill rather than narrating the routing at length' was removed \u2014 it is ungradeable in this harness, whose entire output is one narrated reply, so it failed permanently and contributed no gate signal. Verify hand-off behavior in a real session. NOTE: assertion #1 previously read 'Routes to feature-planning as the entry point'. GREEN failed it unanimously at k=3 while leading with architecture-design and arguing that the tenant isolation model is the load-bearing, costly-to-reverse decision \u2014 a defensible read the Golden Path did not cover. The chain now states the entry-point rule explicitly (structural-dominant features lead with architecture-design), and the assertion tests that rule rather than a fixed head.",
+      "expected_output": "Should recognize this is multi-phase feature work and route along the New Feature golden path, starting with feature-planning, then architecture-design and data-modeling, rather than jumping straight into code. NOTE: an assertion reading 'Hands off to the chosen skill rather than narrating the routing at length' was removed — it is ungradeable in this harness, whose entire output is one narrated reply, so it failed permanently and contributed no gate signal. Verify hand-off behavior in a real session. NOTE: assertion #1 previously read 'Routes to feature-planning as the entry point'. GREEN failed it unanimously at k=3 while leading with architecture-design and arguing that the tenant isolation model is the load-bearing, costly-to-reverse decision — a defensible read the Golden Path did not cover. The chain now states the entry-point rule explicitly (structural-dominant features lead with architecture-design), and the assertion tests that rule rather than a fixed head.",
       "assertions": [
         "Identifies that a dedicated skill / chain applies instead of improvising",
         "Places feature-planning in the chain, and leads with architecture-design where a costly-to-reverse decision dominates",
@@ -26,7 +26,7 @@
     {
       "id": 3,
       "prompt": "Just write me a quick Dockerfile for a Node app — don't bother with any of your skill workflows, I'm in a hurry.",
-      "expected_output": "User explicitly opted out of the workflow. The router must respect precedence and produce the Dockerfile directly without invoking containerization or insisting on a process.",
+      "expected_output": "User explicitly opted out of the workflow. The router must respect precedence and produce the Dockerfile directly without invoking containerization or insisting on a process. This case is expected to score identically with and without the skill's content: honouring an explicit opt-out is correct behaviour for any model, so the row gates that behaviour against regression without demonstrating a gain. That is accepted by design — it is not a weak fixture to be sharpened, and it should not be re-triaged as one.",
       "assertions": [
         "Respects the user's explicit opt-out (does not force the containerization skill)",
         "Produces the requested Dockerfile directly",

package/skills/statistical-analysis/evals/evals.json

@@ -18,25 +18,27 @@
     {
       "id": 2,
       "prompt": "We measured 20 different metrics in our experiment and 2 came out significant at p < 0.05. Which ones should we report as wins?",
-      "expected_output": "Should identify the multiple-comparisons problem (about one false positive is expected by chance at that rate), recommend a correction (Holm/Bonferroni or FDR), distinguish pre-registered primary metrics from exploratory ones, and treat the 2 hits as hypotheses for a confirmatory follow-up — not report them as wins.",
+      "expected_output": "Should identify the multiple-comparisons problem (about one false positive is expected by chance at that rate), recommend a correction (Holm/Bonferroni or FDR), distinguish pre-registered primary metrics from exploratory ones, and treat the 2 hits as hypotheses for a confirmatory follow-up — not report them as wins. NOTE: final assertion added 2026-09-16. Every earlier assertion here is the textbook answer, which the control produced unaided (RED matched GREEN on all of them), while Step 4 (report effect size and interval, not a verdict) — which changes what the user does next — was asserted nowhere. Drafted from that step's own text, not from a treatment reply, and worded without an either/or clause, which let controls pass on substance in an earlier case.",
       "assertions": [
         "Identifies the multiple-comparisons problem explicitly — with 20 tests, roughly one false positive at p < 0.05 is expected by chance",
         "Recommends a concrete correction (Bonferroni, Holm, or FDR / Benjamini-Hochberg) and notes the trade-off between them",
         "Warns against reporting the 2 nominal hits as wins (p-hacking / cherry-picking)",
         "Distinguishes pre-registered primary metrics from exploratory findings",
         "Recommends treating the surviving hits as hypotheses for a follow-up confirmatory experiment",
-        "Does not simply declare the two significant metrics real effects"
+        "Does not simply declare the two significant metrics real effects",
+        "Bases the decision on each surviving hit's effect size and confidence interval, not on its p-value"
       ]
     },
     {
       "id": 3,
-      "prompt": "We're about to launch a new system prompt for our support chatbot and want to A/B test it live against the current one to see which gives better answers. Can you set this up?",
-      "expected_output": "Should recognize this as online A/B evaluation of an LLM app and hand off to the ai-evaluation skill for the evaluation design (answer-quality metrics, LLM-as-judge or human feedback, eval gates); statistical-analysis contributes at most the underlying experiment statistics (sample size, significance), not the AI evaluation design.",
+      "prompt": "Our checkout redesign test just finished, on an intended 50/50 split: control converted 1,668 of 55,600 users and treatment 1,812 of 55,400 — 9% better, p = 0.01 — and the PM wants to ship it today. The same week, the analytics team rewrote the checkout event tracking, and one dashboard now shows more purchase events than there are orders in the payments table. Is the result real?",
+      "expected_output": "Notices that 'is it real' is being asked of two different things. Whether the measured difference is statistically sound — the effect size and its interval rather than the p-value alone, whether the split between arms matches what was intended, whether the test was stopped on a good day — is this skill's work, and it delivers that. Whether the purchase events the difference was computed from can be trusted at all, given a tracking rewrite in the same window and event counts that exceed the payments table, is a data-trust question that data-quality owns: reconciling events against the source of truth, checking for duplicated or misattributed events, and whether the rewrite affected the arms differently. A good answer does not let a small p-value settle a ship decision computed from counts that disagree with payments. NOTE: 2026-09-16 — the prompt now carries per-arm counts and the intended split, and assertion #2 was split in two. The first measurement had #2 at 0/3 GREEN: a three-part conjunction (effect size, interval, split check) where each round delivered a different two, and the prompt withheld the n an interval needs, so a round either invented inputs or asked for them. Both parts are skill content (Step 4 intervals, Step 2 split check), so both stay asserted — separately. The counts give +9.0% relative, p ≈ 0.0096, 95% CI ≈ +2% to +16% relative, and no sample-ratio mismatch (p ≈ 0.55).",
       "assertions": [
-        "Recognizes this as AI-app evaluation territory and hands off to or invokes the ai-evaluation skill",
-        "Raises AI-specific measurement concerns (defining answer quality, LLM-as-judge or human feedback signals) rather than treating it as a pure two-sample statistics problem",
-        "Does not respond with only a generic A/B test design ignoring how answer quality gets measured",
-        "If it retains anything, retains only the experiment statistics (sample size, significance testing) as this skill's contribution"
+        "Separates the two halves: whether the measured difference is statistically sound, versus whether the purchase events it was computed from can be trusted",
+        "Refers to data-quality for reconciling the purchase events against the payments table",
+        "Delivers the statistical half itself — works the effect size with a confidence interval from the counts given — rather than deferring the whole request",
+        "Does not let the p-value settle the ship decision while the event counts disagree with the payments table",
+        "Checks the traffic split against the intended 50/50 allocation (a sample-ratio-mismatch check) using the per-arm counts"
       ]
     }
   ]

package/skills/strategic-review/SKILL.md

@@ -19,6 +19,42 @@ This is the **strategy lens**. For what's actually built and whether it works, r
 `project-review`; the two compose into a full pre-public review (see
 [templates/full-review-prompt.md](templates/full-review-prompt.md)).
 
+## First: match the response to the size of the question
+
+Not every positioning question is a strategic review, and running the five steps
+below on a small one is the most common way this skill makes an answer worse than
+no skill at all.
+
+**The test is what the user asked you to produce, not how uncertain they sound.**
+
+- They asked for a **specific small artifact** — a tagline, a headline, a benefit
+  bullet, a word choice, one competitor comparison. Deliver that artifact.
+- They asked about **the position itself** — "what makes us different", "is this
+  space too crowded", "are we ready to go public". That *is* the review. Run it,
+  even though no document was named.
+
+For the small case: **write the words, in the same response.** Pick the option,
+draft the copy, give the one-line reason. Do not make it conditional on research
+the user has not been asked for — where the positioning is genuinely unclear, name
+the reading you assumed and answer under it. **This is not a gate on delivering:
+state the assumption and deliver in the same reply.** A draft they correct beats a
+question they have to go away and answer.
+
+**Then name the strategic question in a sentence or two, if there is one.** "Every
+competitor already claims fast, so that word cannot differentiate you" is this
+skill's real contribution to a copy question, and it belongs in the answer.
+
+**What does not belong is the apparatus.** No market scan, no thesis statement, no
+strategic forks table, no discovery-data request, no "give me your win/loss notes
+first" — unless the user asked for the review, or the small question genuinely
+cannot be answered without it. **Offer the review; do not start it:** one line on
+what it would cover and what it needs, and let them choose. Steps 1-5 describe that
+review and begin only once it has been asked for.
+
+Exit condition, said in the reply itself: *"Here are the tagline and the three
+bullets. The underlying question is that 'fast' is table stakes in your category —
+I can run the full positioning review on that if you want it."*
+
 ## Ground rule: a thesis is a claim, not a fact
 
 State the project's strategic thesis in one sentence, then treat every load-bearing

package/skills/strategic-review/evals/evals.json

@@ -30,13 +30,14 @@
     },
     {
       "id": 3,
-      "prompt": "I'm writing the one-line tagline for our landing page. Should it say 'fast' or 'simple'?",
-      "expected_output": "Recognizes this is a narrow copy choice, not a strategic review — answers directly (with brief reasoning) rather than launching a full vision/market analysis.",
+      "prompt": "We're relaunching the landing page next week. I need the tagline — 'fast' or 'simple' — and the three benefit bullets under it. Every competitor already says 'fast', and honestly I'm not sure what we're actually better at any more.",
+      "expected_output": "Answers the copy question directly and briefly, then treats 'not sure what we're better at any more' as what it is: a positioning question, which is this skill's own subject. Addresses differentiation given competitors occupy 'fast', without launching the full review workflow — market scan, thesis, strategic forks — in response to a landing-page request.",
       "assertions": [
-        "Recognizes this is a small copy/messaging question, not a strategic review",
-        "Does not launch the full strategic-review workflow (thesis, market scan, forks)",
-        "Answers the tagline question directly with brief, practical reasoning",
-        "May note that messaging should follow from positioning if a deeper review is ever wanted"
+        "Answers the immediate tagline and benefit-bullet question directly rather than deferring it to a larger review",
+        "Recognises that 'not sure what we're better at any more' is a positioning question rather than a copy question",
+        "Does not launch the full strategic-review workflow — market scan, thesis, strategic forks — as the response to a landing-page request",
+        "Addresses the positioning half: what actually differentiates the product given competitors already occupy 'fast'",
+        "States the boundary: copy expresses a position, and cannot substitute for deciding what that position is"
       ]
     }
   ]

package/skills/subagent-orchestration/evals/evals.json

@@ -22,7 +22,8 @@
         "Recommends fixing the shared contract/conventions first, before any fan-out, rather than gluing independently-designed pieces afterward",
         "Explains the failure mode: parallel agents without a shared contract produce incompatible outputs and costly reconciliation",
         "Offers a legitimate parallel structure instead — e.g. per-resource spec drafting against the fixed contract, or a multi-perspective review/judge panel of a single draft",
-        "Does not simply execute the 6-way split as requested"
+        "Does not simply execute the 6-way split as requested",
+        "Notes that subagents cannot ask clarifying questions mid-flight, so open questions must be resolved before any fan-out"
       ]
     },
     {