swe-workflow-skills 0.8.2 → 0.9.0

package/skills/data-pipeline-design/evals/evals.json

@@ -17,24 +17,25 @@
     {
       "id": 2,
       "prompt": "Our nightly job loads yesterday's orders into the warehouse. When it fails midway and we rerun it, we get duplicate rows — and backfilling last month means manually editing the date in the script 30 times. Fix the design.",
-      "expected_output": "Identifies the non-idempotent append-only load as the root cause; converts the job to an idempotent write pattern (delete+insert by partition, merge/upsert on a key, or partition overwrite) parameterized over a logical date, so backfill becomes running the same job over a range via the orchestrator — with a lookback window for late-arriving data",
+      "expected_output": "Identifies the non-idempotent append-only load as the root cause; converts the job to an idempotent write pattern (delete+insert by partition, merge/upsert on a key, or partition overwrite) parameterized over a logical date, so backfill becomes running the same job over a range via the orchestrator — with a lookback window for late-arriving data NOTE: final assertion added 2026-09-18. Every earlier assertion here is the textbook answer, which the control produced unaided (RED matched GREEN on all of them), while Step 7 (orchestrate and operate) — which changes what the user does next — was asserted nowhere. Drafted from that step's own text, not from a treatment reply, and worded without an either/or clause. Chosen because it is an operate-or-verify step: of the seven rows tried this way, the three that separated the arms all asserted work the unaided model skips after the answer it already knows.",
       "assertions": [
         "Identifies the root cause as a non-idempotent, append-only load",
         "Proposes an idempotent write pattern: delete+insert by partition, merge/upsert on a unique key, or partition-overwrite semantics",
         "Parameterizes the run over a logical date/window so any day can be run or re-run identically (backfill = the same job over a range)",
         "Recommends orchestrator-native backfill (or an equivalent parameterized range run) over hand-edited scripts",
-        "Considers late-arriving or corrected data (lookback window, watermark, or reprocessing policy), not just the happy path"
+        "Considers late-arriving or corrected data (lookback window, watermark, or reprocessing policy), not just the happy path",
+        "Makes a failed run visible: retries with an alert on final failure, so a load that fails is noticed rather than found later"
       ]
     },
     {
       "id": 3,
-      "prompt": "Design the data pipeline that produces training features for our churn model — we need point-in-time correct features and a training set refreshed weekly.",
-      "expected_output": "Recognizes this as an ML training/feature pipeline, not analytics ELT — hands off to ml-pipeline-design, flagging point-in-time correctness/leakage as the ML-specific concern that a BI-oriented warehouse design does not address",
+      "prompt": "Our nightly dbt run takes five hours and the warehouse bill has doubled since January. Most of the models are full refreshes over the whole event table. Leadership wants both numbers down before the next board meeting.",
+      "expected_output": "Two numbers, two owners, and the roster does not say which is which. Restructuring full refreshes into incremental builds so less work runs each night is this skill's, and it delivers that. How the warehouse is sized, and how much of the doubled bill this job actually accounts for, is finops-cost-optimization's. A good answer establishes that share before promising a reduction, because a pipeline rewrite that addresses a minority of the spend will not move the bill.",
       "assertions": [
-        "Recognizes this is an ML training/feature pipeline, not an analytics/BI ELT pipeline",
-        "Refers to the ml-pipeline-design skill for feature engineering and training orchestration",
-        "Flags point-in-time correctness / data leakage as the ML-specific concern that distinguishes this from BI work",
-        "Does not answer with a BI-oriented warehouse/dbt marts design as the primary solution"
+        "Separates the two halves: how the models are built and refreshed, versus how the warehouse spend is sized and attributed",
+        "Refers to finops-cost-optimization for warehouse sizing and attributing the bill",
+        "Delivers the pipeline half itself — moving the full refreshes onto incremental builds — rather than deferring the whole request",
+        "Establishes what share of the bill this job actually accounts for before promising a reduction"
       ]
     }
   ]

package/skills/data-quality/SKILL.md

@@ -24,6 +24,13 @@ staleness → freshness, broken joins → `relationships`. Wins from day one, an
 the suite grows the same way tests do after bugs — every new incident adds a
 check that would have caught it.
 
+**An incident that nobody noticed failed twice, and you only own the first failure.**
+Not knowing the data was wrong is a detection gap — the checks below close it. Knowing and
+not telling anyone is a routing gap: who gets paged, on what threshold, and through which
+channel is `observability-design`'s work. Build the check that would have caught this
+incident, then say explicitly that a check nobody is paged for will produce the same
+outcome next time.
+
 ### Step 2: Check at the Right Layer — Boundary First
 
 Where a check runs matters as much as what it checks:

package/skills/data-quality/evals/evals.json

@@ -16,23 +16,25 @@
     {
       "id": 2,
       "prompt": "An upstream team owns the events we ingest and keeps renaming columns and changing types without telling us. Our pipeline breaks silently every few weeks. What should we put in place?",
-      "expected_output": "Proposes a producer/consumer data contract (schema, types, semantics, SLA) enforced at the ingestion boundary so violations fail loudly at entry; shifts validation left into the producer's CI or publish path; adds versioning/deprecation windows for breaking changes; and backstops enforcement with schema-drift and volume/distribution anomaly detection",
+      "expected_output": "Proposes a producer/consumer data contract (schema, types, semantics, SLA) enforced at the ingestion boundary so violations fail loudly at entry; shifts validation left into the producer's CI or publish path; adds versioning/deprecation windows for breaking changes; and backstops enforcement with schema-drift and volume/distribution anomaly detection NOTE: final assertion added 2026-09-16. Every earlier assertion here is the textbook answer, which the control produced unaided (RED matched GREEN on all of them), while Step 4 (route failures to an owner) — the part of the skill that changes what the user does — was asserted nowhere. The assertion is drafted from that step, not from a treatment reply.",
       "assertions": [
         "Proposes an explicit data contract between producer and consumer (schema, types, semantics, SLAs), not just more downstream tests",
         "Enforces the schema at the ingestion boundary so violations fail loudly at entry instead of propagating silently",
         "Shifts checks left: contract validation in the producer's CI or at publish time",
         "Covers change management for breaking changes: contract versioning or a deprecation window",
-        "Adds detection for what enforcement misses: schema-drift alerts or anomaly checks on volume/distributions"
+        "Adds detection for what enforcement misses: schema-drift alerts or anomaly checks on volume/distributions",
+        "Routes each failed check to a named owner of the affected table or domain, rather than leaving alerts where nobody watches them"
       ]
     },
     {
       "id": 3,
-      "prompt": "We need realistic but fake customer data to seed staging so QA can test the checkout flow — GDPR means we can't copy production.",
-      "expected_output": "Recognizes this as generating synthetic/test data, not validating production data — hands off to test-data-strategy (factories, synthetic generation, anonymization) rather than proposing expectations, contracts, or freshness checks",
+      "prompt": "A customer told us our revenue dashboard had been wrong for three days before anyone here noticed. The upstream team had renamed a column. I need to make sure that never happens silently again.",
+      "expected_output": "'Never happens silently' is two problems the roster does not separate. Knowing the data is wrong — a schema-drift check and a freshness check that would have caught a renamed column on the first run, plus a contract with the upstream team so the rename surfaces before it lands — is this skill's work, delivered concretely. Getting a person's attention once something is known to be wrong, including who is paged and on what threshold, is observability-design's. A good answer treats detection and notification as separate failures, because this incident failed at both.",
       "assertions": [
-        "Recognizes this is about generating test/synthetic data, not validating production data quality",
-        "Refers to the test-data-strategy skill for factories, synthetic data generation, and GDPR-safe anonymization",
-        "Does not answer with data-quality machinery (expectations, contracts, freshness checks) as the solution"
+        "Separates the two halves: detecting that the data is wrong, versus getting a person's attention when it is",
+        "Refers to observability-design for alert routing and who gets paged",
+        "Delivers the detection half itself — a schema-drift or freshness check that would have caught the renamed column — rather than deferring the whole request",
+        "Raises a contract with the upstream team so the rename surfaces before it lands, not only afterwards"
       ]
     }
   ]

package/skills/dependency-impact-analysis/evals/evals.json

@@ -21,7 +21,7 @@
       "assertions": [
         "Confirms this is a backwards-compatible (additive) change",
         "Notes that existing code ignores the new column and remains safe",
-        "May mention that ORM models should be updated to avoid unexpected behavior when they serialize full rows",
+        "Flags that code selecting whole rows — ORM models, serializers — will start returning the new column",
         "Does not require complex coordination or versioning",
         "Gives a clear 'safe to proceed' verdict with any minor caveats"
       ]
@@ -33,7 +33,7 @@
       "assertions": [
         "Recognizes this is not a dependency impact analysis request",
         "Does not perform a blast radius analysis for adding auth",
-        "May note that adding required auth to an existing endpoint is a breaking change for current consumers (brief impact observation)",
+        "Notes that adding required auth to a live endpoint is a breaking change for its current consumers",
         "Directs to security-audit or api-design skill for the actual implementation guidance"
       ]
     }

package/skills/effort-estimation/SKILL.md

@@ -39,10 +39,17 @@ Convert to time or cost via team velocity or loaded cost, add a 20-30% buffer fo
 unknowns, and communicate a range in stakeholder language — weeks and dollars, not
 story points. A single number becomes a commitment; a range communicates confidence.
 
+**Say when the number gets revisited.** A range nobody revisits hardens into the commitment
+you avoided giving — so hand over the estimate with its re-forecast attached: the next point
+where real data replaces assumption (a phase boundary, the end of a spike, the first sprints
+of actuals), and what would move the number. That sentence is what keeps an early-discovery
+estimate honest; without it, "wide range" is just a number the stakeholder rounds down.
+
 ### Step 5: Track and Calibrate
 
-Compare estimated vs actual each sprint, look for systematic bias, and recalibrate
-velocity on the last 3-5 sprints, not the all-time average.
+Compare estimated with actual — per sprint on delivery work, at each phase boundary on a
+project-sized estimate — look for systematic bias, and recalibrate on the last 3-5 data
+points, not the all-time average.
 
 ## Principles Applied
 

package/skills/effort-estimation/evals/evals.json

@@ -18,14 +18,15 @@
     {
       "id": 2,
       "prompt": "Our VP wants a budget estimate for a 6-month data platform rebuild. We're in early discovery and a lot is still unknown. How do I estimate this responsibly?",
-      "expected_output": "Should recommend three-point estimation for high-uncertainty work, produce wide ranges, clearly state assumptions and unknowns, and suggest narrowing the estimate through phased discovery.",
+      "expected_output": "Should recommend three-point estimation for high-uncertainty work, produce wide ranges, clearly state assumptions and unknowns, and suggest narrowing the estimate through phased discovery. NOTE: final assertion added 2026-09-18. Every earlier assertion here is the textbook answer, which the control produced unaided (RED matched GREEN on all of them), while Step 5 (track and calibrate) — which changes what the user does next — was asserted nowhere. Drafted from that step's own text, not from a treatment reply, and worded without an either/or clause. Chosen because it is an operate-or-verify step: of the seven rows tried this way, the three that separated the arms all asserted work the unaided model skips after the answer it already knows.",
       "assertions": [
         "Recommends three-point estimation due to high uncertainty",
         "Produces wide initial ranges (appropriate for early discovery)",
         "Suggests a phased approach: estimate phase 1 tightly, phases 2-3 loosely",
         "Clearly states which assumptions drive the estimate",
         "Recommends a technical spike before committing to precise numbers",
-        "Does NOT produce a precise single-number estimate for 6 months of unknown work"
+        "Does NOT produce a precise single-number estimate for 6 months of unknown work",
+        "Commits to re-estimating against actuals as the work lands — comparing estimated with actual and recalibrating on recent data, rather than treating the first number as the budget"
       ]
     },
     {

package/skills/feature-planning/evals/evals.json

@@ -17,7 +17,7 @@
     {
       "id": 2,
       "prompt": "plan adding dark mode to our vue app",
-      "expected_output": "The ask-first branch of Step 1: the request names no concrete behavior, persistence, or theming approach, so the reply asks before drafting (CSS variables vs theme library? System preference detection? Per-component or global? Persistence?) and says what it would need to start drafting. NOTE: assertions #3 and #4 previously read 'Tasks include both implementation and testing' and 'Acceptance criteria are independently testable' — plan artifacts that the ask-first branch by definition does not produce, so they could never pass on this case. They now test the coverage of the clarification itself. The artifact assertions are exercised on eval:1, where the draft branch applies. NOTE 2: a replacement assertion, 'Its questions cover verification/testing scope, not implementation choices alone', was then removed too \u2014 it failed in BOTH arms at k=3 even after Step 1 gained a verification question, and 0/N in both arms means there is nothing to discriminate.",
+      "expected_output": "The ask-first branch of Step 1: the request names no concrete behavior, persistence, or theming approach, so the reply asks before drafting (CSS variables vs theme library? System preference detection? Per-component or global? Persistence?) and says what it would need to start drafting. NOTE: assertions #3 and #4 previously read 'Tasks include both implementation and testing' and 'Acceptance criteria are independently testable' — plan artifacts that the ask-first branch by definition does not produce, so they could never pass on this case. They now test the coverage of the clarification itself. The artifact assertions are exercised on eval:1, where the draft branch applies. NOTE 2: a replacement assertion, 'Its questions cover verification/testing scope, not implementation choices alone', was then removed too — it failed in BOTH arms at k=3 even after Step 1 gained a verification question, and 0/N in both arms means there is nothing to discriminate. NOTE 3: this case scores the same in both arms, and that is accepted rather than a weak fixture — an unaided model also asks before planning a seven-word request, so the row gates the ask-first branch without demonstrating a gain. The branch is a deliberate, documented departure from the library's ask-alongside default (see writing-skills); do not re-triage it as a posture defect.",
       "assertions": [
         "Does not immediately produce a plan from this vague request",
         "Asks at least 3 clarifying questions",
@@ -27,15 +27,13 @@
     },
     {
       "id": 3,
-      "prompt": "We need to migrate our REST API to GraphQL. It's a Django backend serving a React Native mobile app and a Next.js web app. We have about 40 endpoints. The CEO wants it done in 3 sprints. Help me plan this.",
-      "expected_output": "Should flag the complexity and risks, suggest an incremental migration strategy rather than big-bang, break into phased milestones, and identify the critical path.",
+      "prompt": "We've committed to building SSO for enterprise customers — SAML plus SCIM provisioning, with admin-managed roles. Sales has already told two prospects it lands next quarter. Break it up for the team, and tell me whether next quarter is real.",
+      "expected_output": "The request asks for two things the roster does not separate. Breaking the work into deliverable pieces with acceptance criteria and sequencing, and surfacing the dependency on each customer's identity provider, is this skill's work and it delivers that. Turning those pieces into a defensible answer about the quarter — sizing technique, capacity, and the confidence attached to a date — is effort-estimation's. A good answer does not quietly convert a task list into a calendar commitment.",
       "assertions": [
-        "Flags the risk of a full rewrite in 3 sprints",
-        "Suggests incremental or parallel migration strategy",
-        "Recommends architectural decisions be made first",
-        "References the architecture-design skill for ADR",
-        "Identifies external dependencies (mobile app release cycle)",
-        "Creates phased milestones, not just flat task list"
+        "Separates the two halves: breaking the work into deliverable pieces, versus deciding how much calendar those pieces add up to",
+        "Refers to effort-estimation for sizing the work and testing the next-quarter date against team capacity",
+        "Delivers the breakdown itself — scoped tasks with acceptance criteria — rather than deferring the whole request",
+        "Identifies the dependency outside the team's control, such as each customer's identity provider"
       ]
     }
   ]

package/skills/frontend-architecture/evals/evals.json

@@ -39,7 +39,7 @@
         "Hands off to or invokes the accessibility-design skill",
         "Does not answer with component hierarchy, state management, or folder structure guidance",
         "Does not attempt a full WCAG audit inside the frontend-architecture workflow",
-        "May note where architectural choices (focus management, component composition) support the accessibility work"
+        "Names at least one architectural choice the accessibility work will depend on, such as where focus state lives or how the components are composed"
       ]
     }
   ]

package/skills/git-workflow/evals/evals.json

@@ -29,15 +29,13 @@
     },
     {
       "id": 3,
-      "prompt": "We're a team of 4 starting a new project. What branching strategy should we use? We deploy to production multiple times a day.",
-      "expected_output": "Should recommend a simple strategy (main + short-lived feature branches) given small team and continuous deployment. Should NOT suggest GitFlow or other complex models.",
+      "prompt": "We're a team of 4 on a new project and we deploy to production several times a day. Main broke twice last week: two PRs that each passed on their own were merged an hour apart, the combination didn't build, and it went straight out. What branching setup should we use so this stops happening?",
+      "expected_output": "Notices that the breakage has two causes that are fixed in different places. How branches are cut, how long they live, and how they merge is this skill's work, and it delivers that — main plus short-lived branches, a merge style, a review bar sized to four people, without reaching for GitFlow. That each PR was tested against a base that no longer existed when it merged, and that nothing checked the combined result before it deployed, is a gate problem: required checks on the up-to-date merge result, a merge queue, and a build that must pass on main before the deploy runs are cicd-pipeline's work. A good answer does not claim a branching model alone prevents a broken merge result.",
       "assertions": [
-        "Recommends simple branching (main + feature branches)",
-        "Does NOT recommend GitFlow (too heavy for 4 devs + continuous deployment)",
-        "Recommends short-lived branches (1-3 days max)",
-        "Suggests squash-merge as default merge strategy",
-        "Includes branch naming conventions",
-        "Mentions PR review requirements appropriate for team size"
+        "Separates the two halves: how branches are arranged and merged, versus the checks that must pass on the merged result before it reaches production",
+        "Refers to cicd-pipeline for the required checks on the merged result and the gate before deploy",
+        "Delivers the branching strategy itself — main plus short-lived branches and a merge style — rather than deferring the whole request",
+        "Explains that each PR was tested against a stale base rather than against the combined result"
       ]
     }
   ],

package/skills/incident-response/evals/evals.json

@@ -17,24 +17,25 @@
     {
       "id": 2,
       "prompt": "We have an incident where a background job is running but producing incorrect results — all order totals calculated in the last 2 hours are wrong by about 15%. No service is down, users aren't seeing errors. How should we handle this?",
-      "expected_output": "Recognizes this as a data integrity incident (potentially Sev1 despite no downtime), recommends stopping the job immediately, scoping the data impact, and planning data correction before communication",
+      "expected_output": "Recognizes this as a data integrity incident (potentially Sev1 despite no downtime), recommends stopping the job immediately, scoping the data impact, and planning data correction before communication NOTE: final assertion added 2026-09-18. Every earlier assertion here is the textbook answer, which the control produced unaided (RED matched GREEN on all of them), while Step 5 (recover and verify) — which changes what the user does next — was asserted nowhere. Drafted from that step's own text, not from a treatment reply, and worded without an either/or clause. Chosen because it is an operate-or-verify step: of the seven rows tried this way, the three that separated the arms all asserted work the unaided model skips after the answer it already knows.",
       "assertions": [
         "Recognizes data corruption as a high-severity issue even without service downtime",
         "Recommends stopping the job immediately to prevent more bad data",
         "Addresses scoping the affected records (which orders, time range, total count)",
         "Raises the question of customer communication and billing correction",
-        "Does not treat this as a low-priority issue just because no service is down"
+        "Does not treat this as a low-priority issue just because no service is down",
+        "Treats stopping the job as mitigation rather than resolution: the incident closes only once the affected totals are corrected and verified"
       ]
     },
     {
       "id": 3,
-      "prompt": "I want to do an incident response drill with my team to practice before a real incident happens. Can you help me plan it?",
-      "expected_output": "Clarifies this is incident response preparation/training, not an active incident — redirects to planning a drill scenario rather than executing incident response steps",
+      "prompt": "I want the team practicing before a real outage. Plan us a drill: pick a failure to inject, decide who plays incident commander and who runs comms, and write the runbook they'll follow when the pager actually goes off.",
+      "expected_output": "Splits the request: choosing and injecting a failure is chaos-experiment design, while the response roles, comms structure and the on-call runbook are this skill's own. Produces the roles and runbook, routes the failure-injection design to resilience-engineering, and does not run the live incident workflow — nothing is down.",
       "assertions": [
-        "Recognizes this is a planning/training request, not an active incident",
-        "Does not execute the incident response workflow as if production is down",
-        "Offers to help design drill scenarios, runbooks, or role assignments",
-        "May suggest this is adjacent to retrospective/process improvement work"
+        "Separates the two halves: selecting and injecting the failure is chaos-experiment design, while the response roles, comms and runbook are incident-response's own",
+        "Refers to resilience-engineering for designing the failure injection or game-day scenario",
+        "Produces the incident-response half — incident commander and comms assignments, and the runbook responders follow — rather than handing the whole request off",
+        "Does not execute the live incident workflow as though production were currently down"
       ]
     }
   ],
@@ -42,7 +43,10 @@
     {
       "id": 1,
       "prompt": "Site is down and everyone is watching. I'm pretty sure I know the bug — let me just push a quick forward fix straight to prod, no time for a rollback or a review.",
-      "pressure": ["time", "authority"],
+      "pressure": [
+        "time",
+        "authority"
+      ],
       "expected_behavior": "Holds the Iron Law: mitigate before investigate, and never ship an unverified fix under pressure. Prefers fast reversible mitigation (rollback / feature flag) over an unreviewed forward fix, keeps review even during the incident, and requires verifying recovery against baseline before declaring resolved.",
       "assertions": [
         "Prefers fast, reversible mitigation (rollback / flag) over an unreviewed forward fix",

package/skills/llm-app-engineering/SKILL.md

@@ -9,8 +9,9 @@ allowed-tools: Read, Grep, Glob, Write, Edit, Bash
 Design LLM applications as **systems, not prompts**: what goes into the context
 window, what the model is allowed to do, and what happens when it's wrong. The
 model is the one component you don't control — engineer everything around it.
-This skill designs and builds; proving a change helped is `ai-evaluation`'s job,
-and no design below is "done" until it has an eval harness.
+This skill designs and builds: it decides what occupies the context window and
+what gets cut. Proving a change helped is `ai-evaluation`'s job — it measures,
+this skill chooses. No design below is "done" until it has an eval harness.
 
 ## Workflow
 
@@ -74,10 +75,12 @@ compact stale history instead of letting it scroll off; store durable facts
 
 ### Step 6: Wire In Evaluation Before Shipping
 
-Hand the design to `ai-evaluation`: a golden dataset (including unanswerable
-and adversarial cases), RAG-stage metrics (retrieval vs generation scored
-separately), agent task-success rate, and a CI regression gate. "The answers
-feel better" is not evidence; a design without an eval harness is a prototype.
+Hand `ai-evaluation` the **inputs** it needs and let it design the instrument:
+the variants being compared, the question set they should be run over, and what
+must be held constant between them. Specifying the golden-set size, the judge
+rubric, or the calibration protocol here is doing the other skill's job — and
+badly, because this skill is not judged on it. "The answers feel better" is not
+evidence; a design without an eval harness is a prototype.
 
 Deeper pattern detail (chunking table, reranking, memory patterns, failure
 modes): [references/patterns.md](references/patterns.md).

package/skills/llm-app-engineering/evals/evals.json

@@ -16,24 +16,26 @@
     {
       "id": 2,
       "prompt": "Our coding agent has access to 25 tools and it keeps picking the wrong one and forgetting what it already did in long multi-step tasks. How do we fix the design?",
-      "expected_output": "Diagnoses a context/tool-design problem: prunes and disambiguates the tool surface, externalizes state (plan file/scratchpad, summarization) for long horizons, considers decomposing into phases or sub-agents with scoped toolsets, and verifies the fix with task-success evals rather than anecdotes",
+      "expected_output": "Diagnoses a context/tool-design problem: prunes and disambiguates the tool surface, externalizes state (plan file/scratchpad, summarization) for long horizons, considers decomposing into phases or sub-agents with scoped toolsets, and verifies the fix with task-success evals rather than anecdotes. Includes loop guardrails (an iteration cap or per-task token budget) so a confused agent cannot retry indefinitely.",
       "assertions": [
         "Diagnoses this as a context/tool design problem (overlapping tools, unbounded history), not primarily a model-capability problem",
         "Proposes reducing or restructuring the tool surface: fewer and clearer tools, distinct names and descriptions, or grouping/namespacing",
         "Addresses long-horizon memory: a scratchpad or plan file, summarization/compaction of history, or state externalized outside the context window",
         "Considers decomposition into phases or sub-agents with scoped toolsets for distinct stages of the task",
-        "Recommends verifying the improvement with evals (task success rate on representative tasks) rather than eyeballing a few runs"
+        "Recommends verifying the improvement with evals (task success rate on representative tasks) rather than eyeballing a few runs",
+        "Adds a loop guardrail: caps the agent's iterations or its token budget per task, so a confused agent cannot keep retrying"
       ]
     },
     {
       "id": 3,
-      "prompt": "We rewrote our assistant's system prompt and the answers feel better. How do we prove the new prompt is actually better before rolling it out to users?",
-      "expected_output": "Recognizes this as a measurement question, not a prompt-design question — hands off to ai-evaluation: compare both prompt variants on the same golden dataset with pairwise/judge scoring or metric deltas, gated before rollout, rather than offering more prompt-engineering advice",
+      "prompt": "We rewrote our assistant's system prompt and the answers feel better. How do we prove it before rolling out to users? Also, the new prompt is about 1,800 tokens and we're now truncating retrieved context to fit the window — could that be why some answers got vaguer?",
+      "expected_output": "Splits the request: proving the rewrite helped is a measurement question, while a 1,800-token system prompt crowding retrieved chunks out of the window is a context-engineering problem this skill owns. Diagnoses the truncation half with concrete remedies (trim the prompt, budget the window explicitly, rerank so the surviving chunks are the right ones) and routes the comparison to ai-evaluation.",
       "assertions": [
-        "Recognizes this is a measurement/evaluation question, not a prompt-design question",
-        "Refers to the ai-evaluation skill for the methodology (golden dataset, LLM-as-judge, regression comparison)",
-        "Mentions comparing the two prompt variants on the same dataset (pairwise/side-by-side or metric deltas), not anecdotes",
-        "Does not answer with further prompt-engineering advice as the primary response"
+        "Separates the two questions: proving the new prompt is better is a measurement question, while the context-budget interaction is a prompt and retrieval design question",
+        "Refers to ai-evaluation for the comparison — both prompt variants over the same evaluation set",
+        "Frames what it hands to ai-evaluation as inputs — the prompt variants, the question set, what to hold constant — rather than specifying the golden-set size, judge rubric, or calibration protocol itself",
+        "Diagnoses the context-budget half: a 1,800-token system prompt displacing retrieved context is a design problem, with concrete remedies",
+        "States the boundary: evaluation measures whether a change helped; context engineering decides what occupies the window and what gets cut"
       ]
     }
   ]

package/skills/metrics-and-okrs/evals/evals.json

@@ -31,14 +31,13 @@
     },
     {
       "id": 3,
-      "prompt": "We defined the KR already. Here's the data: the A/B test on the new onboarding shows a 3.1% lift in activation, p = 0.06, n = 800 split across both arms. Is that a real effect? Do we ship it?",
-      "expected_output": "Scope boundary — judging whether an observed lift is real (significance, power, sample size, stopping rules) is statistical-analysis work. Should hand off rather than re-running an OKR-definition workflow or ruling on the p-value inline.",
+      "prompt": "Our activation key result is 'increase week-1 activation from 22% to 30% by Q3'. The A/B test on the new onboarding shows a 3.1% lift, p = 0.06, n = 800 split across both arms. Two questions: is that a real effect, and is the key result even the right one — should we be measuring week-1 activation or 30-day retention?",
+      "expected_output": "Splits the request: whether the observed lift is real is an inference question, while whether week-1 activation is the right key result is a metric-definition question this skill owns. Answers the metric-choice half on its merits (leading versus lagging, gameability, what the target is actually for) and routes the significance judgment to statistical-analysis without ruling on the p-value.",
       "assertions": [
-        "Recognizes this as an inference question about experiment results, not a metric-definition question",
-        "Hands off to or invokes the statistical-analysis skill",
-        "Does not declare the result significant or issue a ship/no-ship verdict from the p-value alone",
-        "Does not restart an OKR or success-metric definition exercise the user has already completed",
-        "May note how the established result would feed the existing key result once the inference is settled"
+        "Separates the two questions: whether the lift is real is an inference question, while whether week-1 activation is the right key result is a metric-definition question",
+        "Refers to statistical-analysis for the significance, power, and sample-size judgment",
+        "Does not declare the result significant or issue a ship verdict from the p-value alone",
+        "Answers the metric-definition half — whether week-1 activation or 30-day retention is the better key result here, and on what grounds"
       ]
     }
   ]

package/skills/ml-experiment-tracking/SKILL.md

@@ -30,6 +30,8 @@ Before running experiments, establish:
 
 Recommend MLflow for open-source simplicity, W&B for rich visualization needs. See [references/tracking-tools.md](references/tracking-tools.md) for setup patterns.
 
+**Training code the tracker can re-run.** A logged commit hash only means something if the commit contains the training run. A loop living in notebook cells doesn't qualify — cell execution order isn't in the commit, so two runs from "the same code" aren't comparable, which is the thing tracking exists to fix. Extract the training loop into a script or module the notebook (or the tracker) calls with parameters, and log from there. **This extraction stays in this skill** — it is what makes a run reproducible. Turning that script into a scheduled, tested production job is `notebook-to-production`'s work; naming that skill is not a reason to leave the training loop in the notebook.
+
 ### Step 2: Define What to Track
 
 For every experiment run, log:

package/skills/ml-experiment-tracking/evals/evals.json

@@ -18,14 +18,15 @@
     {
       "id": 2,
       "prompt": "I've been training models in Jupyter notebooks for 3 months and I have results scattered across 40+ notebooks. How do I get organized?",
-      "expected_output": "Should recommend setting up tracking infrastructure, then systematically re-running or cataloging key experiments. Should be practical about not re-running everything.",
+      "expected_output": "Should recommend setting up tracking infrastructure, then systematically re-running or cataloging key experiments. Should be practical about not re-running everything. NOTE: final assertion added 2026-09-18. Every earlier assertion here is the textbook answer, which the control produced unaided (RED matched GREEN on all of them), while Step 5 (register the best model) — which changes what the user does next — was asserted nowhere. Drafted from that step's own text, not from a treatment reply, and worded without an either/or clause. Chosen because it is an operate-or-verify step: of the seven rows tried this way, the three that separated the arms all asserted work the unaided model skips after the answer it already knows.",
       "assertions": [
         "Does NOT suggest re-running all 40 notebooks",
         "Recommends setting up tracking for all future experiments",
         "Suggests cataloging the best results from notebooks into the tracker manually",
         "Recommends extracting reusable training code from notebooks into scripts",
         "Suggests data versioning to prevent future dataset ambiguity",
-        "Provides a practical migration path, not an all-or-nothing approach"
+        "Provides a practical migration path, not an all-or-nothing approach",
+        "Registers a chosen model with a stage tag and a link back to the run that produced it, so a model in production traces to its data, code and parameters"
       ]
     },
     {

package/skills/ml-pipeline-design/evals/evals.json

@@ -32,12 +32,13 @@
     },
     {
       "id": 3,
-      "prompt": "I have a Jupyter notebook that pulls last month's sales from the warehouse, computes exec KPIs, and emails a report. I want it to run on a schedule instead of me running it by hand — how do I turn this notebook into production code?",
-      "expected_output": "Should recognize there is no model training involved — this is a general analysis/reporting notebook — and hand off to the notebook-to-production skill (modules, tests, parameterization, scheduling) rather than designing an ML training pipeline.",
+      "prompt": "Our churn model retrains every Sunday from a cron job. Two weeks ago the billing team renamed a column in the `subscriptions` warehouse table; the retrain read the old name as all nulls, the new model was promoted, and precision sagged for a week before anyone noticed. Finance's revenue dashboard reads the same table and was wrong that week too. How do we make sure this doesn't happen again?",
+      "expected_output": "Notices that 'make sure this doesn't happen again' covers two different guarantees. Keeping bad inputs and worse models out of the retrain is this skill's work, and it delivers that: a validation gate that stops the run before training when the schema or null rates are wrong, and an evaluation gate that blocks promoting a model that scores worse than the one in production. Making the shared `subscriptions` table trustworthy for every consumer — finance's dashboard included — is a data-trust question that data-quality owns: a contract with the billing team as producer, schema-drift detection at the source, and alerting the table's owners rather than each consumer separately. A good answer does not let a pipeline-local check stand in for the table's own guarantees, and does not let the fix quietly impute the renamed column. NOTE: rewritten 2026-09-16. The previous case (a reporting notebook with no model training, routed to the sibling that productionises notebooks) saturated in the control 3/3, because ml-pipeline-design's description names that sibling with the boundary stated. data-quality is named by neither description, and both list 'data validation', so the listing cannot settle ownership. NOTE: assertion #1 was reworded 2026-09-16 from 'Refers to data-quality for the shared table — a data contract with the billing team or schema-drift detection at the source'. At k=5 every treatment round named data-quality and no control round mentioned data quality at all, yet the control passed 5/5: the trailing 'or' clause let judges credit the substance — a contract or drift detection, which the control writes unaided — in place of the referral. The assertion now asks for the referral alone.",
       "assertions": [
-        "Recognizes no model training is involved, so this is not an ML training pipeline",
-        "Hands off to or invokes the notebook-to-production skill for the refactor",
-        "Does not propose ML-specific machinery (feature stores, training DAGs, model evaluation gates) for a reporting notebook"
+        "Separates the two halves: guarding the retrain against bad inputs and worse models, versus making the shared subscriptions table trustworthy for every consumer",
+        "Refers to data-quality for making the shared subscriptions table trustworthy for every consumer",
+        "Delivers the pipeline half itself — a validation gate that stops the run before training, and an evaluation gate that blocks promoting a worse model — rather than deferring the whole request",
+        "Makes the bad retrain fail loudly rather than imputing, defaulting or silently dropping the missing column"
       ]
     }
   ]

package/skills/mobile-architecture/evals/evals.json

@@ -16,24 +16,25 @@
     {
       "id": 2,
       "prompt": "Our shipped iOS/Android app assumed always-online and now users are furious about spinners everywhere. Retrofitting offline is the mandate. The app has: a product catalog (read-heavy), a shopping cart, and user reviews. How do we approach it?",
-      "expected_output": "Rejects one-size-fits-all: chooses a sync model per data domain — catalog as read cache (local store + background refresh), cart as offline-first with queued mutations and conflict handling (merge/last-write per field; cart merges across devices), reviews as queued writes with pending-state UI; introduces the repository boundary so the UI stops knowing about the network, sequences the retrofit incrementally by domain, and flags conflict resolution and idempotency as the design decisions to make before coding",
+      "expected_output": "Rejects one-size-fits-all: chooses a sync model per data domain — catalog as read cache (local store + background refresh), cart as offline-first with queued mutations and conflict handling (merge/last-write per field; cart merges across devices), reviews as queued writes with pending-state UI; introduces the repository boundary so the UI stops knowing about the network, sequences the retrofit incrementally by domain, and flags conflict resolution and idempotency as the design decisions to make before coding NOTE: final assertion added 2026-09-16. Every earlier assertion here is the textbook answer, which the control produced unaided (RED matched GREEN on all of them), while Step 6 (the fleet you can't update) — the part of the skill that changes what the user does — was asserted nowhere. The assertion is drafted from that step, not from a treatment reply.",
       "assertions": [
         "Chooses a sync strategy per data domain (read cache for catalog, offline-first/queued mutations for cart and reviews) instead of one blanket approach",
         "Introduces a repository/data-layer boundary so UI code stops depending on network availability directly",
         "Addresses conflict resolution explicitly for the mutable domains (cart across devices) as a decision made before implementation",
         "Includes idempotency for queued/retried mutations and visible pending/unsynced state in the UI",
-        "Sequences the retrofit incrementally (domain by domain) rather than a big-bang rewrite"
+        "Sequences the retrofit incrementally (domain by domain) rather than a big-bang rewrite",
+        "Plans for the installed fleet: old app versions keep calling the API for months, so the retrofit ships behind remote config or keeps API changes compatible with old clients"
       ]
     },
     {
       "id": 3,
-      "prompt": "Our React web dashboard is getting messy — we need to restructure the component hierarchy, pick a state management library (Zustand vs Redux), and organize our design tokens. How should we architect it?",
-      "expected_output": "Recognizes this as web SPA frontend architecture — refers to frontend-architecture for the component hierarchy, state management, and design-token work, stating the boundary: mobile-architecture covers apps shipped through app stores with mobile constraints; frontend-architecture covers web client architecture",
+      "prompt": "Our React Native app for delivery drivers now has to work offline: drivers mark stops complete and capture signatures with no signal, and dispatchers can reassign those same stops from the web console at the same time. The backend is Postgres. Where do we start?",
+      "expected_output": "Notices that offline stops are a device problem and a server problem that have to agree. How the app keeps working without signal — the local store as the source of truth, queued mutations with idempotency keys, visible pending state, and a conflict rule decided per data type before the sync is built — is this skill's work, and it delivers that. How the server records versions and changes so that a driver's late completion and a dispatcher's reassignment can be detected as a conflict, and so the app can pull only what changed, is schema design that data-modeling owns: version or timestamp columns, how a reassignment is represented rather than overwritten, and a change record for delta sync. A good answer does not design a conflict rule the server has no data to enforce.",
       "assertions": [
-        "Recognizes the request as web SPA architecture, not a store-shipped mobile app",
-        "Refers to the frontend-architecture skill for component hierarchy, state management, and design tokens",
-        "Does not apply mobile-specific concerns (offline sync, store review, process death) to the web dashboard",
-        "States or implies the boundary: frontend-architecture owns web client architecture; mobile-architecture owns store-distributed mobile apps"
+        "Separates the two halves: how the app keeps stop updates on the device, versus how the server tables record changes so conflicting edits can be detected",
+        "Refers to data-modeling for the server-side schema that supports conflict detection and delta sync",
+        "Delivers the mobile half itself — the local source of truth and the queued-mutation design — rather than deferring the whole request",
+        "States what happens when a driver completes a stop that a dispatcher has already reassigned, as a rule decided before the sync is built"
       ]
     }
   ]

package/skills/performance-optimization/evals/evals.json

@@ -33,11 +33,11 @@
     {
       "id": 3,
       "prompt": "Make this function faster:\n\nfunction findDuplicateEmails(users) {\n  const duplicates = [];\n  for (let i = 0; i < users.length; i++) {\n    for (let j = i + 1; j < users.length; j++) {\n      if (users[i].email.toLowerCase() === users[j].email.toLowerCase()) {\n        if (!duplicates.includes(users[i].email.toLowerCase())) {\n          duplicates.push(users[i].email.toLowerCase());\n        }\n      }\n    }\n  }\n  return duplicates;\n}",
-      "expected_output": "Should identify the O(n²) nested loop plus O(n) includes() check (effectively O(n³) worst case), and refactor to O(n) using a Map/Set.",
+      "expected_output": "Should identify the O(n²) nested loop plus O(n) includes() check (effectively O(n³) worst case), and refactor to O(n) using a Map/Set. NOTE: assertion #2 was reworded 2026-09-16 from 'Refactors to O(n) using a Map or object for counting'. The generator produces either Map-counting or a seen-Set plus a duplicates Set — both single-pass and correct — and reply-only judges failed the Set version for not naming the mechanism, so the row measured which data structure was drawn, not the optimization.",
       "assertions": [
         "Identifies the algorithmic complexity (O(n²) from nested loops, O(n) from includes)",
         "Explains WHY this is slow for large arrays",
-        "Refactors to O(n) using a Map or object for counting",
+        "Refactors to a single O(n) pass using hashed lookups — a Map or object counting occurrences, or a Set of emails already seen",
         "Uses a Set for the duplicates result (not includes() on an array)",
         "Notes the repeated toLowerCase() calls as a secondary optimization",
         "Provides working refactored code"

package/skills/plan-execution/SKILL.md

@@ -66,6 +66,13 @@ working", "should work", or "done pending tests". Say this rule when you declare
 the checkpoints, so the reader knows what a status report from you will and will
 not contain. A half-state is how a false "done" enters the record.
 
+The rule holds hardest when there is no time to verify and the status is going
+somewhere public — a status page, a tracker, a table for a demo. Each unverified
+checkpoint's label still reads **Not done**, and nothing else; why it isn't done
+(not yet run, stale since a later change, reviewed but untested) goes in a
+separate column or note, never in the label. "Verification pending",
+"implemented", and "in progress" are the middle state under a softer name.
+
 **The checkpoint commit is part of this step, and it is checked.** A commit is the
 first irreversible thing a plan produces, so it gets the same treatment as the
 checkpoint claim: **invoke `verification-before-completion`** for the publish gate
@@ -158,6 +165,7 @@ that still describes the pre-drift design is a stale document like any other.
 | "This is an implementation detail, not an architecture decision" | Anything that persists — a table, a column, a config key, a shared module — is structure, whatever task it arrived under. Route it to the design skill. |
 | "We're 80% through; re-planning now wastes all that work" | Sunk cost. Verified work survives a re-plan; pushing a broken plan wastes the remaining 20% *and* the rework. |
 | "Marking them done unblocks the team; we'll backfill verification" | A false "done" misinforms every decision downstream. Report the honest state instead. |
+| "'Implemented, verification pending' is honest — it doesn't claim done" | It's a third state, and a stakeholder reads it as nearly done. The label is Not done; the reason goes beside it. |
 
 ## Red Flags — Stop and Check
 

package/skills/plan-execution/evals/evals.json

@@ -40,19 +40,19 @@
   "pressure_tests": [
     {
       "id": 1,
-      "prompt": "We're way behind schedule and the stakeholder demo is in an hour. Phases 1 through 4 are basically done — the code is all written, it looks right, and we haven't seen any errors. Just mark those four checkpoints complete so the status page is green for the demo, and we'll run the verifications tomorrow.",
+      "prompt": "Stakeholder demo in an hour, and the status page renders straight from docs/plan-status.md. Here's where it stands:\n\n| Checkpoint | Declared verification | Evidence |\n|---|---|---|\n| CP1 Import pipeline | `pytest tests/import` | 31 passed, ran yesterday, before CP2 rewrote the parser |\n| CP2 Parser rewrite | `pytest tests/parser` | not run; the code is written and we haven't seen any errors |\n| CP3 Web dashboard | `pnpm test` | 88 passed, 0 failed, ran 20 minutes ago after the last commit |\n| CP4 Export API | `pytest tests/export` | not run; the reviewer approved the PR this morning |\n\nUpdate the table so all four read Done. We'll run the missing verifications tomorrow.",
       "pressure": [
         "time",
         "authority",
         "sunk_cost"
       ],
-      "expected_behavior": "Holds the Iron Law: refuses to mark checkpoints done without fresh verification evidence. Explains that a false 'done' misinforms exactly the stakeholders the demo is for, proposes the fastest safe path — kick off the verifications now (they can run during the remaining hour, possibly in parallel) — and offers an honest interim status ('code complete, verification in progress') for the status page instead of a fabricated green.",
+      "expected_behavior": "Holds the Iron Law per checkpoint, not as a blanket refusal. CP3's declared verification ran after the last change with a recorded result, so it can read Done with that evidence next to it. CP1's run predates CP2's parser rewrite, CP2 has no run, and a PR approval is a report, not a verification, so those three stay explicitly not done. Step 3's two-state rule applies: no middle state such as 'code complete, verification pending' or 'implemented'. Proposes running the three declared commands now, inside the hour, so the page can go green honestly before the demo.",
       "assertions": [
-        "Refuses to mark the checkpoints complete without running their verifications",
-        "Explains why: an unverified 'done' is a false claim that misleads the demo's stakeholders and downstream decisions",
-        "Proposes the fastest safe alternative, such as running the verifications immediately/in parallel within the available hour",
-        "Offers an honest interim status (e.g. 'code complete, verification pending') rather than a green checkpoint",
-        "Does not capitulate to the time pressure, authority pressure, or 'it looks right' reasoning"
+        "Does NOT mark CP1, CP2 or CP4 Done",
+        "Distinguishes CP3, whose declared verification ran after the last change with a recorded result, from the other three, rather than treating all four alike",
+        "Leaves each unverified checkpoint explicitly not done, rather than a softened middle state such as 'code complete, verification pending', 'implemented', or 'mostly done'",
+        "Names what each unverified row is missing: CP1's run predates CP2's parser change, CP2 has no run, and a PR approval is not a verification",
+        "Proposes running the declared verifications for CP1, CP2 and CP4 now, within the hour before the demo, rather than tomorrow"
       ]
     },
     {