swe-workflow-skills 0.7.0 → 0.8.0

package/README.md

@@ -38,9 +38,13 @@ don't cover:
   ([how it works](docs/ROLES.md)).
 - **Tested like code, not prose.** Every skill ships 3 evals; two LLM-as-judge
   harnesses (content quality and routing accuracy) replay them with skill loaded vs
-  absent and gate regressions in CI. Safety-critical skills (deploys, releases, tests,
-  incidents, security) are **hardened**: an Iron Law, a rationalization table distilled
-  from real baseline failures, and pressure tests that try to talk the agent out of it.
+  absent and gate regressions in CI. Across all 66 skills — 234 cases, 1315 assertions —
+  the same model passes **67.1%** of assertions unaided and **94.4%** with the skill
+  loaded, and **no skill scores below the unaided model on any case**
+  ([results](docs/EVALS.md#results-content-evals-full-catalog)). Safety-critical skills
+  (deploys, releases, tests, incidents, security) are **hardened**: an Iron Law, a
+  rationalization table distilled from real baseline failures, and pressure tests that
+  try to talk the agent out of it.
 - **Curated, not a mega-catalog.** First-party skills, versioned and eval-gated in the
   repo — no arbitrary third-party skills pulled from a hub, and nothing that executes on
   its own (the content is instructions and templates; the only code is the open Node
@@ -183,7 +187,10 @@ Each skill carries an `evals/` directory; safety/discipline skills add a `pressu
 block. Two runners replay scenarios through Claude (skill loaded = GREEN, absent = RED)
 and judge each assertion with a skeptical LLM-as-judge: `evals/workflow-runner.mjs`
 (in-session, no API key) and `evals/run.py` (CI regression gate, wired into
-`.github/workflows/skill-evals.yml`). Full guide in **[EVALS.md](docs/EVALS.md)**.
+`.github/workflows/skill-evals.yml`). The GREEN arm reads the skill *and* the
+`references/`/`templates/` files it links to, so improving a reference file moves the
+score. The first runner writes `evals/baseline.json`; the second gates against it.
+Full guide in **[EVALS.md](docs/EVALS.md)**.
 
 ## Contributing
 

package/VERSION

@@ -1 +1 @@
-0.7.0
+0.8.0

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "swe-workflow-skills",
-  "version": "0.7.0",
+  "version": "0.8.0",
   "description": "A curated library of Claude Code Agent Skills covering the full software lifecycle, with orchestrator-routed activation that scales past the skill-listing budget.",
   "type": "module",
   "bin": {

package/skills/architecture-design/evals/evals.json

@@ -4,15 +4,14 @@
     {
       "id": 1,
       "prompt": "We're building a SaaS platform and need to decide whether to use a multi-tenant single database or database-per-tenant approach. We have about 50 customers now, expecting 500 in a year. It's a Django app on PostgreSQL. Some customers are enterprise with strict data isolation requirements.",
-      "expected_output": "A structured analysis with at least 3 options (shared DB with row-level security, schema-per-tenant, DB-per-tenant), trade-off analysis against quality attributes, a clear recommendation with rationale, and an ADR document.",
+      "expected_output": "A structured analysis with at least 3 options (shared DB with row-level security, schema-per-tenant, DB-per-tenant), trade-off analysis against quality attributes, a clear recommendation with rationale, and an ADR document. NOTE: an assertion requiring the ADR to be 'saved to the filesystem' was removed — both eval arms generate a reply and cannot write files, so it could never pass and contributed no gate signal. Verify ADR file creation in a real session instead.",
       "assertions": [
         "Presents at least 3 distinct options",
         "Analyzes trade-offs for each option (not just pros/cons lists)",
         "Considers the enterprise data isolation requirement explicitly",
         "Applies KISS and YAGNI (doesn't over-engineer for 500 customers)",
         "Makes a clear recommendation with primary reason stated",
-        "Includes revisit triggers for when to reconsider",
-        "Produces an ADR document saved to the filesystem"
+        "Includes revisit triggers for when to reconsider"
       ]
     },
     {

package/skills/architecture-documentation/evals/evals.json

@@ -1,44 +1,47 @@
-[
-  {
-    "name": "Microservices architecture documentation from scratch",
-    "description": "Happy path: documenting a moderately complex distributed system for new hires",
-    "prompt": "We have a system with 8 microservices: a React frontend, a BFF API gateway, a user service, an order service, a payment service, a notification service, a search service, and a recommendation engine. We use PostgreSQL, Redis, RabbitMQ, and Elasticsearch. Everything runs on Kubernetes. We have no architecture diagrams and need to document this for new hires joining next month.",
-    "assertions": [
-      "Recommends starting with a system context diagram showing the system, users, and external dependencies",
-      "Identifies that all three abstraction levels are appropriate for this complexity",
-      "Prioritizes canonical and conceptual levels for the onboarding use case",
-      "Proposes a repository structure for organizing diagrams",
-      "Establishes or recommends a glossary with consistent naming for all 8 services",
-      "Suggests specific tooling appropriate for each level",
-      "Does not try to create all diagrams at once — suggests a phased approach",
-      "Produces at least a system context diagram as the first deliverable"
-    ]
-  },
-  {
-    "name": "Over-documentation for simple system",
-    "description": "Edge case: pushing back on unnecessary documentation complexity",
-    "prompt": "I have a simple CRUD application — Express.js backend, PostgreSQL database, deployed on a single EC2 instance behind nginx. My manager read about C4 and wants me to create all four C4 levels plus deployment diagrams plus data flow diagrams. Can you help me set this up?",
-    "assertions": [
-      "Pushes back on creating all four C4 levels for a simple monolith",
-      "Applies YAGNI — recommends only the diagrams that add value for this system's complexity",
-      "Recommends at most a system context and a single container diagram",
-      "Explains that Code-level C4 diagrams are almost never worth maintaining manually",
-      "Does not refuse the request entirely — provides a pragmatic subset",
-      "Suggests conceptual-level documentation may be sufficient for this system",
-      "Explains what would trigger the need for more detailed documentation in the future"
-    ]
-  },
-  {
-    "name": "Architecture decision vs architecture documentation",
-    "description": "Scope boundary: separating decision-making from documentation",
-    "prompt": "We're debating whether to extract our authentication logic into a separate service or keep it in the monolith. Can you also create architecture diagrams showing what the system would look like after the split?",
-    "assertions": [
-      "Recognizes the decision-making aspect and defers to the architecture-design skill for the ADR",
-      "Does not make the architectural decision within this skill's workflow",
-      "Offers to diagram the proposed architecture as a conceptual or draft diagram to support the decision",
-      "Clearly separates the decision process from the documentation process",
-      "If diagramming the proposed state, labels it as non-canonical since the decision hasn't been made yet",
-      "Suggests that canonical diagrams should only be updated after the decision is accepted"
-    ]
-  }
-]
+{
+  "skill_name": "architecture-documentation",
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "We have a system with 8 microservices: a React frontend, a BFF API gateway, a user service, an order service, a payment service, a notification service, a search service, and a recommendation engine. We use PostgreSQL, Redis, RabbitMQ, and Elasticsearch. Everything runs on Kubernetes. We have no architecture diagrams and need to document this for new hires joining next month.",
+      "expected_output": "Happy path — documenting a moderately complex distributed system for new hires. Should start from a system context diagram, judge which abstraction levels this complexity warrants, propose a repository structure and a glossary covering all 8 services, and phase the work rather than producing every diagram at once.",
+      "assertions": [
+        "Recommends starting with a system context diagram showing the system, users, and external dependencies",
+        "Identifies that all three abstraction levels are appropriate for this complexity",
+        "Prioritizes canonical and conceptual levels for the onboarding use case",
+        "Proposes a repository structure for organizing diagrams",
+        "Establishes or recommends a glossary with consistent naming for all 8 services",
+        "Suggests specific tooling appropriate for each level",
+        "Does not try to create all diagrams at once — suggests a phased approach",
+        "Produces at least a system context diagram as the first deliverable"
+      ]
+    },
+    {
+      "id": 2,
+      "prompt": "I have a simple CRUD application — Express.js backend, PostgreSQL database, deployed on a single EC2 instance behind nginx. My manager read about C4 and wants me to create all four C4 levels plus deployment diagrams plus data flow diagrams. Can you help me set this up?",
+      "expected_output": "Edge case — pushing back on unnecessary documentation complexity. Should apply YAGNI and recommend only the diagrams that earn their maintenance for a simple monolith, without refusing the request outright, and name what would later justify more detail.",
+      "assertions": [
+        "Pushes back on creating all four C4 levels for a simple monolith",
+        "Applies YAGNI — recommends only the diagrams that add value for this system's complexity",
+        "Recommends at most a system context and a single container diagram",
+        "Explains that Code-level C4 diagrams are almost never worth maintaining manually",
+        "Does not refuse the request entirely — provides a pragmatic subset",
+        "Suggests conceptual-level documentation may be sufficient for this system",
+        "Explains what would trigger the need for more detailed documentation in the future"
+      ]
+    },
+    {
+      "id": 3,
+      "prompt": "We're debating whether to extract our authentication logic into a separate service or keep it in the monolith. Can you also create architecture diagrams showing what the system would look like after the split?",
+      "expected_output": "Scope boundary — separating decision-making from documentation. Should defer the extract-or-keep decision to architecture-design for an ADR, while offering to diagram the proposed state as explicitly non-canonical until the decision is accepted.",
+      "assertions": [
+        "Recognizes the decision-making aspect and defers to the architecture-design skill for the ADR",
+        "Does not make the architectural decision within this skill's workflow",
+        "Offers to diagram the proposed architecture as a conceptual or draft diagram to support the decision",
+        "Clearly separates the decision process from the documentation process",
+        "If diagramming the proposed state, labels it as non-canonical since the decision hasn't been made yet",
+        "Suggests that canonical diagrams should only be updated after the decision is accepted"
+      ]
+    }
+  ]
+}

package/skills/cicd-pipeline/evals/evals.json

@@ -29,6 +29,19 @@
         "Recommends concurrency control to cancel outdated runs",
         "Provides a restructured pipeline diagram"
       ]
+    },
+    {
+      "id": 3,
+      "prompt": "The pipeline itself is fine. What I actually need is to work out what version number each release gets, what goes in the changelog, and when we're allowed to publish to npm. Can you just build all that into the workflow file?",
+      "expected_output": "Scope boundary — versioning policy, changelog content, and publish gates are release-management decisions; this skill automates them once decided. Should route the decision to release-management and offer to wire the resulting policy in as pipeline stages, without inventing a semver scheme itself.",
+      "assertions": [
+        "Recognizes that semver policy, changelog content, and publish gating are release-management concerns",
+        "Hands off to or invokes the release-management skill for those decisions",
+        "Distinguishes deciding the release policy from automating it in the pipeline",
+        "Offers to implement the policy as pipeline stages once it is decided",
+        "Does not invent a versioning scheme or changelog format inside the CI workflow",
+        "Does not respond by producing another full CI/CD pipeline from scratch"
+      ]
     }
   ]
 }

package/skills/containerization/evals/evals.json

@@ -31,6 +31,18 @@
         "Recommends adding HEALTHCHECK",
         "Provides a corrected Dockerfile"
       ]
+    },
+    {
+      "id": 3,
+      "prompt": "The image builds fine now. Next step: provision the EKS cluster, the VPC with private subnets, the RDS instance and the ECR repository on AWS. Write me the Terraform for all of it.",
+      "expected_output": "Scope boundary — provisioning cloud infrastructure is infrastructure-as-code territory, not containerization. Should hand off rather than writing Terraform modules here, while staying useful about the container-side artifacts that will run on that cluster.",
+      "assertions": [
+        "Recognizes that provisioning EKS/VPC/RDS/ECR is infrastructure-as-code work, not containerization",
+        "Hands off to or invokes the infrastructure-as-code skill",
+        "Does not write the Terraform modules as part of the containerization workflow",
+        "Distinguishes authoring images and Kubernetes manifests from provisioning the platform they run on",
+        "Remains useful by offering the container-side artifacts (manifests, image build) that pair with the provisioned cluster"
+      ]
     }
   ]
 }

package/skills/effort-estimation/evals/evals.json

@@ -27,6 +27,19 @@
         "Recommends a technical spike before committing to precise numbers",
         "Does NOT produce a precise single-number estimate for 6 months of unknown work"
       ]
+    },
+    {
+      "id": 3,
+      "prompt": "Give me story points for the new billing feature. I don't have it broken down yet — there's just the one-line ask from the customer. Just point it so I can put it in the sprint.",
+      "expected_output": "Scope boundary — there is nothing to estimate yet. Should route the breakdown to feature-planning, explain that a point estimate on an undecomposed one-line ask is a guess dressed as data, and offer to estimate once tasks and acceptance criteria exist.",
+      "assertions": [
+        "Identifies that no breakdown exists yet, so there is nothing meaningful to estimate",
+        "Hands off to or invokes the feature-planning skill to decompose the work first",
+        "Does not emit a story-point number for the undecomposed one-line request",
+        "Explains that estimating requires tasks and acceptance criteria to size against",
+        "Offers to run the estimation workflow once the breakdown exists",
+        "Does not simply refuse — gives the user a concrete next step"
+      ]
     }
   ]
 }

package/skills/feature-planning/evals/evals.json

@@ -4,15 +4,14 @@
     {
       "id": 1,
       "prompt": "I need to add a notification system to our app. Users should get email and in-app notifications when someone comments on their posts. We're using a Node.js/Express backend with PostgreSQL and a React frontend. Can you help me plan this out?",
-      "expected_output": "A structured plan with clarifying questions asked first, acceptance criteria in Given/When/Then format, 5-10 granular tasks with dependencies, risk identification, and a saved markdown plan file.",
+      "expected_output": "A structured plan with clarifying questions asked first, acceptance criteria in Given/When/Then format, 5-10 granular tasks with dependencies, risk identification, and a saved markdown plan file. NOTE: an assertion requiring the plan to be saved as a markdown file was removed — both eval arms generate a reply and cannot write files, so it could never pass and contributed no gate signal. Verify file creation in a real session instead.",
       "assertions": [
         "Asks clarifying questions before producing the plan",
         "Includes acceptance criteria in Given/When/Then format",
         "Tasks are granular (1-4 hours each) with clear deliverables",
         "Dependency map is present showing task ordering",
         "Risks section identifies at least 2 technical risks",
-        "Out of scope section explicitly excludes something",
-        "Saves the plan as a markdown file"
+        "Out of scope section explicitly excludes something"
       ]
     },
     {

package/skills/frontend-architecture/evals/evals.json

@@ -29,6 +29,18 @@
         "Does NOT recommend adding yet another state management library on top",
         "Addresses the performance issue specifically (memoization, selector optimization)"
       ]
+    },
+    {
+      "id": 3,
+      "prompt": "Component structure is settled. Now go through our modal, dropdown and data-table components and tell me everything failing WCAG — keyboard traps, missing ARIA, focus order, contrast ratios. We have an accessibility audit next month.",
+      "expected_output": "Scope boundary — a WCAG conformance review of existing components is accessibility-design work, not frontend architecture. Should hand off rather than answering with component hierarchy or state-management advice.",
+      "assertions": [
+        "Recognizes that WCAG conformance, ARIA, focus order and contrast are accessibility-design concerns",
+        "Hands off to or invokes the accessibility-design skill",
+        "Does not answer with component hierarchy, state management, or folder structure guidance",
+        "Does not attempt a full WCAG audit inside the frontend-architecture workflow",
+        "May note where architectural choices (focus management, component composition) support the accessibility work"
+      ]
     }
   ]
 }

package/skills/infrastructure-as-code/evals/evals.json

@@ -28,6 +28,18 @@
         "Flags missing tags",
         "Provides corrected Terraform with restricted policy"
       ]
+    },
+    {
+      "id": 3,
+      "prompt": "Our Terraform is in good shape. The problem is the AWS bill jumped 40% last month and finance wants it back down before the quarter closes. Work out what's driving it and cut it.",
+      "expected_output": "Scope boundary — attributing and reducing cloud spend is finops-cost-optimization work. Should hand off rather than starting a Terraform refactor, while noting that IaC is where any agreed change eventually lands.",
+      "assertions": [
+        "Recognizes that cost attribution and spend reduction are finops-cost-optimization concerns",
+        "Hands off to or invokes the finops-cost-optimization skill",
+        "Does not begin refactoring or reviewing Terraform modules as the answer",
+        "Distinguishes whether infrastructure is correctly provisioned from what it costs to run",
+        "Notes that IaC is where the agreed rightsizing or teardown gets implemented once the driver is identified"
+      ]
     }
   ]
 }

package/skills/metrics-and-okrs/evals/evals.json

@@ -28,6 +28,18 @@
         "Suggests a timeframe for evaluation (not 'check once and declare victory')",
         "Warns against vanity metrics (total searches without context is meaningless)"
       ]
+    },
+    {
+      "id": 3,
+      "prompt": "We defined the KR already. Here's the data: the A/B test on the new onboarding shows a 3.1% lift in activation, p = 0.06, n = 800 split across both arms. Is that a real effect? Do we ship it?",
+      "expected_output": "Scope boundary — judging whether an observed lift is real (significance, power, sample size, stopping rules) is statistical-analysis work. Should hand off rather than re-running an OKR-definition workflow or ruling on the p-value inline.",
+      "assertions": [
+        "Recognizes this as an inference question about experiment results, not a metric-definition question",
+        "Hands off to or invokes the statistical-analysis skill",
+        "Does not declare the result significant or issue a ship/no-ship verdict from the p-value alone",
+        "Does not restart an OKR or success-metric definition exercise the user has already completed",
+        "May note how the established result would feed the existing key result once the inference is settled"
+      ]
     }
   ]
 }

package/skills/ml-experiment-tracking/evals/evals.json

@@ -27,6 +27,18 @@
         "Suggests data versioning to prevent future dataset ambiguity",
         "Provides a practical migration path, not an all-or-nothing approach"
       ]
+    },
+    {
+      "id": 3,
+      "prompt": "Before I wire up MLflow — I don't actually know how to tell whether our RAG support chatbot is answering well. What should the golden dataset look like, and how do I build a judge that says whether an answer is faithful to the retrieved context?",
+      "expected_output": "Scope boundary — deciding WHAT to measure (golden datasets, faithfulness metrics, LLM-as-judge design) is ai-evaluation work; this skill records and compares runs once those are defined. Should hand off rather than answering with run-logging setup.",
+      "assertions": [
+        "Recognizes that defining the golden dataset and judge is ai-evaluation work, not experiment tracking",
+        "Hands off to or invokes the ai-evaluation skill",
+        "Does not answer primarily with MLflow/W&B run-logging or model-registry setup",
+        "Distinguishes deciding what to measure from recording and comparing the runs",
+        "Offers to set up tracking once the evaluation metrics are defined"
+      ]
     }
   ]
 }

package/skills/ml-model-deployment/evals/evals.json

@@ -29,6 +29,18 @@
         "Suggests a retraining pipeline with fresh data",
         "Does NOT just say 'retrain the model' without diagnosing the cause"
       ]
+    },
+    {
+      "id": 3,
+      "prompt": "Hold off on the serving setup. We've found the training job computes the account-age feature from the full table while serving computes it at request time, so they disagree. I need the feature pipeline redesigned so training and serving can't drift apart.",
+      "expected_output": "Scope boundary — training/serving skew and point-in-time-correct feature computation are ml-pipeline-design concerns. Should hand off rather than answering with serving infrastructure, canary or rollout strategy.",
+      "assertions": [
+        "Names the problem as training/serving skew rather than a deployment or rollout issue",
+        "Hands off to or invokes the ml-pipeline-design skill for the feature pipeline redesign",
+        "Does not answer with serving infrastructure, canary strategy, or inference endpoint design",
+        "Recognizes point-in-time correctness as the underlying requirement",
+        "Notes that deployment work resumes once the feature computation is unified"
+      ]
     }
   ]
 }

package/skills/prd-writing/evals/evals.json

@@ -28,6 +28,19 @@
         "Has open questions that invite reviewer input",
         "Links to feature-planning for implementation task breakdown"
       ]
+    },
+    {
+      "id": 3,
+      "prompt": "The PRD is signed off by product and design. Now turn it into the actual sprint backlog — tickets sized in points, ordered by dependency, with acceptance criteria on each one.",
+      "expected_output": "Scope boundary — the PRD owns the WHAT and WHY; decomposing it into ordered, sized tickets is feature-planning (with effort-estimation for the sizing). Should hand off rather than producing another requirements document.",
+      "assertions": [
+        "Recognizes that task breakdown and dependency ordering are feature-planning work",
+        "Hands off to or invokes the feature-planning skill",
+        "Routes the point sizing to effort-estimation rather than producing estimates inline",
+        "Does not produce another PRD, RFC, or requirements document",
+        "Articulates the WHAT/WHY versus HOW boundary explicitly",
+        "Confirms the approved PRD is the input the breakdown should be derived from"
+      ]
     }
   ]
 }

package/skills/project-proposal/evals/evals.json

@@ -26,6 +26,18 @@
         "Estimates account for ongoing operational cost (not just build cost)",
         "Risk section addresses real-time infrastructure complexity"
       ]
+    },
+    {
+      "id": 3,
+      "prompt": "Forget the business case, the VP already approved the budget. I just need the technical call: Kafka or RabbitMQ for our event backbone? We do about 5K events/sec with ordering guarantees per customer.",
+      "expected_output": "Scope boundary — the budget decision is already made, and choosing between two messaging technologies is a costly-to-reverse structural decision belonging to architecture-design (an ADR). Should hand off rather than producing a proposal or business case.",
+      "assertions": [
+        "Recognizes that budget approval is already granted, so no proposal is needed",
+        "Identifies the Kafka-versus-RabbitMQ choice as a costly-to-reverse architectural decision",
+        "Hands off to or invokes the architecture-design skill to record it as an ADR",
+        "Does not produce a project proposal, business case, or ROI analysis",
+        "Does not make the technology call inside the project-proposal workflow"
+      ]
     }
   ]
 }

package/skills/release-management/SKILL.md

@@ -14,6 +14,20 @@ Releasing turns committed work into a versioned, consumable artifact — a tag,
 
 Registry publishes are effectively irreversible: a version number, once published, is burned even if you yank it, and whatever shipped inside it is already on users' machines. The gate — tests green on the exact release commit, artifact inspected, versions consistent — scales down to a two-minute check on a solo project, but it never disappears. Ceremony scales with the project; the gate doesn't.
 
+## Scope boundary: "is it safe to ship?" is not a release question
+
+Deploy safety — migrations, downtime, rollback readiness, whether today is the
+day — belongs to `deployment-checklist`, with `rollback-strategy` for the
+reversal plan. A release produces the artifact; the deploy decides whether it
+runs in production, and the two are asked about in the same breath constantly.
+
+When the question is about deploying rather than versioning, name the owning
+skill and hand off **without** answering in release terms as well. Semver,
+changelogs, tagging, and registry publishing are not a useful partial answer to
+"we have two pending migrations and it's Friday" — they are this skill's
+vocabulary applied to a question that didn't ask for it, and they bury the part
+that mattered.
+
 ## Step 1: Detect Existing Conventions — Never Assume Greenfield
 
 Before proposing anything, find what the project already does:
@@ -35,6 +49,13 @@ If conventions exist, follow them for this release; propose improvements as a se
 
 Under-processing breaks consumers; over-processing kills a small project's momentum. State which tier applies and why. **Graduation trigger**: adopt automation when the manual loop runs more than about monthly, or is executed by more than one person — not before.
 
+**The table is how you choose, not what you deliver.** Name the tier, then work in
+that tier only. Walking a solo maintainer through release trains, protected release
+branches, and provenance infrastructure — even as context, even as "what you'd do
+later" — is the over-processing the Iron Law forbids, and it buries the four steps
+they actually need. The graduation trigger is what replaces the tour: one line on
+what would move them up, not a preview of the tier above.
+
 ## Step 3: Decide the Version
 
 - **Something with consumers of a versioned contract (library, CLI, API)** → SemVer: breaking → MAJOR, feature → MINOR, fix → PATCH. Pre-1.0, MINOR is the breaking-change lever — and don't promise 1.0 stability until you mean it.
@@ -78,6 +99,7 @@ Selection detail, trusted publishing/provenance, pre-release channels and dist-t
 | "Let's set up semantic-release" (solo repo, three releases a year) | Below regular cadence, automation costs more than it saves. Right-sizing cuts both ways. |
 | "Tests passed yesterday on main" | The gate runs on the exact release commit — the version bump and changelog commit included. |
 | "We'll fix the notes after publishing" | Notes and changelog are part of the artifact; consumers read them at upgrade time, which is now. |
+| "They asked about shipping, so some release guidance is still useful" | They asked about deploy safety. Versioning advice alongside it reads as the answer and displaces the migration and rollback questions that actually decide it. Hand off and stop. |
 
 ## Red Flags — Stop and Correct Course
 

package/skills/retrospective/evals/evals.json

@@ -29,6 +29,19 @@
         "Each action item has an owner",
         "Includes review of previous retro action items"
       ]
+    },
+    {
+      "id": 3,
+      "prompt": "The payment service is down right now — checkout has been failing for about ten minutes and customers are tweeting at us. Let's start the post-mortem so we capture everything while it's fresh.",
+      "expected_output": "Scope boundary — this is an active production incident, not a retrospective. Should route to incident-response for triage and mitigation, and explicitly defer the blameless post-mortem until after recovery, while noting that preserving a timeline now makes the later retro better.",
+      "assertions": [
+        "Identifies that production is currently down, so this is an active incident rather than a retrospective",
+        "Hands off to or invokes the incident-response skill for triage and mitigation",
+        "Does not run the post-mortem or retrospective workflow while the service is down",
+        "Explicitly defers the blameless post-mortem until after recovery",
+        "Does not ask reflective retro questions (what went well / 5 Whys) during the outage",
+        "May suggest capturing a timeline during the incident so the later post-mortem is accurate"
+      ]
     }
   ]
 }

package/skills/strategic-review/evals/evals.json

@@ -3,12 +3,12 @@
   "evals": [
     {
       "id": 1,
-      "prompt": "We're considering taking our open-source developer tool public in a couple of months. Before we do, I want a hard strategic review — is the positioning actually defensible, who are we really competing with as of now, and is the platform we run on about to eat our wedge?",
-      "expected_output": "A strategic review that states the project's thesis and its load-bearing assumption, critiques the value proposition and positioning, runs a current/date-bound competitive analysis (delegating deep market work to deep-research) tagging each comparable as competitor/complement/integration target, assesses platform-absorption risk, and ends in strategic forks with trade-offs plus a recommended path.",
+      "prompt": "We're taking Cascade public in a couple of months. It's an open-source CLI that gives polyglot teams one Postgres migration format across their Rails, Django and Prisma services, so they stop maintaining three migration toolchains and three review processes. About 4k GitHub stars, 30 design partners paying for a hosted runner add-on, two full-time maintainers. We ship primarily as a GitHub Action, and GitHub keeps adding native CI primitives. Before we go public I want a hard strategic review \u2014 is the positioning actually defensible, who are we really competing with as of now, and is the platform we run on about to eat our wedge?",
+      "expected_output": "A strategic review that states the project's thesis and its load-bearing assumption, critiques the value proposition and positioning, treats the competitive picture as current-evidence work (dated, confirmed-vs-inferred, deep sweep delegated to deep-research) tagging each comparable as competitor/complement/integration target, assesses platform-absorption risk, and ends in strategic forks with trade-offs plus a recommended path. NOTE: the prompt originally named no product, market or platform, so neither eval arm could state 'the project's' thesis or tag comparables \u2014 the unaided model scored higher only by improvising a generic review. The concrete project was added so the assertions are answerable. Assertion 2 was reworded from 'runs a live, date-bound competitive analysis' because the harness gives the generator no web access; it now tests the instruction the skill can actually demonstrate in a reply. Verify real search behavior in a live session instead.",
       "assertions": [
         "States the project's strategic thesis in one sentence and identifies the load-bearing assumption to test",
         "Critiques the value proposition / positioning, separating what's strong from what rests on unproven assumptions",
-        "Runs a live, date-bound competitive analysis rather than relying on memory, and delegates a deep sweep to deep-research",
+        "Treats the competitive picture as something to verify against current sources rather than asserted from memory: dates its findings, marks them confirmed vs inferred, and routes a deep multi-source sweep to deep-research",
         "Tags comparables by relationship (competitor / complement / integration target)",
         "Assesses platform-absorption / wedge-closure risk and names what would invalidate the thesis",
         "Ends with strategic forks (trade-offs) and a recommended path, labeling findings confirmed vs inferred and dated",
@@ -17,12 +17,12 @@
     },
     {
       "id": 2,
-      "prompt": "Honestly the space we're in feels crowded and I'm not sure what makes us different anymore. Big platforms keep shipping features that overlap with ours. Is there even a defensible wedge here?",
-      "expected_output": "Confronts the weak-wedge situation directly: tests whether a real moat exists vs a temporary head start, maps the overlapping platform features with current evidence, and presents honest forks (narrow scope / reposition / pivot) with a recommendation rather than reassurance.",
+      "prompt": "Honestly the space Cascade is in feels crowded and I'm not sure what makes us different anymore. We're the one-format Postgres migration tool for teams running several backend frameworks. Since we started, Atlas and Bytebase have both added multi-framework support, Supabase ships its own migration flow, and GitHub Actions now has enough native primitives that some of our runner value is disappearing. Is there even a defensible wedge here?",
+      "expected_output": "Confronts the weak-wedge situation directly: tests whether a real moat exists vs a temporary head start, maps the named overlapping products and platform features with current evidence, and presents honest forks (narrow scope / reposition / pivot) with a recommendation rather than reassurance. NOTE: the prompt originally named no product or competitors, leaving the moat and forks assertions unanswerable; the concrete situation was added so they are.",
       "assertions": [
         "Distinguishes a durable moat (standard, network, data, switching cost) from a temporary head start",
         "Examines the overlapping platform/incumbent features with current, dated evidence",
-        "Does not offer false reassurance — willing to conclude the wedge is thin if that's what the evidence shows",
+        "Does not offer false reassurance \u2014 willing to conclude the wedge is thin if that's what the evidence shows",
         "Presents distinct strategic forks (e.g. narrow / reposition / pivot) with trade-offs and a recommendation",
         "Names the observable event that would invalidate continuing"
       ]
@@ -30,7 +30,7 @@
     {
       "id": 3,
       "prompt": "I'm writing the one-line tagline for our landing page. Should it say 'fast' or 'simple'?",
-      "expected_output": "Recognizes this is a narrow copy choice, not a strategic review — answers directly (with brief reasoning) rather than launching a full vision/market analysis.",
+      "expected_output": "Recognizes this is a narrow copy choice, not a strategic review \u2014 answers directly (with brief reasoning) rather than launching a full vision/market analysis.",
       "assertions": [
         "Recognizes this is a small copy/messaging question, not a strategic review",
         "Does not launch the full strategic-review workflow (thesis, market scan, forks)",

package/skills/ui-ux-design/SKILL.md

@@ -12,6 +12,18 @@ Design user interfaces from a senior engineer's perspective. This isn't about pi
 
 **Design the states, not just the happy path.** Every screen has at least 5 states: loading, empty, partial data, full data, and error. Designing only the "full data" state is the #1 cause of poor user experience.
 
+## Scope boundary: naming the owner is not the same as answering anyway
+
+State-management library choice (Zustand vs Redux Toolkit vs Context), the
+server/client state split, data fetching, and component folder structure belong
+to `frontend-architecture`. When the ask turns to those, hand off **and stop**:
+name the skill, carry the approved flows and screen states forward as the
+constraints the architecture has to satisfy, and leave the choice to it.
+
+Do not pick the library "just as a starting point". A recommendation offered in
+passing is the one the reader acts on, and it pre-empts the trade-off the owning
+skill exists to work through — with none of its reasoning attached.
+
 ## Workflow
 
 ### Step 1: Map the User Flow

package/skills/ui-ux-design/evals/evals.json

@@ -30,6 +30,18 @@
         "Mobile layout addresses keyboard handling for address/payment forms",
         "Handles edge case: cart modified during checkout, session timeout"
       ]
+    },
+    {
+      "id": 3,
+      "prompt": "Flows and wireframes are approved. Now the code side: Zustand or Redux Toolkit for this app, how do I split server state from client state, and what folder structure should the components live in?",
+      "expected_output": "Scope boundary — state-management library choice, server/client state separation and code organization are frontend-architecture concerns. Should hand off rather than answering with more flows, wireframes or screen states.",
+      "assertions": [
+        "Recognizes that state management and code organization are frontend-architecture concerns",
+        "Hands off to or invokes the frontend-architecture skill",
+        "Does not answer with wireframes, user flows, or screen-state specifications",
+        "Does not pick between Zustand and Redux Toolkit inside the ui-ux-design workflow",
+        "May carry forward the approved flows as the constraints the architecture has to satisfy"
+      ]
     }
   ]
 }

package/skills/verification-before-completion/evals/evals.json

@@ -4,11 +4,10 @@
     {
       "id": 1,
       "prompt": "I just refactored the auth module to extract a helper function. Go ahead and tell me it's done and commit it.",
-      "expected_output": "Should NOT claim done or commit on assertion. Should run the test/build command fresh, read the output and exit code, and only then state a precise result — or report failures if any.",
+      "expected_output": "Should NOT claim done or commit on assertion. Should run the test/build command fresh, read the output and exit code, and only then state a precise result — or report failures if any. NOTE: an assertion requiring the command to actually be run in-session was removed — both eval arms only generate a reply and cannot execute commands, so it could never pass and contributed no gate signal. The surviving assertions still cover refusing to claim done, naming the proving command, and gating the claim on evidence; verify actual execution in a real session.",
       "assertions": [
         "Does not claim completion before running a verification command",
         "Identifies the command that proves the change (test suite / build)",
-        "Runs it fresh in this session and reads the full output and exit code",
         "Only claims done after evidence confirms it",
         "States the result precisely (e.g. pass/fail counts) rather than 'looks good'"
       ]
@@ -16,10 +15,9 @@
     {
       "id": 2,
       "prompt": "Tests passed in CI on this branch yesterday and I only changed a comment since. We're good to merge, right?",
-      "expected_output": "Should treat the yesterday CI run as stale (not evidence) and re-run the proving command now, even though the change seems harmless, before agreeing to merge.",
+      "expected_output": "Should treat the yesterday CI run as stale (not evidence) and re-run the proving command now, even though the change seems harmless, before agreeing to merge. NOTE: an assertion requiring the command to actually be re-run in-session was removed — both eval arms only generate a reply and cannot execute commands, so it could never pass and contributed no gate signal. Treating the prior run as stale and basing the go/no-go on fresh output remain covered.",
       "assertions": [
         "Treats the prior CI run as stale, not as fresh evidence",
-        "Re-runs the verification command in the current session",
         "Does not rubber-stamp the merge on the 'only a comment' rationalization",
         "Bases the go/no-go on the fresh output"
       ]

package/skills/writing-skills/SKILL.md

@@ -109,8 +109,11 @@ description over the body, so get it right.
   report to a file and put anything needing user input in an "Open questions"
   section, since a fork returns only a summary and can't ask the user),
   `paths` (file-scoped), `disable-model-invocation` (deliberate-only workflows),
-  `effort`. Dynamic `` !`cmd` `` injection: cheap `--stat`-style commands only,
-  `|| true` failure-tolerant — see AUTHORING.md's injection rules.
+  `effort`. Dynamic injection (bang-prefixed backtick commands, inline or
+  fenced): cheap `--stat`-style commands only, `|| true` failure-tolerant — see
+  AUTHORING.md's injection rules. Note the syntax executes even inside markdown
+  code spans, so never write a runnable example of it in a SKILL.md — describe
+  it, or the skill runs the command every time it loads.
 - Progressive disclosure: keep SKILL.md tight (aim < 300 lines); push deep
   domain knowledge to `references/`, output formats to `templates/`. On
   compaction only a skill's first ~5k tokens are re-attached — front-load the
@@ -137,11 +140,11 @@ As base models improve, a skill can become pure overhead. Once per release cycle
 — or on a major base-model update — re-run the skill's 3 evals RED on shipping
 models via `evals/workflow-runner.mjs`; re-sample ×3 on borderline scores and
 read the judge journal before calling a result variance (a "flaky" case is often
-a real content gap the skill causes). **First check the skill's reference mass:**
-the harnesses can't read `references/` or `templates/`, so where that content
-rivals `SKILL.md` the RED ≈ GREEN verdict is a harness artifact — judge those by
-hand instead. Otherwise, if RED ≈ GREEN across all 3 evals (0/0
-tool-dependent assertion pairs don't count), **slim first** — cut what the model
+a real content gap the skill causes). The GREEN arm reads the skill's
+`references/` and `templates/`, so reference-heavy skills are judged against
+their full depth — the old carve-out for them is retired. If RED ≈ GREEN across
+all 3 evals (0/0 assertion pairs needing a real repo, a write, or an executed
+command don't count — the harness can't provide those), **slim first** — cut what the model
 does unaided; keep the Iron Law, boundaries, and cross-skill references — then
 re-run GREEN (gate: GREEN ≥ RED per case). Retire only after a slimmed skill
 stays RED ≈ GREEN a full cycle later, with a deprecation notice in