skilldrop-cli 0.16.29 → 0.16.31

package/packs/dev-team/skills/refactor-plan/examples/search-ranker.md

@@ -0,0 +1,104 @@
+# Worked example — search ranker with a flag pile
+
+## Input given to the skill
+
+> TD-07 from our register got the go-ahead. `search/ranker.py` has one 600-line function,
+> `rank(query, docs, **flags)`, driven by 14 boolean flags listed in `RANK_FLAGS` at the top
+> of the file. Register row: "5 of the last 8 search PRs changed `rank()`; INC-540 (Jul,
+> `legacy_tiebreak` reversed the order of equal scores) and INC-552 (Aug, `personalise` is
+> ignored when `boost_recent` is on — not fixed; the web client sends `personalise` only
+> when `boost_recent` is off)." `tests/search/test_ranker.py` covers 4 of the 14 flags. PR
+> #212 "Add `boost_local` flag" is open and edits `rank()`. Next quarter the mobile app needs
+> its own ranking with different weights. Owner: Search team.
+
+---
+
+## Reply in chat (the default: the user asked how, not for a document)
+
+Turn each flag into its own scoring function and build a ranking from a list of them, so the
+mobile ranking is a new list rather than an edit to `rank()`. First step: approval tests for
+every flag in a new test file, this week, because `rank()` has tests for only 4 of 14 flags.
+
+**What the code shows**
+- `search/ranker.py:rank` reads all 14 `RANK_FLAGS` in one body; `tests/search/test_ranker.py` covers 4 of them.
+- INC-552 is live behaviour a client depends on: the web client sends `personalise` only when `boost_recent` is off, so the test pins it as-is.
+- #212 is editing `rank()` now; moves wait for it, tests don't.
+
+**Steps** (each one PR to main; you can stop after any of them)
+1. [refactor] Approval tests in `tests/search/test_ranker_characterisation.py`: `personalise` + `boost_recent` together (INC-552, as-is), `legacy_tiebreak` on equal scores (INC-540), then each flag alone.
+2. [refactor] After #212 merges: extract the tie-break block into `_tiebreak`, unchanged; it caused INC-540.
+3. [refactor] Add `rank_with_profile(query, docs, profile)`; `rank(**flags)` builds a profile and delegates. The mobile team can start from here.
+4. [refactor] Move the 15 flags (with `boost_local`) into scoring functions, five per PR, three PRs.
+5. [refactor] Move callers to `rank_with_profile`, delete `**flags`, add a CI check that forbids new flags.
+
+**Target:** search PRs that edit `rank()`, 5 of the last 8 → the mobile ranking ships without editing it. **Stop point:** every flag is a scoring function (step 4); step 5 is scheduled but optional.
+**Not touched:** INC-552 behaviour, `RANK_FLAGS` names, weights, the API response. Anything found during the work that isn't a step goes to the debt register.
+**To confirm:** does anything outside this repo call `rank()` with flags? It decides whether step 5 deletes `**flags` or keeps a shim.
+
+---
+
+## The same plan as a document (when the user asks for one)
+
+# Refactor plan: take the scoring rules out of `rank()` so new rankings stop editing it
+
+**Target:** search PRs that edit `rank()` — baseline 5 of the last 8 (TD-07) → the mobile
+ranking ships without editing `rank()`.
+**Stop point:** every flag in `RANK_FLAGS` is its own scoring function, and a ranking is
+built from a list of them. Removing `**flags` from `rank()` is scheduled but outside the stop
+point (step 7).
+**First step:** characterisation tests for all 14 flags, then the expand step for a
+profile-based `rank` — first because the mobile ranking next quarter is the work this
+unblocks.
+**Type:** every step is a refactor; ranking output is identical throughout. The mobile
+weights are feature work built on step 3.
+**Not touched:** INC-552 behaviour, `RANK_FLAGS` names, the search API response shape.
+[assumption] `rank()` is called only from inside this repo; see To confirm.
+
+## Steps
+
+| # | Change | Technique | Verified by | Rollback (live side) | Risk | Size | Owner |
+|---|---|---|---|---|---|---|---|
+| 1 | Approval tests in a new `tests/search/test_ranker_characterisation.py`: `personalise` + `boost_recent` together, `legacy_tiebreak` on equal scores, then each flag alone | characterisation tests | new tests pass on current code | revert PR | none, tests only | M | Search team |
+| — | **Wait for #212 to merge** | — | — | — | — | — | — |
+| 2 | Extract the tie-break block into `_tiebreak(docs, legacy)` in `ranker.py`, unchanged | extract | step 1 tests | revert PR | ordering of equal scores (INC-540) | S | Search team |
+| 3 | Add `rank_with_profile(query, docs, profile)`; `rank(**flags)` builds a profile from the flags and delegates | parallel change (expand) | step 1 tests; `rank` output identical | revert PR; `rank` is the only caller | flag-to-profile mapping misses a flag | M | Search team |
+| 4–6 | Move flags into scoring functions, five, five and five per PR (15 with `boost_local`) | extract, behind step 3 | step 1 tests + #212's tests | revert that batch's PR | interaction between flags in one batch | M each | Search team |
+| 7 | Move callers to `rank_with_profile`; delete `**flags`; CI check forbids new flags | parallel change (contract) | suite + check | revert PR | caller outside the repo | S | Search team |
+
+Stop after any step: each leaves `rank()` shorter and better tested. Two entry points exist
+from step 3 to step 7 and nowhere else; step 7 is scheduled for the sprint after step 6.
+
+## Step detail (step 3 shown)
+
+- **Change:** new `rank_with_profile` in `search/ranker.py`; `rank` keeps its signature and builds a profile from `**flags`.
+- **Technique:** parallel change — callers move over later, so the old entry point stays live.
+- **Verified by:** every step 1 approval test, run through `rank` (unchanged callers).
+- **Rollback:** revert the PR; nothing calls `rank_with_profile` yet.
+- **Risk:** a flag dropped from the profile mapping changes ranking silently; the per-flag approval tests catch it.
+- **Size:** M. **Owner:** Search team.
+- **Clashes:** none once #212 has merged.
+
+## Characterisation tests
+
+| Function | Pin | Includes current bug? | Lives in |
+|---|---|---|---|
+| `rank` | each of the 14 flags alone over a recorded query set | no | `tests/search/test_ranker_characterisation.py` (new, so it can't conflict with #212) |
+| `rank` | equal scores with `legacy_tiebreak` on and off | no (INC-540 is fixed) | same |
+| `rank` | `personalise` + `boost_recent` together | **yes: personalise ignored, pinned as-is (INC-552)** | same |
+
+## Clashes with feature work
+
+| In-flight work | Shared files / functions | Decision | Effect on the plan |
+|---|---|---|---|
+| #212 Add `boost_local` | `search/ranker.py::rank` | wait | step 2 starts after #212 merges; `boost_local` joins batch 4–6 |
+| Mobile ranking (next quarter) | `rank` | fold in | the mobile team builds its profile on `rank_with_profile` once step 6 lands |
+
+## Scope fence
+
+- **Not touched:** search API response shape, `RANK_FLAGS` names, weights.
+- **Pulled out of this plan:** fixing INC-552 is a behaviour change; it is a separate ticket, and the web client workaround has to be removed in the same change.
+- **Rule:** anything found during the work that isn't in a step goes to the debt register as a new item, not into this plan.
+
+## To confirm
+
+- Does anything outside this repo call `rank()` with flags? It decides whether step 7 can delete `**flags` or must keep a shim.

package/packs/dev-team/skills/refactor-plan/manifest.json

@@ -0,0 +1,36 @@
+{
+  "name": "refactor-plan",
+  "version": "0.2.0",
+  "description": "Plan paying down one piece of technical debt as a refactor that serves the goal it is blocking — characterisation tests on the riskiest behaviour first, then small steps each merged to main with main releasable after every one, pure refactors kept apart from the behaviour changes the goal needs, a scope fence, a stop point, a technique per step (extract, branch by abstraction, parallel change, strangler fig, mikado) and the clashes with in-flight feature work. Use when the user asks for a refactor plan, how to untangle, break up, replace or clean up a module without a rewrite, how to pay down a tech-debt item, \"how do we refactor X without stopping feature work\", \"how do we get off X without a big bang\", or wants a strangler-fig or mikado plan.",
+  "entrypoint": "SKILL.md",
+  "deps": { "npm": [], "pip": [] },
+  "env": { "required": [], "optional": [] },
+  "related": [
+    "adr-generator",
+    "design-doc",
+    "feature-implement-loop",
+    "migration-plan",
+    "pre-merge-review",
+    "tech-debt-register",
+    "user-story-splitter"
+  ],
+  "handoff": [
+    {
+      "to": "feature-implement-loop",
+      "when": "the plan is agreed and a step is ready to build",
+      "purpose": "Implement one step, using its Change and Verified-by fields as the acceptance criteria.",
+      "fallback": "If feature-implement-loop is absent, implement the step as one PR, run the characterisation tests and the existing suite before and after, and merge only when both are green with no test edited to pass."
+    },
+    {
+      "to": "tech-debt-register",
+      "when": "planning or a step turns up debt outside the scope fence, or the stop point is reached",
+      "purpose": "Record new debt as its own item and update the paid-down item's cost and decision.",
+      "fallback": "If tech-debt-register is absent, list each new item with where it lives, the evidence of its cost, and a decision (pay now, schedule, accept, watch) at the end of the plan."
+    }
+  ],
+  "tags": ["refactoring", "tech-debt", "legacy-code", "planning", "engineering"],
+  "model": {
+    "tier": "standard",
+    "rationale": "Sequencing behaviour-preserving steps follows known patterns (characterise, parallel change, strangle); escalate via ambiguous-input for contested rewrites."
+  }
+}

package/packs/dev-team/skills/refactor-plan/reference.md

@@ -0,0 +1,149 @@
+# Refactoring techniques: when each fits and what rollback means
+
+**Refactor or behaviour change.** A refactor step keeps output identical and is proven by the
+pinned tests passing unchanged. A behaviour change step changes a pinned value on purpose,
+says which, and has its own rollback (usually a flag). A plan names the type of every step and
+never puts both in one PR.
+
+Every technique here keeps main releasable between PRs. Pick the smallest one that fits the
+step; a heavier technique than the step needs adds seams nobody removes.
+
+## Characterisation tests
+
+**Fits:** code with no tests on the paths callers use, before any step touches it.
+(Michael Feathers, *Working Effectively with Legacy Code*.)
+
+**Step shape:** one PR that adds tests only. No production code changes, except a seam
+needed to make the code testable (an injected clock, a parameter for a hard-wired client),
+and that seam goes in its own PR first.
+
+**What to pin:**
+- The outputs for the inputs callers actually send — read the callers, not the docstring.
+- Boundary inputs: empty, zero, negative, very large, null, duplicate, unsorted.
+- Rounding, ordering, timezone and locale behaviour.
+- Error behaviour: which exception, which error code, what was half-written before it raised.
+- Known bugs, as they behave today. Name the test for what it is
+  (`test_tax_rounds_per_line_current_behaviour`) and point it at the register item that
+  records the bug.
+
+**When output is too wide to assert by hand:** an approval (golden-master) test. Record
+outputs for a set of real-shape inputs, store them next to the test, and diff on every run.
+Scrub timestamps and ids first or the test flaps.
+
+**Rollback:** revert the PR. Nothing in production changed.
+
+**Open PRs on the same file:** never a reason to wait. The tests touch no production code, so
+they go in a new test file and merge now. If the open PR changes pinned behaviour on purpose,
+its author updates the pinned value in that PR, which makes the change visible in review.
+**Order:** the behaviour behind past incidents and the branches comments warn about first,
+the quiet paths last.
+
+**Failure mode:** asserting what the code should do. The test fails against the current code
+and the team deletes it.
+
+## Extract / inline / move
+
+**Fits:** a cohesive piece (a function, a group of methods sharing state) that can leave its
+current home in one PR without changing callers.
+
+**Step shape:** two PRs when the code moves files. PR A moves it unchanged and leaves the old
+name pointing at the new one (re-export, delegating method). PR B changes it. Callers move
+to the new name in later PRs or in the contract step.
+
+**Rollback:** revert the PR. A pure move reverts cleanly.
+
+**Failure mode:** move and edit in one PR. The reviewer can't see the edit in a 900-line diff.
+
+## Branch by abstraction
+
+**Fits:** replacing a dependency (a client, a store, a library) that many callers use, when
+the swap takes more than one PR. (Paul Hammant; Martin Fowler's write-up.)
+
+**Step shape:**
+1. Introduce an abstraction over the current dependency; route callers through it in
+   batches. Behaviour unchanged.
+2. Build the new implementation behind the same abstraction, dark.
+3. Switch which implementation is live with a flag or config, for a slice first.
+4. Remove the old implementation and, if it adds nothing, the abstraction.
+
+**Rollback:** before step 3, revert the PR. From step 3, flip the flag back — the plan names
+the flag and which side is live after each step.
+
+**Failure mode:** stopping after step 3. Two implementations stay forever.
+
+## Parallel change (expand–contract) at code level
+
+**Fits:** changing a function signature, an interface or a data shape that many callers
+use. (Danilo Sato's write-up of the pattern on Martin Fowler's site.)
+
+**Step shape:**
+1. **Expand** — add the new signature alongside the old; the old delegates to the new.
+2. **Migrate** — move callers in batches, one PR per batch.
+3. **Contract** — remove the old signature once nothing calls it, and add a check (lint
+   rule, test, grep in CI) that stops it coming back.
+
+**Rollback:** each batch reverts on its own. The old signature stays live until contract.
+
+**Failure mode:** no contract step. Schedule it within two steps of the last migrate batch.
+
+**Caller contracts:** the same shape removes a side effect a caller relies on (a mutated
+argument, a swallowed error reported as success). Expand: give the caller what it reads
+another way. Migrate: move that caller. Contract: remove the side effect. Removing it is a
+behaviour change and gets its own step; it is never folded into an extract.
+
+This is the code-level version. When the change crosses a datastore with a backfill, it is a
+migration, not a refactor, and it gets its own migration plan.
+
+## Strangler fig
+
+**Fits:** replacing a whole module or service piece by piece, when there is an edge (a
+router, a facade, an entry function) where each call can go to old or new. (Martin Fowler.)
+
+**Step shape:**
+1. Put a facade at the edge; everything still goes to the old code.
+2. Move one capability at a time to new code; the facade routes that capability to it.
+3. When the old code serves nothing, delete it.
+
+**Rollback:** route the capability back to the old code at the facade. The plan names, per
+step, which capabilities the new code serves.
+
+**Failure mode:** using it for something one extract would do. A facade over a single
+function is a seam nobody needs.
+
+## Mikado method
+
+**Fits:** tangled dependencies where each attempted change breaks on something that must
+change first, and nobody knows the full list. (Ola Ellnestam and Daniel Brolund, *The Mikado
+Method*.)
+
+**Step shape:**
+1. Write the goal at the top of a graph ("`billing` imports nothing from `web`").
+2. Try it naively. Note every breakage as a prerequisite node. **Revert.**
+3. Repeat for each prerequisite until reaching leaves that change cleanly.
+4. Ship the leaves first, one PR each, working back up to the goal.
+
+**Rollback:** each leaf is one PR; revert it. The exploration commits are never merged.
+
+**Failure mode:** keeping the naive attempt and fixing forward. That is the big-bang rewrite
+arrived at by accident.
+
+When an import graph or a compiler error list is already in the input, the first pass of the
+graph comes from it: each forbidden edge is a node, and edges with no further dependency are
+the leaves.
+
+## Measuring the target
+
+Pick one metric, two at most, and take the baseline before step 1.
+
+| Metric | Read it from |
+|---|---|
+| Share of feature PRs touching the file | git log over merged PRs in a window |
+| Files touched per typical change | git log for the last N changes in the area |
+| Change lead time for the module | PR open to merge for PRs touching it |
+| Review time for PRs touching it | the code host's PR data |
+| Test run time for the module's suite | CI timings |
+| Defects or incidents traced to it | the incident log or bug tracker |
+| Forbidden imports or direct call sites | a grep or import-linter count, enforced in CI at the end |
+
+A count that can be enforced in CI (forbidden imports, direct call sites) makes the best stop
+point: the last step turns the count into a check, so the debt can't come back.

package/packs/dev-team/skills/refactor-plan/templates/refactor-plan.md

@@ -0,0 +1,82 @@
+# Refactor-plan template
+
+Two shapes. Use the chat shape unless the user asked for a document (a file, "a proper plan",
+something to take to a meeting) or the plan runs past about ten steps. Never send both at
+full length: when a document is written, the reply is its lead, the findings and the path.
+
+## Chat shape (default)
+
+```markdown
+{Approach in one sentence: the technique and the end state the goal needs.}
+First step: {the step} — {why it goes first}.
+
+**What the code shows**
+- `{file}:{function}` {a caller contract, a lying return value, a copy outside the module, real test coverage, a stale signal}
+- … (three to six bullets, each one that changes the plan)
+
+**Steps** (each one PR to main; you can stop after any of them)
+1. [refactor] Characterisation tests for {riskiest behaviour first: …}, in {new test file}. {why}
+2. [refactor] {change} — {why}. {clash or rollback, if it matters}
+3. [behaviour change] {change the goal needs} — after step {n} makes it safe; {caller moved first}; rollback: {flag off / revert}.
+…
+n. {contract: delete the old path; add a check so it can't come back}
+
+**Target:** {metric}: {baseline from the files, or "to measure"} → {goal}. **Stop point:** {checkable state}.
+**Not touched:** {fence, one line}. Anything found during the work that isn't a step goes to the debt register.
+**To confirm:** {at most three questions or checks to run}
+```
+
+## Document shape (asked for, or large)
+
+```markdown
+# Refactor plan: {the debt, in one line — "split OrderService so order features stop editing one file"}
+
+{Approach in one sentence.} **First step:** {step} — first because {the work it unblocks}.
+
+**Goal:** {the user's words} → **End state:** {the structure that goal needs}
+**Target:** {metric}: {baseline from the input, or "not measured; step 0 measures it"} → {goal}
+**Stop point:** {the "done enough" line, checkable — and what stays as it is after it}
+**Not touched:** {modules, contracts, behaviours outside the fence}
+[assumption] {anything inferred; one line each}
+
+## What the code shows
+
+- `{file}:{function}` — {finding, and the step it shapes}
+
+## Steps
+
+| # | Change | Type | Technique (why) | Verified by | Rollback (live side) | Risk | Size | Owner |
+|---|---|---|---|---|---|---|---|---|
+| 1 | Characterisation tests for {incident-prone behaviour first} in {new test file} | refactor | characterisation tests | pass on current code | revert PR | none, tests only | S | {role} |
+| 2 | Move {function} to {file} unchanged; old name re-exports | refactor | move | step 1 tests + suite | revert PR | import cycles | S | {role} |
+| 3 | Route {callers} through {interface} | refactor | branch by abstraction | step 1 tests | revert PR; old impl live | missed call site | M | {role} |
+| 4 | {behaviour the goal needs} behind flag `{name}` | behaviour change | {technique} | updated pins, reviewed | flag off; old side live | {what} | M | {role} |
+| … | | | | | | | | |
+| n | Contract: delete {old path}; add {CI check} | refactor | parallel change (contract) | suite + check green | revert PR | caller outside the repo | S | {role} |
+
+Stop after any step: each leaves the code better than before. Two paths exist from step {x}
+to step {y}; nowhere else.
+
+## Characterisation tests
+
+| Function | Pin (riskiest first) | Includes current bug? | Lives in |
+|---|---|---|---|
+| `{file}:{function}` | {inputs → outputs, edge cases} | {yes: "{bug}", pinned as-is / no} | {new test file if an open PR edits the existing one} |
+
+## Clashes with feature work
+
+| In-flight work | Shared `file:function` | Decision | Effect on the plan |
+|---|---|---|---|
+| {PR or feature} | {…} | land first / wait / fold in / through the seam / freeze window | {…} |
+
+## Scope fence
+
+- **Not touched:** {list}
+- **Pulled out of this plan:** {behaviour changes the goal doesn't need, and where each goes instead}
+- **Copies outside the module:** {in or out, by name; if out, the plan doesn't claim them fixed}
+- **Rule:** anything found during the work that isn't in a step goes to the debt register as a new item, not into this plan.
+
+## To confirm
+
+- {the facts the plan needs and the input didn't give, as checks to run}
+```

package/packs/dev-team/skills/rollout-plan/SKILL.md

@@ -0,0 +1,157 @@
+---
+name: rollout-plan
+description: Plan the progressive release of a code change behind a feature flag or canary — the exposure ladder (internal, beta cohort, 1%, 5%, 25%, 50%, 100%, or what the traffic supports), and for each step who is exposed, decision and guardrail metrics with a baseline and a threshold against a control cohort, the minimum traffic and time before the numbers mean anything, a promote / hold / roll back rule set in advance, who watches and what aborts automatically; then what turning the flag off does not undo, and who removes the flag by when. Use when the user asks for a rollout plan, canary plan, ramp schedule, progressive delivery or percentage rollout, "how long should the canary run", "when do we go from 5% to 25%", or how to turn on a flagged feature safely.
+---
+
+# rollout-plan
+
+Plans how a code path already behind a flag or canary reaches users: in what steps, judged
+by which numbers, decided by which rule, and undone how. The plan's unit is **exposure of a
+code path to users**. Moving data or changing a schema is a migration and belongs to
+`migration-plan`; whether the change may go live at all is `launch-readiness`'s go/no-go.
+This plan starts once both of those are settled, or names them as open.
+
+## How to respond
+
+**Four rules come before the steps and outrank them:**
+
+- **Answer what was asked, first.** Open with the answer, the decision or the artifact, in plain words. Scores, matrices, frameworks and tags come after it, and anything that doesn't change the answer is cut.
+- **Use only what you were given.** Don't add facts, names, numbers, incidents, history, steps or sections the input doesn't contain. What you need and don't have is left out of the artifact and listed once at the end under "To confirm".
+- **Deliver from what you have.** When the request gives you something to work on, state your assumptions in a line and produce the result. When it gives you nothing to work on, ask for it in one or two plain sentences and say what you will do once you have it.
+- **Write for someone who has never heard of this skill.** No skill names, no paths or scripts from this folder, no internal terms, and nothing about how the run was set up. A next step is one plain sentence at the end that describes the work.
+- **For this skill:** the plan is built around the input's binding constraint, never around a stock ladder. Sample sizes are working, not the answer: state what they mean for the plan in one sentence, with the inputs that produced them.
+
+1. **Answer the question in the first two lines.** "Who's right?" gets a name. "Is 1% Monday to 100%
+   Thursday OK?" gets yes or no. "How long should the canary run?" gets a duration. Then the one
+   fact from the input that settles it, and the shape of the plan.
+   ✅ *"The payroll lead is right: 1% of employers for an hour sees no pay run at all, so a clean hour proves nothing. Shadow the new calculator on Friday's run, then move named employers through the next two pay cycles."*
+   ❌ *"Each arm needs 9,412 units (19 days at 1%)."* (A statistic the reader didn't ask for, built on a rate they never gave.)
+
+2. **Find the binding constraint before drawing a ladder.** Read the input for the fact that
+   decides pacing, and name it in one line. The common ones:
+   - **A partner or quota cap** ("stay under 500 new accounts a day"): convert each step's exposure into the capped unit (exposure × volume × take rate), state the take rate as an `[assumption]` if it isn't given, and hold the step that would cross it.
+   - **A calendar cycle** (billing run on the 1st, payroll, month-end close, a weekly batch): the evidence arrives on the cycle, not by the hour. Count the cycles left before the deadline and place the steps on them.
+   - **A deadline or freeze** ("before the holiday freeze on Dec 1", "before year-end close"): the schedule works back from it. If the evidence the steps need doesn't fit, say so in the opening, not in a footnote.
+   - **Segments the flag targets** (tenants, accounts, regions, plans, allow-lists) or a flag that cannot target by %: steps are named cohorts chosen for a reason, not hash buckets.
+   - **A result checkable without exposure** (tax, prices, ranking, balances): a shadow run beside the old path, writing nothing a customer or downstream system reads, is the main gate, whether the design already has one or the plan adds it.
+   - **Low volume**: when a step sees single-digit counts, percentages and power calculations mean nothing. Judge in counts (zero tolerance, "no more than 2 per run") and widen the step to a segment or a cycle.
+   The fixed internal → 1% → 5% → 25% → 50% → 100% ladder is the default **only** when traffic is high and nothing above shapes the pacing.
+
+3. **Pin the mechanics.** The code path; the flag (key, where it is evaluated: server, client,
+   edge, and what it can target); the **randomisation unit** (user, account, tenant, device,
+   request); volume in that unit; the baselines given. Ask at most 2 questions, spent on volume
+   and on what the new path writes or sends. A schema change, backfill or dual-write in the same
+   change is a migration that finishes its expand phase before step 1; say so in one line.
+
+4. **Pick cohorts with reasons.** Every step names who is in it and why:
+   - **Internal or test accounts first.** They catch crashes, not rates.
+   - **Chosen cohorts beat random ones when the risk sits in a segment.** Put the cases the change exists to fix (the edge cases, the large accounts, the unusual configurations) into an early cohort deliberately, with willing customers, or hold them back deliberately and say why. ❌ a random hash that may contain none of them.
+   - **Sticky, nested assignment** for percentage steps: hash on the randomisation unit so the 1% stays inside the 5%. ✅ *"`hash(salt + user_id) mod 1000 < 10`"* — ❌ *"random 1% of requests"* when the flag changes what a user sees.
+   - **SLA or contractual-notice customers** stay out before 50% unless they asked to be early; say how they are excluded.
+   - **Units in another experiment on the same surface** are excluded, or the comparison is called confounded.
+   - **B2B: assign by account or tenant**, so one team never sees two versions.
+
+5. **Choose the metrics and write thresholds that can fire.** One decision metric (what the change
+   exists to move, or the main correctness or safety rate) plus guardrails: error rate of the touched
+   path, p95 of the touched endpoint, saturation of what it leans on (pool, queue, third-party quota),
+   one business guardrail. Each carries the baseline from the input and a limit **against the
+   concurrent flag-off control**, never against last week.
+   - ✅ *"checkout errors, flag-on ≤ flag-off + 0.3 pp (baseline 0.8%)"* — ❌ *"monitor errors"*.
+   - **Small counts get count rules.** ✅ *"any unexplained difference over $0.01 between old and new totals holds that account's statement from the send"* — ❌ *"mismatch rate ≤ 0.1%"* on 30 statements.
+   - p95 needs 2,000+ requests per arm and p99 10,000+; below that, read the median.
+   - A metric with no instrument yet is a precondition for step 1, written once.
+
+6. **Size steps from the arithmetic, and say what it means.** For a rate metric on real traffic, run
+   `python3 ${CLAUDE_SKILL_DIR}/scripts/canary_math.py --baseline 1% --detect-abs 0.3pp --traffic 2000 --unit users --steps 1,10,50`
+   (other IDEs: `python3 scripts/canary_math.py ...` from the skill folder; `--direction down` when a
+   fall is the harm, `--control-pct` for a fixed holdback; formula in [`reference.md`](reference.md)).
+   - **Put one sentence in the reply, not the table.** ✅ *"At 2,000 users an hour, a 1% step needs about 20 days to see a 0.3 pp rise in errors, so it is a 2-hour crash check and errors are judged at 10% (about 2 days)."* Show the inputs (traffic, baseline, regression size) beside it.
+   - **No figure without a source.** Every number in the reply is from the input, from arithmetic on the input, or tagged `[assumption]` in one line. With no baseline or traffic, write the step as a unit count and one line saying what was assumed; never a duration built on an invented rate.
+   - **Step minimum = the largest of** the statistical time, one daily peak (24 h when traffic varies by hour), a weekday/weekend cycle where the business has one, and the delay of any late effect (nightly send, billing run).
+   - **A step that can't detect anything** is merged, raised, or kept as a crash-only check, said plainly. ✅ *"1% for 2 h: crash check only"* — ❌ *"1% for 24 h to be safe"*.
+   - **100% has no concurrent control**: promote on the 50% comparison, or keep a small holdback for a stated time.
+
+7. **Put dates on it when the input has a date.** With a deadline, freeze, cycle or "this week",
+   each step gets a calendar date counted from today, and the last line of the ladder shows the
+   deadline is met or by how much it is missed. Name the slack: ✅ *"one failed cycle pushes 100% past year-end close."*
+
+8. **Write each step's rule before it starts.**
+   - **Promote** when the minimum time or count is reached and every guardrail is inside its threshold.
+   - **Hold** when the sample is short or a metric sits between threshold and abort: extend once, by a stated time or cycle, then roll back.
+   - **Roll back** when a guardrail breaches at the planned sample, or an abort fires.
+   - **Abort, automated where the tooling allows**: wide, fast triggers. ✅ *"flag-on 5xx > 3× flag-off for 5 min → flag off automatically, page payments on-call"*; ✅ *"unexplained diff on a live statement → account back to the old engine, its unsent statements held"*.
+   - **Watcher and override**: the role watching, the sign-off before each promotion where one is needed (finance, tax, a partner contact), and the one role allowed to promote past a breach.
+   - **No promotion on peeks.** Promotion is read at the step's end; aborts fire any time.
+
+9. **Name what flag-off does not undo.** Who flips the flag and how fast every client stops taking
+   the new path (server: seconds; cached client config: its TTL; shipped binary: only a server-side
+   switch). Then only the categories this change touches, each with its handling:
+   - **Data in a new format or table**: what still reads it after rollback (the renderer, re-runs, credit notes, exports). Verified before step 1; the table can't be dropped while records point at it.
+   - **Messages sent** (email, push, webhooks, accounting syncs): no recall; the correction or reissue path exists before the first live step.
+   - **Money and third-party records** (charges, agreements, filings, per-call fees): the reversal procedure and who runs it, or "permanent".
+   - **Jobs queued, caches written, clients shipped**: when they stop.
+   Name the **point of no return**, or rule it out in a line.
+
+10. **Schedule the flag's removal.** An owner (the named owner if the input gives one, else a role), a
+    date (default 30 days after 100%; for a cycle-driven change, after the first full cycle at 100%
+    with no rollback), and what goes: the flag, the dead path, its config and dashboards. Say what
+    must stay (a reader for records the old path wrote; the minimum app version for shipped code).
+
+11. **Deliver lean.** One plan, shaped by [`templates/rollout-plan.md`](templates/rollout-plan.md):
+    the answer, the binding constraint, the dated ladder, the rules, the not-undone list, flag removal.
+    Either the plan in the reply or a saved file with a five-line reply, never both at full length.
+    "To confirm" holds at most 4 facts, each one that would change a step. Sections that don't apply
+    are dropped, not filled with "does not apply".
+
+**Non-interactive runs** (subagent, CI, headless): the regression size, take rates, cohort choices and
+step lengths become `[assumption]`s in one line. Missing traffic or baselines are never invented. If
+the change or its gate can't be identified, emit `BLOCKED: need the change being released and the flag
+or canary that gates it`.
+
+## Useful references in this skill
+
+- [`templates/rollout-plan.md`](templates/rollout-plan.md) — the plan skeleton: answer, constraint, dated ladder, rules, not-undone list, flag removal
+- [`reference.md`](reference.md) — the sample-size formula, the randomisation-unit trap, count rules for small volumes, threshold patterns, the not-undone checklist
+- [`scripts/canary_math.py`](scripts/canary_math.py) — units and hours each percentage step needs; flags steps that can't reach sufficiency (stdlib only)
+- [`examples/sync-engine-workspaces.md`](examples/sync-engine-workspaces.md) — a plan for a sync engine ramped by workspace, where flag-off leaves notes the old engine can't read
+
+## Quality bar
+
+- **The first two lines answer the user's question** in plain words, with the input fact that settles it.
+- **The binding constraint is named** and the ladder visibly follows it (cap, cycle, deadline, segment, shadow, volume).
+- **Every threshold is testable**: metric, baseline, limit against the flag-off control; counts where volumes are small.
+- **Every number traces** to the input, to arithmetic on it, or to a one-line `[assumption]`.
+- **A deadline in the input gets a dated ladder** that shows whether it is met.
+- **Promote, hold, roll back and abort are written before the step**, with a watcher, a sign-off where one is needed, and an override role.
+- **The not-undone list covers what this change sends, writes or charges**, with handling for each.
+- **The flag has an owner and a removal date.**
+- **The reply is shorter than the plan it replaces would be at full length**; "To confirm" is never longer than the ladder.
+
+## When to use this skill
+
+- ✅ A merged change behind a flag needs a ramp, canary schedule or cohort plan
+- ✅ "How long does the 1% have to run?", "when do we promote?", "is this ramp right?"
+- ✅ Turning on a risky path (payments, tax, auth, ranking, a new dependency) for real users
+- ✅ A rollout that sends, charges or writes something the flag can't take back
+
+## When NOT to use this skill
+
+- ❌ Schema changes, backfills, datastore or provider swaps, dual-writes: `migration-plan`
+- ❌ "Are we ready to launch?" go/no-go: `launch-readiness`
+- ❌ SLOs, alerts and dashboards for the service: `observability-plan`
+- ❌ Defining what success means for the feature: `success-metrics`
+- ❌ The on-call procedure for the service as a whole: `runbook-generator`
+- ❌ A change with no flag, canary or staged deploy: there is nothing to ramp; plan the gate first
+
+## Anti-patterns to avoid
+
+- ❌ **Statistics as the headline.** "11,734 per arm, 26 days" before the user knows whether their plan is right.
+- ❌ **The ritual ladder.** 1% → 5% → 25% → 50% → 100% on a change whose evidence arrives once a month, or under a partner cap it would cross.
+- ❌ **Random buckets where the risk is a segment.** A pricing change hashed by account may never touch the multi-currency accounts it exists to fix.
+- ❌ **Percent thresholds on single-digit counts.** "Failure rate ≤ 0.5%" on 30 statements fires on one failure or never.
+- ❌ **Invented inputs.** An adoption rate, mismatch rate or per-account volume the input never gave, presented as fact.
+- ❌ **No dates under a deadline.** "Step 3: 48 h" when the user named a freeze date.
+- ❌ **Comparing against yesterday** instead of the concurrent flag-off cohort.
+- ❌ **"Rollback: turn the flag off"** when the path sent emails, charged cards or wrote rows old code can't read.
+- ❌ **The padded reply.** A full plan in the message and again in a file, a "To confirm" table longer than the plan.
+- ❌ **The flag that never leaves.** No owner and no date.

package/packs/dev-team/skills/rollout-plan/evals/eval_queries.json

@@ -0,0 +1,12 @@
+[
+  { "query": "Plan a canary rollout for the new pricing engine — it's behind a feature flag", "should_trigger": true },
+  { "query": "What percentages should we ramp the checkout_v2 flag through, and when do we promote each step?", "should_trigger": true },
+  { "query": "How long does our 1% canary need to run before the error numbers mean anything?", "should_trigger": true },
+  { "query": "Write a progressive delivery plan with rollback criteria for the new recommendations widget", "should_trigger": true },
+  { "query": "We're turning on the new login flow for users next week — how do we do it gradually and safely?", "should_trigger": true },
+  { "query": "Plan the backfill and expand-contract migration for splitting the users table", "should_trigger": false },
+  { "query": "Are we ready to launch the new checkout on Monday? Give me a go/no-go", "should_trigger": false },
+  { "query": "Write the release notes for version 4.2", "should_trigger": false },
+  { "query": "Design SLOs and burn-rate alerts for the payments API", "should_trigger": false },
+  { "query": "How will we know if the new onboarding flow worked? Define the success metrics", "should_trigger": false }
+]