skilldrop-cli 0.16.30 → 0.16.32

package/packs/dev-team/skills/rollout-plan/evals/evals.json

@@ -0,0 +1,109 @@
+{
+  "skill_name": "rollout-plan",
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "Plan the rollout for PR #2291. Checkout now calls our new tax service, Ledgerline, instead of the in-process tax tables. It's behind the flag `checkout_tax_v2`, evaluated server-side per user id. Checkout gets about 9,000 attempts an hour on average, peaking around 15,000 at 19:00. Today's checkout error rate is 0.8% and checkout conversion (attempt to paid order) is 62%. p95 checkout latency is 410 ms. The new path writes a `tax_breakdown` JSON into `orders.tax_details` (the column was added and deployed last sprint; the old code ignores it) and the order-confirmation email now shows the tax breakdown. Ledgerline bills us per call. About 40 enterprise accounts have a 99.95% checkout availability SLA. We'd like to be at 100% within a week.",
+      "assertions": [
+        "Opens with the ladder and total time in a line or two, and says whether 100% within a week is achievable with these numbers",
+        "Lays the steps out on a day-by-day schedule (day 1 to day 7, or dates) so the one-week target can be checked against it, and says what slips if a step holds",
+        "Every step names who is exposed and why, starts with internal users, and keeps the roughly 40 SLA enterprise accounts out of the early percentage steps",
+        "Error rate, p95 latency and conversion each carry the supplied baseline (0.8%, 410 ms, 62%) and a threshold relative to the concurrent flag-off cohort, not to yesterday and not 'monitor'",
+        "Sizes step durations from the 9,000 attempts/hour traffic and says a 1% step cannot detect a regression of about 0.3 pp in error rate in a sensible window (it needs roughly 3.5 days), changing the ladder accordingly",
+        "Every step has promote, hold and roll back rules written in advance, an abort trigger, a watcher role and the role allowed to override",
+        "Lists what turning the flag off does not undo: orders already charged with Ledgerline's tax amounts, confirmation emails already sent, tax_details rows written, and Ledgerline per-call charges, each with how it is handled",
+        "Does not re-plan the tax_details column, which is already deployed, and treats any further schema or data change as separate migration work",
+        "Names an owner role and a date or date rule for removing the flag and the old tax-table code path",
+        "Does not invent numbers the prompt did not give (p99 baseline, Ledgerline latency or error rate, enterprise traffic share); the size of regression to catch is labelled as an assumption to confirm"
+      ]
+    },
+    {
+      "id": 2,
+      "prompt": "Draft the rollout plan for one-tap reorder. The PR is described in pr-description.md and current numbers are in metrics-snapshot.txt. Product wants 1% on Monday and 100% by Thursday — is that OK?",
+      "assertions": [
+        "Answers the question first: the 1%-Monday-to-100%-Thursday schedule cannot detect a meaningful regression at 1,200 users per hour, and gives the time small steps would actually need",
+        "Any sample-size or duration figure appears as one plain sentence saying what it means for the ladder, with its inputs (traffic, baseline, regression size) shown; the reply does not open with a table of units per arm",
+        "Proposes a ladder whose step lengths follow from the 1,200 users/hour traffic and the supplied baselines (1.5% failure rate, 18% conversion), with larger or longer steps where the numbers need them",
+        "Thresholds compare the flag-on cohort with the flag-off cohort and include guardrails beside the decision metric (failure rate, p95/p99 latency of POST /v2/orders, payment declines or substitution complaints)",
+        "Notes that the client-side flag with a 24-hour cached config means flag-off can take up to a day to reach app 5.8 users, and calls for a server-side kill switch on the reorder endpoint",
+        "Lists partner orders sent to FreshPick, cards charged and push notifications sent as not undone by flag-off, with handling for each",
+        "Keeps users in the home_carousel_v3 experiment out of the rollout cohorts or explains how the overlap is handled, and holds business accounts back because of their 2-week notice contract",
+        "States that order metrics need a split by source before step 1, since no metric splits by source yet",
+        "Gives the flag an owner and removal date, noting that the code in app 5.8 cannot be removed until that version stops being supported",
+        "Does not invent traffic figures, baselines, owners' names or partner behaviour that are not in the files"
+      ]
+    },
+    {
+      "id": 3,
+      "prompt": "we're a two-person team and the new search ranking on our recipe site is behind a feature flag. about 2,000 searches a day. how should we roll it out? keep it short",
+      "assertions": [
+        "Is short and in plain language: a handful of steps or bullets, not a full multi-section plan",
+        "Says plainly that at about 2,000 searches a day a small percentage step will not show a reliable difference, and recommends a split large enough (for example 50/50) run for days, or accepting that only large regressions will be caught",
+        "Names a metric to compare between the flag-on and flag-off groups (for example searches that end in a recipe click, or zero-result searches) with a threshold decided before starting",
+        "Says how to roll back (flag off) and whether anything is left behind, such as cached results",
+        "Includes removing the flag and the old ranking code by a set date",
+        "Does not invent a baseline click rate, conversion figure or other numbers the user did not give"
+      ]
+    },
+    {
+      "id": 4,
+      "held_out": "2026-10-20",
+      "prompt": "We're about to turn on pay-in-4 (Lumen Pay installments) on web checkout. It's behind checkout.pay_in_4, merged and deployed dark. PR description is in PR_DESCRIPTION.md and I pulled our funnel numbers into metrics_snapshot.txt. Product wants it at 100% before Black Friday code freeze on Nov 20. Can you write the rollout plan: the ramp schedule, what we watch at each step, how long to sit at each %, and when we pull it? I mostly care that we don't tank checkout conversion or blow up payment errors.",
+      "assertions": [
+        "Buckets exposure by a sticky key (user_id or equivalent, as the PR says the flag supports), not per request or per page view, so a shopper does not flip between payment UIs mid-checkout, and the control cohort is the unexposed users over the same time window.",
+        "Uses the given volume (about 41k sessions/hour, 3.1% conversion) to show roughly how many sessions or orders each step produces, and says plainly that the early small-percentage steps can only catch payment-error, latency or crash regressions. It does not claim a 1% step can detect a change in conversion of the size Product cares about, and it gives a rough sample-size or duration estimate for the step where conversion is actually judged.",
+        "Checks the ladder against Lumen's 2,000-new-agreements/day cap for the first 7 days, converting exposure % into expected agreements/day. It states the pay-in-4 take rate as an explicit assumption, since no take rate is given, and caps or holds the ramp so the limit is not breached.",
+        "Sets promote / hold / roll-back rules with numbers before the rollout starts, tied to the given baselines. Examples: a payment-submit error ceiling relative to the 0.42% baseline against control, a p95 latency ceiling relative to 1.8 s, and a conversion floor relative to 3.1%. At least one guardrail (for example, payment errors or Lumen API 5xx) triggers an automatic flag-off, not just a human review.",
+        "Spends long enough at the decision steps to cover the daily traffic cycle, including the 20:00-22:00 ET peak, and ideally a weekday/weekend cycle, rather than promoting after an hour. The schedule is shown to reach 100% before the Nov 20 freeze, or the plan explicitly flags that the deadline and the evidence requirement conflict.",
+        "Names what flag-off does NOT undo: Lumen agreements already created (with first installments charged and Lumen's emails sent) stay live, and orders with payment_method='installment' must still be refundable and serviceable. It notes that installment refunds have no admin tool yet (#4190 not merged, CS uses the Lumen dashboard), and it prepares CS/support for this, or makes #4190 a gate, before the broad-exposure steps.",
+        "Assigns who watches each step (by role or team, e.g. checkout on-call plus a payments/Lumen contact) and names an owner plus a date for removing the checkout.pay_in_4 flag and its dead code path after full rollout.",
+        "Does not invent figures the user didn't give, such as the pay-in-4 take rate, the Lumen API error rate, the refund or chargeback rate, or mobile traffic. Wherever the plan needs such a number, it labels it as an assumption or something to measure during the internal/beta step."
+      ]
+    },
+    {
+      "id": 5,
+      "held_out": "2026-10-20",
+      "prompt": "Our billing team is switching invoice tax calculation to a new engine (Taxwell) behind billing.tax_engine_v2. Design note is at docs/tax-engine-v2.md. My instinct is the usual 1% for an hour, then 10%, 50%, 100% over a couple of days, but the billing lead says that won't tell us anything. Who's right, and what should the rollout actually look like? We'd like to be fully on v2 before Q1.",
+      "assertions": [
+        "Agrees with the billing lead and shows why using the note's numbers: at ~40-60 invoices/day off-cycle, 1% for an hour is well under one invoice, so zero errors at that step is no evidence at all, and percentage thresholds on such small counts are meaningless.",
+        "Replaces percentage steps with tenant-based cohorts: internal or test tenants first, then a small named set of willing or low-risk tenants, then larger batches. It notes that the flag targets by tenant_id (or a tenant_id hash), and it chooses early cohorts deliberately, for example covering, or deliberately holding back, Canadian HST/PST and US marketplace-facilitator tenants, rather than at random.",
+        "Proposes a shadow or dual-run phase in which v2 computes tax for real invoices without anything being sent, and the results are diffed against v1 before any customer gets a v2 invoice. Because correctness here can be checked without exposing anyone, it treats a near-zero unexplained diff rate (beyond the known bugs v2 is meant to fix) as the main promote gate.",
+        "Measures steps in invoices and billing cycles rather than hours: it requires at least one 1st-of-month run (~6,500 invoices) at a meaningful cohort size before full rollout, and checks that the timeline to Q1 contains enough month-end runs (from late October only the Nov 1, Dec 1 and Jan 1 runs remain) for that.",
+        "Sets promote / hold / roll-back rules in advance with concrete thresholds suited to small counts. Examples: an absolute count or zero-tolerance for tax mismatches above a cent or tolerance; generation failures compared with the ~0.6% baseline using counts, not percentages; Taxwell API errors or timeouts. At least one condition automatically flips the tenant(s) back to v1 or holds the nightly send.",
+        "Names what flag-off does not undo: invoices already emailed to customers' billing contacts and synced to QuickBooks/Xero cannot be recalled, so a wrong tax amount needs a credit-note or reissue process. It also says rows written to invoice_line_tax_v2 must still render after rollback, so the PDF renderer and any re-render path must keep reading v2 invoices, and the v2 table cannot simply be dropped.",
+        "Assigns who watches (for example, billing on-call plus finance or tax ops sign-off on the diff report before each promotion), and names an owner plus a date for removing the flag and retiring the v1 tax code once v2 is at 100% and has run through at least one cycle.",
+        "Does not invent per-tenant invoice volumes, Taxwell latency or error SLAs, or the current tax-mismatch rate, none of which are given. Where the plan depends on them, it says what it assumed or how to measure it during the shadow phase."
+      ]
+    },
+    {
+      "id": 6,
+      "held_out": "2026-10-20",
+      "prompt": "We're about to turn on Smart Reminders in the Larkspur habit app (iOS + Android). The flag is server-side in notification-svc, config is in flags/smart_reminders_v2.yaml and how pushes go out is in notes/push-pipeline.md. Client support for it shipped in 4.12. Roughly 60% of our DAU are on 4.12 or later, the rest are spread across 4.8-4.11 and a chunk of those never update. PM wants it on for everyone before the holiday campaign kicks off mid-November. Can you write me the rollout plan? Honestly my biggest worry is people feeling spammed and turning off notifications for the whole app, because we can't get that permission back.",
+      "assertions": [
+        "Flags that the ramp rule as written would expose users on app versions below 4.12, and requires the targeting to gate on client app version >= 4.12 (so percentages are of eligible users, not all DAU).",
+        "Flags that bucketing by device_id lets one user with two devices land in both arms, and recommends bucketing by user id so exposure is sticky per person.",
+        "States that turning the flag off does not cancel reminders already planned into push_outbox for up to the next 24h (the flag is checked at planning time, not send time), and gives a concrete step to purge or re-plan queued sends as part of rollback.",
+        "States plainly that pushes already delivered and OS-level notification permission lost cannot be undone by the flag, and treats notification opt-out / permission-revoked rate per exposed user, measured against a control cohort on the same app versions, as a primary guardrail with an explicit hold/roll-back threshold.",
+        "Includes a hard automatic safeguard on send volume (e.g. a per-user daily cap or circuit breaker on smart nudges per user per day) independent of human watching, given smart mode can send up to 3 nudges/day per habit.",
+        "Sets a minimum dwell time per step of at least one full week (or justifies an equivalent), because habit and opt-out behaviour follows a weekly cycle and opt-outs lag sends, and checks that the resulting ladder actually fits before mid-November, saying plainly if it does not.",
+        "Does not state invented figures as fact (e.g. a specific DAU count, a baseline opt-out rate or a retention number the user never gave); where a baseline is needed it says to measure it from current data or marks it as an assumption.",
+        "Names who removes the flag and the fixed-time code path, and by when, and notes the removal is constrained by users still on pre-4.12 builds."
+      ]
+    },
+    {
+      "id": 7,
+      "held_out": "2026-10-20",
+      "prompt": "We're rolling out rate limiter v2 on our API gateway (design in docs/ratelimiter-v2.md, recent traffic in ops/traffic-summary.txt). It has to be enforcing for every tenant by Monday 2 November because that's when the new enterprise contract limits take effect. Platform on-call is one person per week, handover Monday 10:00 UTC; this week it's Priya, next week it's Tomasz who just joined the rota. My manager wants to go straight to 50% tomorrow so we 'have plenty of runway'. Is that crazy? If so, what should the ramp look like instead, and what do we watch?",
+      "assertions": [
+        "Answers the 50%-tomorrow question directly near the top with a clear verdict and the specific reasons (not just generic caution), and offers a plan that still meets the 2 November deadline.",
+        "Flags that the flag's default bucketing by request_id would split each tenant's traffic between v1 and v2 (inconsistent limits per tenant, meaningless per-tenant comparison) and requires bucketing by tenant_id.",
+        "Recommends a shadow / log-only phase where v2 evaluates on all or most traffic and records would-have-throttled decisions without returning 429s, compared against v1 throttles, before any enforcement.",
+        "Recognises that percentage of tenants is not percentage of traffic (top 20 tenants carry 71% of requests) and handles the large and enterprise tenants deliberately, e.g. via allow/deny lists or a separate step, rather than leaving them to the random draw.",
+        "Treats the shared redis-gw cluster (58% CPU at peak, also serving session cache) and the fail-open path as rollout risks, with guardrails such as Redis CPU/latency, rl_v2.redis_error rate and added gateway p99 latency, and automatic abort or hold thresholds on them.",
+        "Uses the on-call rota concretely: places ramp steps on weekday working hours with the named watcher, avoids ramping across the Monday handover or into a weekend, and accounts for Tomasz being new (e.g. completing risky steps during Priya's week or pairing).",
+        "Says what turning the flag off does not undo: 429s already returned to tenants, their client retries/backoff or broken integrations, and any customer communication about the new limits; and notes that the rl_v2.* metrics must be wired to dashboards before the first step.",
+        "Does not invent numbers the user did not provide (e.g. a current 5xx rate, a v2 false-positive rate, or tenant-specific limits) and states them as figures to be measured; any thresholds proposed are framed relative to the v1/control baseline."
+      ]
+    }
+  ]
+}

package/packs/dev-team/skills/rollout-plan/evals/files/2/metrics-snapshot.txt

@@ -0,0 +1,11 @@
+Basketful — orders dashboard, trailing 28 days (exported 2026-10-14)
+
+Users who open the home screen with a past order:   1,200 per hour (average), 2,600 peak (18:00-20:00)
+Order placement failure rate (all orders):          1.5%
+Home-screen-to-order conversion (users with a past order): 18%
+POST /v2/orders p95 latency:                        640 ms
+POST /v2/orders p99 latency:                        1,900 ms
+Payment declines (share of charge attempts):        2.2%
+Substitution complaints (support tickets per 1,000 orders): 4.1
+
+Note: no metric yet splits orders by `source`.

package/packs/dev-team/skills/rollout-plan/evals/files/2/pr-description.md

@@ -0,0 +1,28 @@
+# PR #874 — One-tap reorder
+
+**Author:** Basketful app team · **Flag:** `one_tap_reorder` · **Ships in:** iOS/Android 5.8 and web
+
+## What it does
+
+Adds a "Reorder" button to past orders on the home screen. One tap rebuilds the basket from the
+previous order, charges the saved card and places the order.
+
+## How it works
+
+- New endpoint `POST /v2/orders/reorder`. The web client and app 5.8+ call it when the flag is on.
+- The flag is evaluated **in the client**. The app fetches flag config at launch and caches it
+  for 24 hours. Web fetches it on page load.
+- On success the order is sent to the fulfilling store partner (FreshPick) through the existing
+  order webhook, and the user gets a push notification: "Your reorder is on its way".
+- Reorders are tagged `source=one_tap` in the `orders` table (existing `source` column; new value).
+  The order-history screen in app versions before 5.8 shows unknown `source` values as "Other".
+- The basket rebuild substitutes out-of-stock items with the store's default substitute.
+
+## Rollout
+
+Product would like 1% on Monday, 100% by Thursday.
+
+## Notes
+
+- The home screen is also running the `home_carousel_v3` experiment (50% of users, until 2 Nov).
+- Business accounts (office grocery orders) are on a contract with a 2-week notice for UI changes.

package/packs/dev-team/skills/rollout-plan/evals/files/4/PR_DESCRIPTION.md

@@ -0,0 +1,24 @@
+# PR #4182: Pay-in-4 at checkout (Lumen Pay integration)
+
+Adds "Pay in 4 interest-free payments" as a payment option on web checkout, behind
+flag `checkout.pay_in_4` (our in-house flag service, supports % of user_id and
+allow-lists).
+
+## What it does
+- New payment tile on /checkout/payment when cart total is between $50 and $1,000.
+- On submit we call Lumen Pay `POST /v2/agreements` which creates an installment
+  agreement and runs a soft credit check. Lumen charges the first installment immediately.
+- Order is written with `payment_method = 'installment'` (new enum value) and a
+  `lumen_agreement_id`. Refunds for these orders go through Lumen's refund API, not Stripe.
+- Lumen sends the customer its own welcome + schedule email on agreement creation.
+- Order confirmation email template has a new installment-schedule block.
+
+## Not in this PR
+- iOS/Android apps (they hide the tile; app support lands in 6.14).
+- Admin tool support for installment refunds is in #4190 (not merged yet; CS can
+  refund via the Lumen dashboard in the meantime).
+
+## Notes
+- Lumen asked us to stay under 2,000 new agreements/day for the first 7 days while they
+  scale their risk model for us.
+- Migration 0412 adds the enum value; already applied to prod.

package/packs/dev-team/skills/rollout-plan/evals/files/4/metrics_snapshot.txt

@@ -0,0 +1,7 @@
+Checkout funnel, web only, trailing 28 days (from Looker, pulled 2026-10-19)
+
+checkout sessions / hour       avg 41,000   (peak ~68,000 at 20:00-22:00 ET, trough ~9,000 at 04:00 ET)
+session -> order conversion     3.1%
+payment step errors (any 4xx/5xx on /checkout/payment submit)   0.42% of submits
+p95 payment submit latency      1.8 s
+orders with cart total $50-$1,000   about 61% of orders

package/packs/dev-team/skills/rollout-plan/evals/files/5/docs/tax-engine-v2.md

@@ -0,0 +1,24 @@
+# Design note: Tax engine v2 for invoice generation
+
+Owner: Priya Okonkwo (Billing platform)
+
+## Why
+The current invoice tax calculation is hand-rolled and gets Canadian HST/PST
+combinations and US marketplace-facilitator states wrong in edge cases. v2 calls
+Taxwell (external tax API) per invoice line and stores the result.
+
+## Change
+- `InvoiceBuilder` gets a `tax_engine` switch read from flag `billing.tax_engine_v2`
+  (flag can target by tenant_id; it cannot target by %, but we could hash tenant_id).
+- v2 writes line items to the new `invoice_line_tax_v2` table (jurisdiction breakdown
+  per line). v1 code does not read this table. The invoice PDF renderer reads
+  whichever table the invoice was built with.
+- Finished invoices are rendered to PDF and emailed to the tenant's billing contacts
+  by the existing nightly job, and synced to the tenant's accounting system
+  (QuickBooks / Xero connectors) if they have one connected.
+
+## Volume
+- ~340 paying tenants. Most invoices are generated on the 1st of the month
+  (about 6,500 invoices that day); the rest of the month is usage-based and
+  mid-cycle upgrades, roughly 40-60 invoices/day.
+- Current invoice generation failure rate (job errors needing manual re-run): ~0.6%.

package/packs/dev-team/skills/rollout-plan/evals/files/6/flags/smart_reminders_v2.yaml

@@ -0,0 +1,14 @@
+# Evaluated in notification-svc (server-side). Owner: growth-eng
+key: smart_reminders_v2
+description: Model-chosen reminder send times instead of the user's fixed time
+default: false
+rules:
+  - name: staff
+    match:
+      user.email_domain: ["larkspur.app"]
+    serve: true
+  - name: ramp
+    percentage:
+      value: 0
+      bucket_by: device_id
+    serve: true

package/packs/dev-team/skills/rollout-plan/evals/files/6/notes/push-pipeline.md

@@ -0,0 +1,10 @@
+# Reminder push pipeline (notification-svc)
+
+- Scheduler job runs every 15 min and plans each user's reminders for the next 24h.
+- Planned sends are written to the `push_outbox` table with a send_at timestamp,
+  then picked up by the sender workers (FCM for Android, APNs for iOS).
+- The flag is checked by the scheduler at planning time, not by the sender.
+- Smart reminders can schedule up to 3 nudges/day per habit (fixed-time mode: 1/day per habit).
+- Deep link `larkspur://habit/{id}/checkin?src=smart` — handled by the app from 4.12.
+  Older builds open the app home screen and ignore the `src` param.
+- Notification category `SMART_NUDGE` (with "Snooze 1h" action) registered by the client from 4.12.

package/packs/dev-team/skills/rollout-plan/evals/files/7/docs/ratelimiter-v2.md

@@ -0,0 +1,21 @@
+# Rate limiter v2 (api-gateway)
+
+## Why
+v1 is an in-memory token bucket per gateway pod. With 18 pods behind the LB a tenant's
+effective limit is roughly 18x the configured one and moves with autoscaling. Enterprise
+contracts signed for Q4 specify hard per-tenant limits, so we need a global limit.
+
+## Design
+- Sliding-window counter per (tenant_id, route_group) in the shared Redis cluster
+  (`redis-gw`, also used for session cache).
+- One INCR + EXPIRE pipeline per request, ~1 ms budget.
+- On Redis error or timeout (> 5 ms): fail open, allow the request, emit `rl_v2.redis_error`.
+- Returns 429 with `Retry-After`.
+- Flag: `gateway.ratelimit_v2` in our flag service. Supports percentage rollout,
+  bucketing key configurable (default: request_id), and per-tenant allow/deny lists.
+
+## Metrics already emitted
+- `gw.requests` / `gw.responses{status}` per tenant
+- `gw.latency_ms` p50/p95/p99
+- `rl_v1.throttled` per tenant
+- `rl_v2.*` - not yet wired to dashboards

package/packs/dev-team/skills/rollout-plan/evals/files/7/ops/traffic-summary.txt

@@ -0,0 +1,9 @@
+api-gateway traffic, last 30 days (exported from dashboard 2026-10-19)
+
+peak rps (weekday 15:00-17:00 UTC):     ~4,200
+overnight trough rps:                    ~600
+active tenants:                          ~1,150
+top 20 tenants share of requests:        71%
+tenants on enterprise plan:              37
+rl_v1 429s / day (all tenants):          ~2,900
+redis-gw CPU at peak:                    58%

package/packs/dev-team/skills/rollout-plan/examples/sync-engine-workspaces.md

@@ -0,0 +1,102 @@
+# Example — new sync engine, ramped by workspace
+
+A worked input → output. The company, product and numbers are fictional.
+
+## Input
+
+> Fieldnote is moving note sync to the new engine behind the flag `sync_v2`, evaluated
+> server-side per workspace. Desktop and mobile 3.4 contain both engines; the server tells each
+> client which to use. About 20,000 sync sessions an hour on average, peaking at 34,000 around
+> 10:00 UTC on weekdays. Sync failure rate is 1.2% of sessions; p95 sync latency is 900 ms.
+> Notes edited under `sync_v2` are stored in a new op-log format that the old engine can't read
+> until `oplog_export` converts them back (about 1 second per note). Business-plan workspaces
+> have a 99.9% sync availability SLA. A pricing experiment runs on 10% of Free workspaces until
+> 30 Oct. Plan the rollout.
+
+## Working (not shown in the reply)
+
+```bash
+python3 scripts/canary_math.py --baseline 1.2% --detect-abs 0.3pp --traffic 20000 --unit sessions --steps 1,5,25,50,100
+```
+
+| Exposure | Treatment sessions needed | Minimum time | Status |
+|---|---|---|---|
+| 1% | 11,199 | 56.0 h | ok |
+| 5% | 11,718 | 11.7 h | ok |
+| 25% | 15,137 | 3.0 h | ok |
+| 50% | 23,228 | 2.3 h | ok |
+| 100% | — | n/a | no concurrent control |
+
+## Output
+
+**Internal → 1% → 5% → 25% → 50% → 100% over about 7 days. The real risk is the rollback, not
+the ramp: every note edited under the new engine needs converting back before the old engine can
+read it, so each step makes turning the flag off slower.**
+
+**Binding constraint:** rollback cost, which grows with every note edited on `sync_v2`; traffic is
+high enough for the default ladder. At 20,000 sessions an hour and a 1.2% failure rate, even 1% sees
+a 0.3 pp rise in about 2.5 days, so no step is crash-only. The pricing experiment holds its
+workspaces out until 30 Oct.
+
+Assumptions: the smallest regression worth catching is +0.3 pp in sync failures `[assumption]`;
+the flag stays per workspace, so every member of a workspace is on one engine.
+
+### Metrics
+
+All thresholds compare flag-on workspaces with flag-off workspaces in the same hours.
+
+| Metric | Role | Baseline | Threshold | Abort (any time) |
+|---|---|---|---|---|
+| Sync failure rate (share of sessions) | decision | 1.2% | ≤ control + 0.3 pp | > 3× control for 10 min |
+| p95 sync latency | guardrail | 900 ms | ≤ control + 150 ms `[assumption]` | > control + 2 s for 10 min |
+| p99 sync latency | guardrail | to confirm | ≤ control + 400 ms `[assumption]`, read at 10,000+ sessions per arm | |
+| Op-log conversion backlog | guardrail | none today | `oplog_export` dry-run on the step's notes finishes inside 1 h | |
+
+Session counts overstate the evidence: sessions from one workspace are correlated, so the
+times below are floors.
+
+### Ladder
+
+| Step | Exposure | Who and why | Minimum time | Can catch |
+|---|---|---|---|---|
+| 0 | internal | Fieldnote's own workspaces: crashes and format bugs at no customer cost | 24 h | crashes only |
+| 1 | 1% | Free and Pro workspaces by `hash(salt + workspace_id)`; excludes Business (SLA) and the pricing-experiment workspaces | 56 h (covers two weekday 10:00 peaks) | ≥ 0.3 pp |
+| 2 | 5% | same rule, nested | 24 h (one full peak; statistics need 11.7 h) | ≥ 0.3 pp |
+| 3 | 25% | same rule; pricing-experiment workspaces join after 30 Oct | 24 h | ≥ 0.3 pp |
+| 4 | 50% | Business workspaces join here | 24 h | ≥ 0.3 pp |
+| 5 | 100% | everyone; 2% holdback of Free workspaces kept 7 days for the failure-rate comparison | 7 days | holdback only |
+
+### Step rules
+
+- **Promote** when the minimum time is reached and every guardrail is inside its threshold.
+- **Hold** for one extension of 24 h if the failure rate sits between threshold and abort; still there → roll back.
+- **Roll back** on any threshold breach at the planned sample, or any abort.
+- **Abort:** automated if the flag service can turn `sync_v2` off on a metric (to confirm); otherwise manual, with the abort rule as a page to sync on-call.
+- **Watcher:** sync on-call. **Override:** only the sync engineering lead may promote past a breach, with the reason in the rollout channel.
+
+### Rollback
+
+**Flag off:** sync on-call, in the flag service; clients switch engine at their next sync.
+
+| Category | What this change does | Undone by flag-off? | Handling |
+|---|---|---|---|
+| Data written in a new format | Notes edited under `sync_v2` are stored as op-log | **no** | Turn the flag off, then run `oplog_export` on the flag-on workspaces at once; their v2-edited notes stay unreadable to the old engine until it finishes. Rehearse it on step 0 data |
+| Caches | Clients cache the engine choice until next sync | yes, at next sync | none |
+| Clients already shipped | 3.4 clients carry both engines; the server chooses | yes | Clients older than 3.4 never get `sync_v2`, verified at step 0 |
+
+**Point of no return:** none, provided `oplog_export` is rehearsed. The cost of going back grows
+with every step: at about 1 second per note, rolling back 50% of workspaces takes as long as
+converting every note they have edited since step 1. Before step 4, time the conversion on the
+step 3 cohort and hold if it exceeds 1 hour.
+
+### Flag removal
+
+- **Owner:** sync engineering lead · **Date:** 30 days after 100% · **Gate:** 100% for 14 days, no rollback.
+- **Removes:** the `sync_v2` flag, the old sync engine on the server, `oplog_export` once no old-format reader remains, and the flag's dashboard.
+- **Clients:** the old engine leaves the apps in the first release after removal; the server path stays until 3.4 is the minimum supported version.
+
+### To confirm
+
+- p99 sync latency baseline.
+- How many notes an average workspace edits per day, to size the rollback time at each step.
+- Whether the flag service can turn the flag off automatically on a metric, or only on a manual flip.

package/packs/dev-team/skills/rollout-plan/manifest.json

@@ -0,0 +1,35 @@
+{
+  "name": "rollout-plan",
+  "version": "0.2.0",
+  "description": "Plan the progressive release of a code change behind a feature flag or canary — the exposure ladder (internal, beta cohort, 1%, 5%, 25%, 50%, 100%, or what the traffic supports), and for each step who is exposed, decision and guardrail metrics with a baseline and a threshold against a control cohort, the minimum traffic and time before the numbers mean anything, a promote / hold / roll back rule set in advance, who watches and what aborts automatically; then what turning the flag off does not undo, and who removes the flag by when. Use when the user asks for a rollout plan, canary plan, ramp schedule, progressive delivery or percentage rollout, \"how long should the canary run\", \"when do we go from 5% to 25%\", or how to turn on a flagged feature safely.",
+  "entrypoint": "SKILL.md",
+  "deps": { "npm": [], "pip": [] },
+  "env": { "required": [], "optional": [] },
+  "related": [
+    "migration-plan",
+    "launch-readiness",
+    "observability-plan",
+    "success-metrics",
+    "runbook-generator"
+  ],
+  "handoff": [
+    {
+      "to": "migration-plan",
+      "when": "the change also alters a schema, backfills data, swaps a datastore or dual-writes",
+      "purpose": "Plan the expand, migrate and contract phases and the backfill, so the rollout only ramps exposure of a code path whose data is already in place.",
+      "fallback": "If migration-plan is absent, state inline that the schema or data change must be expanded and verified before step 1, list it under To confirm, and keep it out of the ladder."
+    },
+    {
+      "to": "launch-readiness",
+      "when": "the rollout plan is written and someone must decide whether the change may go live at all",
+      "purpose": "Judge the go/no-go against evidence: rollback tested, alerts and runbooks in place, targets measured, comms ready.",
+      "fallback": "If launch-readiness is absent, end the plan with the open preconditions for step 1 (rollback rehearsed, guardrail metrics instrumented, abort wired) as a short checklist with an owner role each."
+    }
+  ],
+  "tags": ["rollout", "feature-flags", "canary", "progressive-delivery", "release"],
+  "permissions": { "network": [], "commands": [], "files": "none", "notes": "canary_math.py takes numbers as arguments and prints to stdout; it reads and writes no files." },
+  "model": {
+    "tier": "standard",
+    "rationale": "Plan synthesis on a strong template; the arithmetic of how long each exposure step must run is done by its script, not the model."
+  }
+}

package/packs/dev-team/skills/rollout-plan/reference.md

@@ -0,0 +1,100 @@
+# rollout-plan — reference
+
+## How long a step must run
+
+A step compares a flag-on cohort (treatment, share `e` of traffic, rate `p2` if the regression
+is real) with a concurrent flag-off cohort (control, share `c`, baseline rate `p1`). With `T`
+units per hour, after `t` hours the arms hold `n1 = T·e·t` and `n2 = T·c·t`. The two-proportion
+z-test (normal approximation) is powered when
+
+```
+t = [ ( zα · √( p̄(1−p̄)(1/e + 1/c) ) + zβ · √( p2(1−p2)/e + p1(1−p1)/c ) ) / |p2 − p1| ]² / T
+p̄ = (e·p2 + c·p1) / (e + c)
+```
+
+`zα` is 1.96 for a two-sided test at α = 0.05 (1.645 one-sided), `zβ` is 0.84 for power 0.8.
+For two equal arms this reduces to the familiar units-per-arm figure
+`n = (zα√(2p̄(1−p̄)) + zβ√(p1(1−p1) + p2(1−p2)))² / (p2 − p1)²`.
+
+What the formula says in practice:
+
+- **The small arm sets the clock.** At 1% exposure the control is enormous and adds almost no
+  information; time is roughly `n_treatment / (T · e)`. Halving exposure doubles the time.
+- **50/50 is the fastest comparison.** Past 50% the control becomes the small arm.
+- **Small baselines are slow.** Catching +0.3 pp on a 0.5% error rate needs about 11,000 units
+  per arm; catching +0.3 pp on a 30% rate needs about 367,000, because variance grows with `p(1−p)`.
+  Relative regressions on rare events ("errors up 10%") need the most data of all.
+- **Two-sided is the default** because a change can also look suspiciously good (a broken
+  error counter reads as "fewer errors"). Use `--one-sided` only when the user accepts that.
+
+## When the formula is the wrong tool
+
+The formula assumes a step sees hundreds of events. When it sees a handful, change the rule,
+not the step length:
+
+- **Expected count first.** Exposure × volume per unit time. ✅ *"5 accounts × 2 statements a
+  day = 10 statements a day"*. If the step expects fewer than one event, it is a smoke test and
+  says so.
+- **Zero-tolerance or count limits.** For correctness (a total, a price, a tax amount) one
+  unexplained difference is the signal: hold or roll back on the first. For failure rates, compare
+  counts with what the baseline predicts: at a 0.5% baseline, 400 runs predict 2 failures; 6 or
+  more is a breach (Poisson, about 1-in-60 by chance).
+- **Shadow beats exposure** when the old and new answers can be compared without sending anything:
+  every real unit is evidence, nobody is exposed, and the gate is the diff report.
+- **Cycles beat hours** when the volume is batched: one monthly run can carry more evidence than
+  the rest of the month together. Count the runs left before the deadline before drawing the ladder.
+
+## The randomisation unit
+
+Count `--traffic` in the unit the flag assigns. When a user-level flag is judged on a
+request-level metric, one user's requests are correlated, so the effective sample is smaller
+than the request count; the script's figure is then a floor. Either count users and use a
+per-user metric ("share of users who hit an error"), or say the time is a lower bound.
+
+## Threshold patterns
+
+| Metric | ✅ Testable | ❌ Untestable |
+|---|---|---|
+| Error rate | flag-on ≤ flag-off + 0.3 pp at 11,000 requests per arm | "no increase in errors" |
+| Latency | p95 flag-on ≤ flag-off + 40 ms; p99 ≤ flag-off + 120 ms once 10,000+ requests per arm | "latency stays healthy" |
+| Saturation | DB pool in use ≤ 80% at peak in the flag-on hosts | "watch the database" |
+| Business | booking conversion flag-on ≥ flag-off − 1 pp | "no impact on revenue" |
+| Abort | flag-on 5xx > 3× flag-off for 5 consecutive minutes | "if things go badly wrong" |
+
+Abort triggers are wide and fast, sized to catch a disaster from small samples. Promotion
+thresholds are tight and slow, read once at the planned sample. Mixing the two produces either
+an abort that fires on noise or a promotion read on noise.
+
+Tail percentiles need samples: an estimate of p99 rests on the 1% above it, so 10,000 requests
+leave about 100 points in the tail; p95 at 2,000 leaves about 100 too. Below those counts, read
+p95 or the median and say so.
+
+## Cohort rules
+
+- **Internal first** catches crashes and obvious breakage at no customer cost. It is not
+  evidence about rates: staff behave differently and are few.
+- **Nested hashing** (`hash(salt + unit_id) mod 1000 < k`, with `k` rising) keeps everyone
+  from earlier steps exposed, so nobody flips back and forth.
+- **Use a fresh salt per flag.** Reusing another flag's salt puts the same users in every 1%.
+- **SLA and regulated customers** join at 50% or later unless they opted in; contractual notice
+  periods can make even that a comms item.
+- **Overlapping experiments** on the same surface confound both readings. Exclude, or make the
+  allocations orthogonal and analyse by the other experiment's arm.
+
+## What turning the flag off does not undo
+
+| Category | Typical residue | Handling to plan |
+|---|---|---|
+| Data written in a new format | New-format JSON, enum values, encodings | Old code reads the new format before step 1, tested; or a cleanup job with an owner |
+| Rows, events, jobs | Orders, records, queued jobs, analytics events | Jobs check the flag at run time, or are safe to run after flag-off |
+| Messages | Emails, push, SMS, webhooks | No recall; a correction template ready; daily volume cap in early steps |
+| Money and third parties | Charges, refunds, partner orders, per-call billing, filings | The reversal API or procedure, or written as permanent |
+| Caches | Entries keyed the same as the old path | Versioned keys, or TTL stated as the rollback's tail |
+| Shipped clients | Mobile and desktop builds with the path compiled in | A server-side switch; flag config TTL stated |
+| Schema expansions | Columns, tables, indexes | Stay; contracted by the migration plan |
+| Search indexes, ML features | Re-indexed documents, logged features used for training | Reindex job or a training-data exclusion window |
+
+A **point of no return** exists when a step's residue changes what flag-off restores for
+the people already exposed: data old code can't read, a partner contract triggered, a one-way
+message campaign. Schedule it as late as the ladder allows and list what is verified just
+before it.