@clipboard-health/ai-rules 2.31.0 → 2.33.0

package/skills/cb-ship/references/pr-template.md

@@ -17,5 +17,5 @@ Concisely explain the user intent from session history and the meaningful behavi
 
 Optional, don't fabricate: ticket links, rollout plan, residual risk, or specific areas for reviewers to focus.
 
-<sub>🤖 <code>cb-ship:created v1 core@3.12.0</code></sub>
+<sub>🤖 <code>cb-ship:created v1 core@3.14.0</code></sub>
 ```

package/skills/flaky-critic/SKILL.md

@@ -0,0 +1,68 @@
+---
+name: flaky-critic
+description: Adversarial reviewer for flaky-test fix plans and PRs, enforcing the flaky-fix rubric. Gates implementation tickets from Triage to Todo (shadow mode posts verdicts without moving tickets). Use when running the recurring flaky-critic task, when asked to review a flaky-fix plan or PR against the rubric, or to backtest verdicts against historical tickets.
+---
+
+The quality gate of the flaky-test pipeline. Implementation tickets created by investigation agents land in the **Triage** state; this skill reviews each plan against `references/rubric.md` and decides whether it dispatches. It replaces per-plan human review; the human sees only disagreements, bounces, and the audit sample.
+
+Your job is adversarial: assume the plan papers over the root cause until its evidence proves otherwise. You route and gate — you never redesign the fix yourself.
+
+## Modes
+
+- **shadow** (default): post the verdict comment only. Never change ticket state. The human still gates; agreement between your verdicts and their actions is the promotion metric.
+- **enforce**: act on verdicts per Phase 3. Only when the task description explicitly says enforce.
+- **backtest**: verdicts on historical plans provided as files or ticket IDs; write nothing anywhere. Judge ONLY from the plan content — never look up ticket states, comments after the plan, linked PRs, or how things actually resolved.
+
+## Rules
+
+- Load `references/rubric.md` (relative to this SKILL.md) before judging anything. Every rejection MUST cite rule IDs and quote the offending plan sentence or diff hunk. Every approval MUST name the fix class (A1–A7, C1–C5, D1–D5) and the evidence that satisfies its requirements.
+- One verdict per ticket per content-state: skip tickets whose latest `<!-- flaky-critic:` marker is newer than the last substantive edit/comment.
+- Findings from automated reviewers (Mendral, CodeRabbit) are advisory inputs, never binding (rubric §1).
+- A plan outside the rubric's taxonomy is **needs-human**, not a guess. Uncertainty bounces up, never through.
+- Verdicts are about the plan in front of you; do not penalize brevity when the required evidence is present, and do not reward thoroughness that lacks it.
+
+## Phase 1: Queue (shadow/enforce only)
+
+Linear: Groundcrew project, label `flaky-implementation`, state `Triage`. Skip already-marked tickets per the rule above. For shadow/enforce runs, fetch the linked investigation ticket (the plan's evidence usually lives there) and note any linked PRs. Use `list_issues` only to enumerate; fetch plan content with `get_issue` — list results truncate descriptions.
+
+In backtest mode, skip the live queue and linked metadata lookups. Use only the supplied historical plan content or ticket snapshot; do not fetch linked investigation tickets, linked PRs, current ticket state, comments after the plan, or how the work actually resolved.
+
+If the queue is empty, skip to the digest. Fetch linked investigation tickets and linked PR metadata in parallel when the available tooling supports it.
+
+## Phase 2: Verdict per ticket
+
+1. **Classify** the proposed fix into the rubric taxonomy. Multi-part plans get classified per part; the worst verdict wins.
+2. **B-rules first** (any violation → reject): B1–B7, per rubric §2.
+3. **A/C requirements**: the named fix class's "Required evidence" must be present (A-rules), or the conditional justification stated verbatim (C-rules). A1 has a shared-helper carve-out that's easy to miss — check it explicitly before approving.
+4. **D dispositions**: a no-code plan must carry the disposition's required evidence (D1–D5).
+5. **Verdict**: `approve` / `reject` / `needs-human`, with confidence high/moderate/low. Low confidence → needs-human.
+
+## Phase 3: Act
+
+Post the verdict comment on the implementation ticket:
+
+```text
+[shadow] VERDICT: reject — B2, B6
+- B2: "increase the navigation timeout to 30s" — timeout inflation without a root-cause statement of why the operation is legitimately slow.
+- B6: plan states confidence 3/5 but has no observability-to-reach-5/5 section.
+Fix class claimed: A1 (readiness gate). Missing required evidence: polled signal is not shown to be the one the failing UI path uses.
+<!-- flaky-critic: verdict=reject mode=shadow -->
+```
+
+- **shadow**: comment only, prefixed `[shadow]`.
+- **enforce**: approve → move the ticket to Todo and assign it to the Linear API user. Reject → comment and leave in Triage; the investigating flow gets one bounce. A second reject on the same ticket → comment `needs-human` and stop touching it.
+- **needs-human**: comment with what the rubric cannot answer; never move the ticket.
+
+## Phase 4: Taste capture
+
+When you encounter a ticket where a human's action contradicts your prior verdict (you rejected, they released; you approved, they bounced), post a follow-up comment containing a drafted rubric amendment — the rule change that would have made your verdict match theirs — marked `<!-- flaky-critic: amendment-proposal -->`. The human approves or discards; you never edit the rubric yourself.
+
+## Phase 5: Digest
+
+End with: tickets reviewed, verdicts by type with rule-ID counts, agreement events observed (human actions vs prior verdicts), amendment proposals drafted, and tickets skipped as already-current. When run as a recurring task, this digest is the run's final message.
+
+## PR checkpoint (when the task description asks for it)
+
+Sweep open pipeline-authored flaky-fix PRs the same way, using the rubric's diff-level detection heuristics: B-rule patterns on the diff itself, the claimed A-class's required evidence in the PR body, attempt-to-fix key present, current-main status stated (B7). Shadow = review comment only. Never merge; merge rights are out of scope.
+
+When posting PR feedback, follow `../simple-review/references/posting-pr-review.md`: use one GitHub Review with event `COMMENT`, anchor actionable comments to changed lines, and keep approval decisions for humans.

package/skills/flaky-critic/references/rubric.md

@@ -0,0 +1,224 @@
+# Flaky-Fix Rubric
+
+Mined from the [Groundcrew project](https://linear.app/clipboardhealth/project/groundcrew-4c31c63c5ed4), `flaky-test-fix` PRs across ClipboardHealth repos, and the `flaky-debug` / `flaky-test-bulk-debugger` skill docs.
+
+## 1. Purpose & how the critic uses this doc
+
+This rubric turns Rocky's observed review standards into checkable rules for flaky-test fix plans and PRs. When rejecting or flagging, the critic MUST cite the specific rule ID (e.g., "violates B1") and quote the offending diff hunk or plan sentence. A plan/PR that violates any B-rule is rejected; A-rules with their required evidence are auto-merge candidates; C-rules require the stated justification to be present verbatim in the plan or PR body.
+
+Findings from automated reviewers (Mendral, CodeRabbit, etc.) are **advisory inputs, not binding rules**: the critic evaluates each on rubric merits and may decline one with stated reasoning, exactly as cb-babysit's Disagree replies do.
+
+**Statement-missing vs substance-missing (backtest amendment, 2026-06-11):** B5/B6/B7 each require an explicit statement in the plan (dedup-check results, confidence score plus observability section, current-main status). When the _statement_ is absent but the violation is not demonstrated — no actual duplicate found, no actual low-confidence code fix, no actual staleness — the verdict is reject with disposition **amend-and-resubmit**: one bounce asking for exactly the missing sections, never escalation. A demonstrated violation (a real duplicate, a real ≤2/5 code fix) is a substantive reject. The 2026-06-11 backtest showed 6 of 7 false rejections were statement-missing on plans predating these conventions; live plans carry the sections by template, but the distinction stays load-bearing for the agreement metric.
+
+## 2. Banned patterns
+
+### B1 — Naked retry (retry without failure classification)
+
+Adding retries that repeat on _any_ failure, or on a failure class that includes genuine errors.
+
+**Why:** "Do not solve this by adding frontend waits or Playwright retries; that increases duplicate token setup and makes the Cognito throttle worse." (STAFF-1122). Org reviewer pushback on PR cbh-admin-frontend#6974: retrying all 422s means "a real validation error will now spin for 90s before timing out instead of failing fast" — the fix was narrowed so only opaque 500s retry while 422s stay gated on the specific readiness message. The skill's idempotency check (plan.md "Decide Fix Approach") must pass before any retry/wait change: safe to repeat, same scenario, doesn't amplify the root cause, no deterministic signal available instead.
+
+**Detection heuristic:** diff adds `retry`, `retries:`, `toPass`, `async-retry`, a retry loop, or `page.reload()` inside a loop, AND the retry predicate does not discriminate transient failures (matches any error / any non-2xx / bare `catch`), OR the PR body lacks (a) why the retried operation is idempotent and (b) why no deterministic signal exists. Special case: retrying token minting, one-time credentials, seed-data creation, or rate-limited APIs without a concurrency cap is an automatic reject (retry amplification).
+
+### B2 — Timeout/sleep inflation
+
+Increasing a numeric timeout, adding `waitForTimeout`/`sleep`, or extending a fixed delay as the fix.
+
+**Why:** plan.md: "Is there a deterministic signal to wait on instead of a longer timeout?" Accepted plan STAFF-1010 explicitly chose a readiness poll "over increasing the existing fixed CDC sleep." The one approved "increase timeout" PR (cbh-mobile-app#11242) was approved only after the diff morphed into removing the slow operation (`user.type()` of ~95 keystrokes → `fireEvent.change`) instead of raising the timeout. Human reviewer pushback on closed #11456: "I am seeing a lot of timeout overrides, waitForResponse should have handled this by itself."
+
+**Detection heuristic:** diff increases any numeric timeout value, adds `waitForTimeout`, `setTimeout`-based sleeps, or `timeout:` overrides in test code, without a root-cause explanation in the PR body that states why the operation is legitimately slow and bounded. Multiple per-call timeout overrides in one spec is a strong reject signal.
+
+### B3 — Assertion loosening
+
+Weakening an assertion (exact → substring, visible → attached, removing an assertion, widening a match set, `force: true` clicks) to make the test pass.
+
+**Why:** the corpus moves in the opposite direction: merged fixes made locators _exact_ (#5940), scoped assertions to the target (#6930), and asserted "durable UI state instead of transient toast visibility" (#6641). `force: true` was removed once the product bug was fixed (#6757, reviewer: "Since we made changes to the src code, I don't think we'd need the force: true anymore, right?"). fix.md: "Do not convert an infrastructure, backend, auth/data, or product-state root cause into a frontend timeout or locator retry."
+
+**Detection heuristic:** diff removes/weakens an `expect`, replaces a specific matcher with a broader one, adds `force: true`, `exact: false`, `.first()`, or regex-broadens a locator, without evidence that the previous assertion was wrong about intended UX (e.g., genuinely duplicate toast as in #5941). Deleting an assertion is allowed only under A6 (whole-test removal with coverage proof) or with UX-drift evidence.
+
+### B4 — Fixing the assertion site instead of the diagnosed surface
+
+Editing the test where the failure surfaced when the evidence points to CI setup, auth/test-data, backend, or product code.
+
+**Why:** SKILL.md: "avoid editing the test just because that is where the failure surfaced." fix.md: "The proposed fix targets the diagnosed failure surface, not only the final assertion." Self-correction example: "Backed out the interviews-list readiness retry. That was chasing the later symptom." (cbh-mobile-app#11338). The fix-locus decision order is: shared setup/CI → backend/data → product → test harness/data → assertion/locator _last_.
+
+**Detection heuristic:** plan's stated failure surface (or the error evidence: 5xx bodies, throttle exceptions, setup-helper stack frames) is backend/setup/infra, but the diff touches only the spec file's waits/assertions. Reject and require the fix at the diagnosed surface or an explicit reason it is out of reach.
+
+### B5 — Per-sighting fixes for a shared root cause (duplicate plans)
+
+Opening one plan/PR per failing test when the failures share one mechanism (same run/burst, same helper, same error shape).
+
+**Why:** Rocky closed five sibling "recover lazy chunk failure" PRs in seven minutes (cbh-admin-frontend #5645–#5649) and merged one canonical fix (#5650). Bulk-debugger skill: "Prefer one implementation ticket per root cause, not one per sighting" and "Do not create per-test fixes for a shared failure." Dedup comments: "No new implementation ticket needed; linking to STAFF-1010 and closing this investigation as a duplicate" (STAFF-1217); PR #6682 was closed as a no-op because the same pin already landed via #6680.
+
+**Detection heuristic:** plan/PR does not state the results of the dedup check (open `flaky-test-fix` PRs touching the same file/helper, recent main commits, canonical Linear ticket for the fingerprint family). If the failing error line matches an existing canonical ticket's fingerprint family or an open sibling PR, reject as duplicate and route to the canonical.
+
+**Prior-sibling signal (backtest amendment, 2026-06-11):** sibling PRs for the same mechanism that were closed _unmerged_ are a strong reject signal, not mere context. A plan citing closed-unmerged siblings must explain what is materially different this time — narrower scope alone is not an answer (the backtest's one genuine false approval cited prior wontfix siblings and explained them away).
+
+### B6 — Code fix on low-confidence diagnosis
+
+Proposing a code change when root-cause confidence is ≤2/5, or any plan below 5/5 that omits observability follow-ups.
+
+**Why:** plan.md: confidence ≤2 → "do not propose a code fix. Instead, recommend specific instrumentation or reproduction steps." Skill PR core-utils#686 (May 25) made it mandatory that "any fix plan below 5/5 confidence must include the frontend and/or backend observability changes that would make the next occurrence diagnosable at 5/5 confidence," spanning repositories if needed. Merged example of diagnostics-as-the-fix: #12109 "add Cognito SMS OTP diagnostics."
+
+**Detection heuristic:** plan lacks a confidence score; or score ≤2 with a code diff attached; or score 3–4 without an "Observability to reach 5/5" section; or evidence section lacks the minimum artifacts (error artifact + network/lifecycle artifact + specific code path).
+
+### B7 — Stale plan applied without re-verification
+
+Implementing a plan against code that has changed on `main`, or for a flake already fixed.
+
+**Why:** fix.md preflight requires "the failing commit's code path still exists on current `main`." STAFF-1122's canonical comment models this: "Main status: **not fixed on `main`** as of `<sha>`" with code citations. Closed PR #6682 (pin already landed on main) is the failure mode.
+
+**Detection heuristic:** PR body lacks a "current main status" statement; or the diff conflicts with/duplicates a commit already on main touching the same lines.
+
+**Staleness clause (backtest amendment, 2026-06-11):** a plan more than ~14 days old, or one whose fingerprint has new sightings since the plan was written, requires re-verification against current main before approval — treat unverified staleness as B7. (Two backtest "rejections" were plans that sat unworked until a sweep canceled them; freshness is part of a plan being gateable.)
+
+## 3. Allowed/safe patterns
+
+These have strong precedent (counts from the 74 merged `flaky-test-fix` PRs; classification approximate) and are auto-merge candidates **only** when the listed evidence is present in the PR body.
+
+### A1 — Deterministic readiness gate (poll a real signal, fail loud)
+
+Wait for the same API/state the UI consumes (or a page-ready signal) before acting; bounded attempts; on exhaustion, throw with the last status/body/IDs.
+
+- **When:** fresh entities not yet visible through a read model/CDC lag; route or dialog needs data before interaction. (~16 merged PRs: #6211, #6912, #12005, #6792, #11365, …)
+- **Required evidence:** the polled signal is the one the failing UI path actually uses (STAFF-1010 polled "the same offer-estimate path used by the dialog"); a bounded timeout; the exhaustion error includes last status/body/request metadata; explicitly NOT a fixed sleep.
+- **Shared-helper rule:** when the repo already has a readiness gate for the same mechanism (e.g., CDC/read-model lag), a new per-spec gate is **rejected** — the fix must use, extend, or extract a shared per-repo helper instead (STAFF-1010 already proposed "extracting a shared fresh-shift offer/readiness helper"). The ~16-gate sprawl in six weeks is the failure mode this prevents. Auto-merge therefore additionally requires: no prior gate for this mechanism exists in the repo, OR the diff goes through the shared helper.
+
+### A2 — Test-data isolation & uniqueness
+
+Stronger unique identities, removal of hardcoded values that interact badly with system constants, per-test state reset, documented service-test app lifecycle (`createTestApp()` + `reset()`).
+
+- **When:** collisions (phone/email/IDs), cross-test state leakage, boundary interaction of seeded values. (~8 merged: #6931, #25161, #24184, #25859, #11662, open-shifts#5188.)
+- **Required evidence:** the collision/leak mechanism named with the artifact that proves it (open-shifts#5188 cited the audit log showing `windowWorkers: []` and quantified the flake probability); fix matches sibling tests' existing pattern where one exists.
+
+### A3 — Clock/time determinism
+
+Fake timers or frozen clocks for tests asserting time-boundary logic; remove weekday/timezone dependence.
+
+- **When:** failures cluster at day/period boundaries or depend on run time. (~3 merged: #25464, #25362, #25003.)
+- **Required evidence:** the time-dependence is identified (which boundary), timers restored in `afterEach`.
+
+### A4 — Assertion/locator hardening (tightening, never loosening)
+
+Exact/scoped locators, assert durable state instead of transient toasts, assert the action's observable outcome.
+
+- **When:** app state is correct and the selector/assertion is the only broken part — the _last_ resort in the fix decision order. (~8 merged: #5940, #6930, #6641, #11409, …)
+- **Required evidence:** artifact showing app state was correct (screenshot/network); the new assertion is strictly more specific or more durable than the old one.
+
+### A5 — Product fix for a real race surfaced by the test
+
+Fix the remount/race/cache/error-handling bug users can also hit; then remove any test workaround.
+
+- **When:** real users can hit the same state. Highest-value class (~14 merged: #6962 remount, #6961 401-redirect loop, #5650 chunk-load recovery, #6173 cache keys, #6757 menu close).
+- **Required evidence:** the user-facing failure mode described (STAFF-1417: "removes the logout loop for real users"); accompanying test workarounds (e.g., `force: true`) removed in the same PR.
+
+### A6 — Test removal when coverage is duplicated
+
+Delete an E2E that re-tests logic covered deterministically at a lower level.
+
+- **When:** failure artifacts show correct app/backend state and a focused lower-level test covers the behavior. (3 merged: #11753, #11612, #11613.)
+- **Required evidence:** PR body names the surviving covering test and shows the artifacts proving the E2E's failure was harness-timing, not product. Auto-merge eligible like other A-rules when that evidence is present; a deletion whose test is the only coverage of the user flow does not qualify as A6 at all.
+
+### A7 — Setup/auth hot-path de-stressing
+
+Remove stateful writes from per-test setup, serialize/cache shared-credential acquisition, pre-provision invariants.
+
+- **When:** throttling errors (`TooManyRequestsException`, rate limits) in setup across shards. (~7 merged: #6697, #11751, #6738, #5733.)
+- **Required evidence:** backend telemetry showing the throttle source (STAFF-1122 cited CloudTrail: 980 `AdminUpdateUserAttributes` throttles); the fix reduces call volume or concurrency rather than just retrying it; validation re-runs the affected specs "with Playwright retries disabled once to prove the root cause is gone rather than hidden."
+
+## 4. Conditional patterns
+
+### C1 — Bounded, classified retry
+
+Allowed only when ALL of: the retried operation is idempotent and preserves the test scenario; the retry predicate matches _only_ transient failure signatures (named errors/status codes — opaque 500s yes, validation 422s no, per #6974's resolution); attempts are bounded; exhaustion reports attempts + last stdout/stderr/status/body; the retry cannot amplify the root cause (rate limits, one-time credentials, duplicate writes). Precedent: #6442 ("retries only transient CLI/network/rate-limit failures … reports attempts/stdout/stderr"), #25431, #11662. A retry as a _bridge_ during a migration must be labeled "a migration guard, not the final design" (STAFF-1122) with the follow-up named.
+
+For uniqueness-collision retries, the retry form must match the error semantics, and the plan must state which case applies: regenerate a _fresh_ identity when the failure is a constraint violation on the identity itself (#11662 regenerated OTP phone numbers); re-attempt the _same_ identity only when the failure is transient (timeout, 5xx) rather than a collision.
+
+### C2 — Polling/web-first assertions (`expect().toPass`, poll-until)
+
+Allowed only when the asserted behavior is genuinely asynchronous/event-driven (eventual consistency, background job) and the poll wraps a _specific_ assertion with a bounded timeout — not a page-reload-until-green loop around a whole flow. The plan must state why no single deterministic completion signal exists.
+
+### C3 — Network mocking / mock fidelity changes
+
+Allowed when the mock is corrected to match the real contract (e.g., #11692 PDF CORS headers, #11936 required `modificationReason`) or when third-party nondeterminism must be removed (#5989 Google Maps autocomplete). Not allowed to mock away the system under test in an E2E whose purpose is to exercise that path.
+
+### C4 — CI/tooling pin or environment guard
+
+Allowed when many tests fail before user-flow code and the evidence is a tool/version/deploy drift (STAFF-1320: deployed UI didn't match `E2E_CHECKOUT_REF`; fix = fail fast or assert deployed commit). Requires: run-level evidence (not single-test), and a dedup check — two such PRs were closed because the pin already landed (#6682) or the approach was superseded (#6824).
+
+### C5 — Cross-repo/backend recommendation
+
+When the root cause is in another team's service, the plan may ship the local resilience fix and must route a recommendation to the owning team (STAFF-1413: frontend rethrow fix + "backend follow-up recommendation (… 503 instead of 401) is noted in the ticket for the owning team"). It may not silently absorb a backend bug into test code.
+
+## 5. Dispositions that need no code
+
+### D1 — Infra blip / environment incident (close as Canceled)
+
+Required evidence (model: STAFF-1419/STAFF-1420 CDC-lag burst):
+
+- A bounded incident window with telemetry: error-count spike (before/during/after), monitor Triggered/Recovered timestamps, deploy/Watchdog version-change correlation.
+- Cross-checks: same-minute failures across repos/shards; affected tests "all passed on retry."
+- A statement that existing harness behaved correctly ("The readiness helpers did their job: they polled … classified the failure, and emitted shift IDs + status + body").
+- Why code can't help: "An ~8-minute backend replication outage isn't fixable in test code."
+- Re-arm note: "The next intake run will create per-test tickets for any contained fingerprint that flakes again outside this burst."
+
+### D2 — Duplicate (close as Duplicate, link canonical)
+
+Required evidence: fingerprint/test/file plus error shape matches an existing investigation family ("the same file/test and HTTP 422 shape were investigated in STAFF-1001"); link both the canonical investigation and its implementation ticket/PR; explicitly state "No new implementation ticket needed." ~25% of the Linear corpus (156/636) closed this way — the critic should _expect_ duplicates and check before approving any new plan.
+
+### D3 — Already fixed / stale
+
+Required evidence: the merged PR/commit on main that covers the flake (e.g., "covered by merged PR #6697"), and confirmation the failing sighting predates the fix. Recurrences after the fix go to the canonical ticket ("Still failing in CI; flake-intake deduplicated this sighting to the canonical fix ticket"), not a new one.
+
+### D4 — Burst with no shared cause identified
+
+Per the flake-intake burst template: "If no shared cause is identifiable, comment with the reason and close this ticket as Won't Do" — per-test tickets regenerate automatically on recurrence outside the burst.
+
+### D5 — Upstream/owning-team bug
+
+Close-out comment must name the owning service/team, the trace/log evidence, and the recommendation handed off (see C5). No test-side band-aid in lieu of the handoff.
+
+## 6. Resolved decisions (Rocky, 2026-06-10)
+
+All seven open questions from the mining pass were answered; the substantive ones are folded into the rules above. Kept here as provenance for the living doc:
+
+1. **Corpus span:** the ~6-week labeled corpus (2026-04-30 onward) is sufficient; no mining of older SYN-/ACT-/ALCH-prefixed history before the critic enforces auto-merge.
+2. **B3 (assertion loosening):** confirmed as written, including hard-reject on `exact: false`/`.first()`; adjust from experience if it over-fires.
+3. **A6 (test removal):** auto-merge eligible when its required evidence is present — no standing human-review carve-out.
+4. **A1 (readiness-gate sprawl):** critic rejects new per-spec gates when a shared helper for the mechanism exists or should be extracted (now A1's shared-helper rule).
+5. **Automated reviewer findings (Mendral/CodeRabbit):** advisory, never binding (now in §1).
+6. **Closed PRs #6824/#6572:** confirmed supersession, not approach rejections; C4's reading stands.
+7. **C1 collision retries:** form depends on identity type and error semantics — regenerate on constraint violations, re-attempt only on transient failures (now in C1).
+
+## 7. Appendix: corpus stats
+
+**Read:** 753 Linear issues enumerated (636 flaky-labeled; statuses: Done 431, Duplicate 156, Canceled 37, open 12); comments read on ~35 sampled issues (weighted to Canceled/canonical/impl tickets — most per-test tickets have only bot sighting comments). 88 `flaky-test-fix` PRs enumerated (74 merged, 13 closed, 1 open); bodies/diffs/review threads read on ~25; Rocky-authored review/issue comments harvested across all 88 (79 inline + 37 conversation comments, mostly cb-babysit); 4 skill-update PRs in core-utils; flake-intake pipeline templates in ClipboardHealth/clipboard.
+
+**Pattern frequency (merged PRs, approximate title+body classification):**
+
+| Pattern                                             | Count |
+| --------------------------------------------------- | ----- |
+| Readiness gate / deterministic wait                 | ~16   |
+| Product fix surfaced by flake                       | ~14   |
+| Assertion/locator tightening                        | ~8    |
+| Auth/setup throttle removal, serialization, caching | ~7    |
+| Bounded classified retry                            | ~6    |
+| Test-data isolation/uniqueness                      | ~6    |
+| Mock fidelity                                       | ~4    |
+| Clock/time determinism                              | ~3    |
+| Test removal (duplicated coverage)                  | 3     |
+| Diagnostics-only                                    | ~2    |
+| Service-test lifecycle isolation                    | ~2    |
+
+**Ten most instructive examples:**
+
+1. STAFF-1122 + cbh-admin-frontend#6697 — canonical throttling cluster; "Do not solve this by adding frontend waits or Playwright retries."
+2. STAFF-1419/STAFF-1420 — gold-standard infra-blip no-code disposition.
+3. cbh-admin-frontend#5645–#5650 — duplicate-PR consolidation; one root cause, one merge.
+4. STAFF-1010 — accepted plan format; readiness poll over CDC sleep.
+5. cbh-admin-frontend#6974 — retry-classification pushback (500 vs 422).
+6. cbh-admin-frontend#6442 (STAFF-942) — anatomy of an acceptable retry.
+7. cbh-mobile-app#11242 — timeout-inflation PR converted to root-cause fix before approval.
+8. cbh-mobile-app#11753 (STAFF-922) — test removal with coverage proof.
+9. open-shifts#5188 (SYN-1919) — test-data fix with quantified evidence (Monte Carlo, audit log).
+10. STAFF-1413 + cbh-admin-frontend#6961 — incident-window diagnosis producing a product fix plus owning-team handoff.

package/skills/{flaky-test-debugger → flaky-debug}/SKILL.md

@@ -1,6 +1,6 @@
 ---
-name: flaky-test-debugger
-description: Debug and fix flaky tests including Playwright E2E, NestJS service/integration, React component, and unit tests. Use this skill when investigating intermittent test failures, triaging flaky tests, or fixing test instability.
+name: flaky-debug
+description: Debug and fix flaky tests including Playwright E2E, NestJS service/integration, React component, and unit tests.
 ---
 
 Phases run in order. Skip a phase if you already have the information it produces. Phase 3 runs only in fix mode.
@@ -20,18 +20,7 @@ Both modes share the same diagnosis path; the plan is the artifact you hand to a
 
 ## Phase 1: Classify Failure Surface and Test Type
 
-For E2E flakes, first classify where the failure surfaced in the lifecycle, then identify the test type. The failure surface dictates how broadly to investigate before reading or editing the test.
-
-Common E2E failure surfaces:
-
-- **CI/job setup:** dependency installation, CLI/tooling, environment setup, build/deploy, artifact download.
-- **Test setup/auth/data:** token minting, login bootstrap, seeded users/entities, one-time credentials, external service setup.
-- **App bootstrap/navigation:** static assets, route load, hydration, browser console/page errors before the user action.
-- **User action:** click/input completed but the expected request, dialog, route change, or state transition did not start.
-- **Backend request:** request emitted; backend returned error, stale data, unexpected shape, or excessive latency.
-- **Assertion/locator:** app state is correct, but the assertion/selector is brittle or out of sync with the intended UX.
-
-Then determine the test type from the user's input. The type dictates the detailed investigation path.
+Determine the test type from the user's input. The type dictates the detailed investigation path.
 
 | Type                             | Signals                                                                                                                                                              |
 | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
@@ -42,6 +31,8 @@ Then determine the test type from the user's input. The type dictates the detail
 
 If the type is ambiguous, check the test file extension and imports to confirm.
 
+If the type is E2E, also classify where the failure surfaced in the lifecycle -- see [Classify the E2E Failure Surface](./references/plan-e2e.md#classify-the-e2e-failure-surface) in `plan-e2e.md`. The failure surface dictates how broadly to investigate before reading or editing the test.
+
 ## Phase 1b: Check for Existing Fixes
 
 Before investigating, check whether someone (or another agent) has already fixed this flake.
@@ -64,9 +55,7 @@ If no existing fix is found, proceed to Phase 2.
 
 ## Phase 2: Produce a plan
 
-Follow [`references/plan.md`](./references/plan.md). It walks investigation, diagnosis, evidence gathering, and the fix decision tree, and produces a structured plan with confidence score.
-
-If the plan's confidence is less than 5/5, it must include the frontend and/or backend observability changes needed to reach 5/5 confidence next time. The plan may request changes across multiple repositories; assume we have access to all code.
+Follow [`references/plan-e2e.md`](./references/plan-e2e.md) for E2E tests, or [`references/plan-fast-path.md`](./references/plan-fast-path.md) for service, component, and unit tests. Both converge on [`references/plan.md`](./references/plan.md) for the fix decision and plan output format, and produce a structured plan with a confidence score.
 
 If you are in plan mode, present the plan and stop here.
 

package/skills/{flaky-test-debugger → flaky-debug}/references/fix.md

@@ -1,6 +1,6 @@
 # Apply a Flaky Test Fix
 
-Apply phase of the flaky-test-debugger skill. Takes a plan produced by [`plan.md`](./plan.md) and applies it.
+Apply phase of the flaky-debug skill. Takes a plan produced by the planning phase ([`plan-e2e.md`](./plan-e2e.md) or [`plan-fast-path.md`](./plan-fast-path.md), finished by [`plan.md`](./plan.md)) and applies it.
 
 ## Preflight
 
@@ -20,7 +20,7 @@ Do not convert an infrastructure, backend, auth/data, or product-state root caus
 
 ## Fix Sibling Instances
 
-After fixing the root cause, search for other tests that exhibit the same anti-pattern and fix them too. A flaky pattern in one test file almost always has siblings nearby.
+After fixing the root cause, search for other tests that exhibit the same anti-pattern. Flaky patterns often have siblings nearby, but do not turn a one-test plan into broad repo churn.
 
 ### When to search
 
@@ -47,7 +47,7 @@ Skip this step when the fix is **specific to one test's logic** -- for example a
 
 2. Scope the search to the same area of the codebase first (same package or directory), then widen if the pattern is pervasive.
 
-3. Apply the same fix to each sibling. Keep changes minimal; fix the anti-pattern, nothing else.
+3. Apply the same fix to each sibling named in the plan or directly confirmed as the same root cause. Keep changes minimal; fix the anti-pattern, nothing else. Report broader candidates instead of editing them unless they clearly share the root cause.
 
 4. List the sibling files you fixed in the output so reviewers can verify them.
 

package/skills/{flaky-test-debugger/references/plan.md → flaky-debug/references/plan-e2e.md}

@@ -1,60 +1,6 @@
-# Plan a Flaky Test Fix
+# Plan a Flaky Test Fix -- E2E (Playwright)
 
-Diagnosis and planning phase of the flaky-test-debugger skill. Produces a structured plan that the user reviews. In fix mode, the plan is consumed by [`fix.md`](./fix.md).
-
-Route by the test type identified in Phase 1 of SKILL.md:
-
-- **E2E (Playwright):** start with [E2E Triage Snapshot](#e2e-triage-snapshot)
-- **Service, React component, or Unit:** start with [Fast Path (non-E2E)](#fast-path-non-e2e)
-
-## Fast Path (non-E2E)
-
-For service, component, and unit tests, the failure information plus the test source code is usually sufficient to diagnose the flake. Do not over-investigate -- read the evidence, read the code, plan the fix.
-
-### Gather Failure Context
-
-Capture from the user's input (ask if missing):
-
-- **Test file and name** -- exact file path and test title
-- **Error message and stack trace** -- the raw failure output
-- **Framework** -- Jest, Vitest, etc.
-- **Failure metadata** -- branch, pipeline URL, duration, shard, timestamp (when available)
-
-### Read the Test and Code Under Test
-
-1. Read the failing test file. Focus on the specific failing test and its surrounding `describe`/`beforeEach`/`afterEach`/`afterAll` blocks.
-2. Read the production code that the test exercises -- follow imports from the test file.
-3. For service tests: also read the test module setup (the `Test.createTestingModule(...)` or app bootstrap code), and check for `afterAll` cleanup that closes the app/database connections.
-
-### Classify the Flake Pattern
-
-| Category                        | Test Types      | Signal                                                                                                             |
-| ------------------------------- | --------------- | ------------------------------------------------------------------------------------------------------------------ |
-| **Connection lifecycle**        | Service         | "connection closed", "topology destroyed", socket errors in stack -- app/DB not fully ready or torn down too early |
-| **Port conflict**               | Service         | EADDRINUSE -- multiple test files bootstrapping on the same port                                                   |
-| **Async teardown race**         | Service         | Errors appear after test passes -- `afterAll` closes the app while background work is still running                |
-| **Database state leakage**      | Service         | Test depends on DB state that a parallel/prior test modified                                                       |
-| **Unresolved async work**       | Component       | "not wrapped in act()" warnings, state updates after unmount                                                       |
-| **Timer/animation not flushed** | Component       | Test asserts before `setTimeout`/`requestAnimationFrame` fires, or `useFakeTimers` not advanced                    |
-| **Mock not restored**           | Component, Unit | `jest.spyOn` or `jest.mock` bleeds into the next test -- missing `mockRestore`/`restoreAllMocks`                   |
-| **Shared mutable state**        | Unit            | Module-level variable or singleton mutated by one test, observed by another                                        |
-| **Date/time sensitivity**       | Unit            | Test assumes a specific date, time zone, or `Date.now()` value that shifts across runs                             |
-| **Test ordering dependency**    | All             | Passes in isolation, fails when run with other tests (or vice versa)                                               |
-
-### Diagnose with Evidence
-
-Before proposing a fix, gather:
-
-- The **error message and stack trace** from the failure
-- The **specific code path** in the test or production code that caused the flake
-- A brief **explanation** of why the flake is intermittent (what timing or state condition triggers it)
-- A **confidence score** (1-5, see [Confidence Score](#confidence-score))
-
-If confidence is less than 5/5, the plan must include the frontend and/or backend observability changes that would provide enough evidence for 5/5 confidence next time. Do not limit that recommendation to the current repository -- the plan may request changes in multiple repositories because we have access to all code.
-
-If confidence is 2 or below, the plan is to gather more data: recommend specific reproduction steps or instrumentation rather than a code fix.
-
-If >2, continue to [Decide Fix Approach](#decide-fix-approach).
+Diagnosis and planning phase of the flaky-debug skill for Playwright E2E tests. Produces a structured plan that the user reviews. Ends at [`plan.md`](./plan.md) for the fix decision and plan output format.
 
 ## E2E Triage Snapshot
 
@@ -69,8 +15,10 @@ Capture these details first so the investigation is reproducible. If the user ha
 
 Downloads the `playwright-llm-report` artifact from a GitHub Actions run.
 
+Resolve the bundled `scripts/` path relative to `SKILL.md`; do not use a target repository's `scripts/` directory.
+
 ```bash
-bash scripts/fetch-llm-report.sh "<github-actions-url>"
+bash "<flaky-debug-skill-dir>/scripts/fetch-llm-report.sh" "<github-actions-url>"
 ```
 
 This downloads and extracts to `/tmp/playwright-llm-report-{runId}/`. The report is a single `llm-report.json` file.
@@ -86,7 +34,7 @@ Before reading the failing test as if it is isolated, check whether the failure
 
 If many tests fail in the same setup helper, dependency install step, auth/token path, or external service call, diagnose the shared mechanism first. Do not create per-test fixes for a shared failure.
 
-### Classify the E2E Failure Surface
+## Classify the E2E Failure Surface
 
 Use the earliest observed failure surface to decide where to investigate first:
 
@@ -102,7 +50,7 @@ Use the earliest observed failure surface to decide where to investigate first:
 
 This classification can change as evidence improves. State the final surface in the plan.
 
-## Quick Classification (E2E)
+## Classify the Flake Pattern (E2E)
 
 For the full report schema, field reference, caps, and example reports:
 
@@ -221,94 +169,4 @@ Do not propose a fix without concrete artifacts. At minimum, include:
 - When available: **screenshot** from `failureArtifacts.screenshotBase64` showing page state at failure
 - When available: **Datadog trace** via `attempts[].network.instances[].traceId` showing backend behavior for the failing request
 - When relevant: **logs/RUM/CI evidence** that confirms whether the issue is app, backend, infra, auth/test-data, or CI tooling
-- A **confidence score** from 1 to 5 rating how certain you are in the root cause diagnosis
-
-If confidence is less than 5/5, identify the missing evidence and propose concrete frontend and/or backend observability changes that would make the next occurrence diagnosable at 5/5 confidence. These changes may span multiple repositories.
-
-If confidence is 2 or below, do not propose a code fix. Instead, recommend specific instrumentation or reproduction steps to raise confidence.
-
-If >2, continue to [Decide Fix Approach](#decide-fix-approach).
-
-### Confidence Score
-
-Rate your confidence in the root cause on a 1-5 scale. Report this score alongside your evidence.
-
-| Score | Meaning             | Criteria                                                                                                                                                                                       |
-| ----- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| **5** | Certain             | Root cause is directly visible in artifacts (e.g., assertion diff shows stale data, network response confirms 5xx, screenshot shows error banner)                                              |
-| **4** | High confidence     | Evidence strongly supports the diagnosis but one link in the chain is inferred rather than observed (e.g., timeline shows the right sequence but no Datadog trace to confirm backend behavior) |
-| **3** | Moderate confidence | Evidence is consistent with the diagnosis but alternative explanations remain plausible. Flag the alternatives explicitly                                                                      |
-| **2** | Low confidence      | Limited evidence, mostly reasoning from code patterns rather than observed artifacts. Recommend gathering more data before committing to a fix                                                 |
-| **1** | Speculative         | No direct evidence for the root cause. The fix is a best guess. Recommend reproducing the failure locally or adding instrumentation before proceeding                                          |
-
-## Decide Fix Approach
-
-Applies to all test types.
-
-Choose the fix locus from the evidence, not from where the assertion failed. A flaky E2E test can be exposing a CI dependency issue, auth/test-data service issue, backend bug, deployment problem, product state bug, or test harness bug.
-
-Use this decision order:
-
-1. **Shared setup or CI fix** when many tests fail before user-flow assertions or all failures share a tool, cache, install, auth, seed-data, or fixture path.
-2. **Backend/service/data fix** when the expected request is emitted and backend telemetry or response bodies show errors, throttling, stale data, inconsistent state, or unexpected latency.
-3. **Product fix** when real users can hit the same unsafe intermediate state, render error, permission/session race, stale cache, or missing error handling.
-4. **Test data or harness fix** when the scenario is not user-realistic, the test setup is semantically wrong, or the test needs a deterministic app-ready signal.
-5. **Assertion/locator fix** only when the app state is correct and the selector/assertion is the only broken part.
-
-Before proposing any retry, timeout, or wait change, pass the idempotency check:
-
-- Is the retried operation safe to repeat?
-- Does retrying preserve the same test scenario?
-- Could retrying amplify the root cause, such as rate limits, one-time credentials, duplicate writes, or destructive mutations?
-- Is there a deterministic signal to wait on instead of a longer timeout?
-
-If the answer is no or unclear, do not add a retry/wait as the fix. Propose a root-cause fix or instrumentation instead.
-
-Common valid fix types:
-
-- **Shared setup / CI**
-  - Pin or lock runtime tools and dependencies
-  - Fail fast on incompatible tool contracts
-  - Remove mutable global state from CI setup
-  - Add setup-level diagnostics before sharding
-
-- **Test harness / data** (when the failure is non-product):
-  - Reset cookies, storage, and session between retries
-  - Isolate test data; generate stronger unique identities
-  - Make retry blocks idempotent
-  - Wait on deterministic app signals, not arbitrary sleeps
-  - (Service tests) Close connections and app properly in `afterAll`
-  - (Component tests) Flush pending state updates and timers before asserting
-  - (Unit tests) Reset shared mutable state in `beforeEach`
-
-- **Product** (when real users would hit the same issue):
-  - Handle stale or intermediate states safely
-  - Make routing/render logic robust to eventual consistency
-  - Add telemetry for ambiguous transitions
-
-- **Backend/service**
-  - Remove avoidable shared mutable writes from hot paths
-  - Make setup operations idempotent or explicitly rate limited
-  - Fix stale reads, cache invalidation, and eventual-consistency assumptions
-  - Add trace/log correlation for ambiguous failures
-
-Choose **both** if user impact exists _and_ tests are fragile.
-
-## Plan Output Format
-
-Produce the plan with these fields:
-
-- **Test ID:** if provided in prompt
-- **Agent session ID:** your running session ID to resume if needed
-- **Confidence:** score (1-5) with brief justification
-- **Failure surface:** CI/job setup, test setup/auth/data, app bootstrap, user action no-op, backend request, post-success render, assertion/locator, or mixed
-- **Current main status:** whether the failing commit's code path still exists on current `main`, has already been fixed, or has changed enough that the plan must be adjusted
-- **Symptom:** what failed and where
-- **Root cause:** concise technical explanation
-- **Evidence:** artifacts supporting the diagnosis (traces, network, error messages, screenshots as applicable)
-- **Proposed fix:** test harness, product, or both — with the specific file(s) and the change you would make
-- **Observability to reach 5/5:** required when confidence is less than 5/5. List the frontend and/or backend telemetry, logging, tracing, reporter, or metric changes that would make this flake diagnosable with 5/5 confidence next time, including any repositories that should change. Use "N/A -- confidence is 5/5" only for a 5/5 plan.
-- **Sibling candidates:** files that appear to share the same anti-pattern, for the reviewer (or fix.md) to confirm. Or "N/A -- fix is test-specific" if the issue is one-off (see [`fix.md`](./fix.md) for what counts as a structural anti-pattern worth searching for).
-- **Validation plan:** lint/typecheck commands and test commands to run after applying the fix
-- **Open questions:** anything that needs human input before fixing
-- **Residual risk:** what could still be flaky after the fix
+- A **confidence score** -- see [Confidence Score](./plan.md#confidence-score) in `plan.md` for the 1-5 scale and what to do with it.

package/skills/flaky-debug/references/plan-fast-path.md

@@ -0,0 +1,44 @@
+# Plan a Flaky Test Fix -- Fast Path (Service, Component, Unit)
+
+Diagnosis and planning phase of the flaky-debug skill for service, React component, and unit tests. Produces a structured plan that the user reviews. Ends at [`plan.md`](./plan.md) for the fix decision and plan output format.
+
+For service, component, and unit tests, the failure information plus the test source code is usually sufficient to diagnose the flake. Do not over-investigate -- read the evidence, read the code, plan the fix.
+
+## Gather Failure Context
+
+Capture from the user's input (ask if missing):
+
+- **Test file and name** -- exact file path and test title
+- **Error message and stack trace** -- the raw failure output
+- **Framework** -- Jest, Vitest, etc.
+- **Failure metadata** -- branch, pipeline URL, duration, shard, timestamp (when available)
+
+## Read the Test and Code Under Test
+
+1. Read the failing test file. Focus on the specific failing test and its surrounding `describe`/`beforeEach`/`afterEach`/`afterAll` blocks.
+2. Read the production code that the test exercises -- follow imports from the test file.
+3. For service tests: also read the test module setup (the `Test.createTestingModule(...)` or app bootstrap code), and check for `afterAll` cleanup that closes the app/database connections.
+
+## Classify the Flake Pattern
+
+| Category                        | Test Types      | Signal                                                                                                             |
+| ------------------------------- | --------------- | ------------------------------------------------------------------------------------------------------------------ |
+| **Connection lifecycle**        | Service         | "connection closed", "topology destroyed", socket errors in stack -- app/DB not fully ready or torn down too early |
+| **Port conflict**               | Service         | EADDRINUSE -- multiple test files bootstrapping on the same port                                                   |
+| **Async teardown race**         | Service         | Errors appear after test passes -- `afterAll` closes the app while background work is still running                |
+| **Database state leakage**      | Service         | Test depends on DB state that a parallel/prior test modified                                                       |
+| **Unresolved async work**       | Component       | "not wrapped in act()" warnings, state updates after unmount                                                       |
+| **Timer/animation not flushed** | Component       | Test asserts before `setTimeout`/`requestAnimationFrame` fires, or `useFakeTimers` not advanced                    |
+| **Mock not restored**           | Component, Unit | `jest.spyOn` or `jest.mock` bleeds into the next test -- missing `mockRestore`/`restoreAllMocks`                   |
+| **Shared mutable state**        | Unit            | Module-level variable or singleton mutated by one test, observed by another                                        |
+| **Date/time sensitivity**       | Unit            | Test assumes a specific date, time zone, or `Date.now()` value that shifts across runs                             |
+| **Test ordering dependency**    | All             | Passes in isolation, fails when run with other tests (or vice versa)                                               |
+
+## Diagnose with Evidence
+
+Before proposing a fix, gather:
+
+- The **error message and stack trace** from the failure
+- The **specific code path** in the test or production code that caused the flake
+- A brief **explanation** of why the flake is intermittent (what timing or state condition triggers it)
+- A **confidence score** -- see [Confidence Score](./plan.md#confidence-score) in `plan.md` for the 1-5 scale and what to do with it.