@clipboard-health/ai-rules 2.41.1 → 2.41.2

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@clipboard-health/ai-rules",
-  "version": "2.41.1",
+  "version": "2.41.2",
   "description": "Pre-built AI agent rules for consistent coding standards.",
   "keywords": [
     "ai",

package/skills/flaky-critic/SKILL.md

@@ -47,9 +47,15 @@ VERDICT: reject — B2, B6
 - B2: "increase the navigation timeout to 30s" — timeout inflation without a root-cause statement of why the operation is legitimately slow.
 - B6: plan states confidence 3/5 but has no observability-to-reach-5/5 section.
 Fix class claimed: A1 (readiness gate). Missing required evidence: polled signal is not shown to be the one the failing UI path uses.
-<!-- flaky-critic: verdict=reject mode=enforce -->
+<!-- flaky-critic: verdict=reject mode=enforce disposition=substantive-reject statement-missing=B6 substantive=B2 -->
 ```
 
+- Every verdict marker must record `disposition`, `statement-missing`, and `substantive`. Use comma-separated rule tokens and `none` when a bucket is empty. Distinguish B5's checks as `B5-dedup` and `B5-prior-attempts`; use the normal rule ID for every other rule. Use these dispositions:
+  - approve: `verdict=approve disposition=release statement-missing=none substantive=none`;
+  - statement-only reject: `verdict=reject disposition=amend-and-resubmit`;
+  - any demonstrated violation: `verdict=reject disposition=substantive-reject`;
+  - low-confidence or out-of-taxonomy escalation: `verdict=needs-human disposition=needs-human`;
+  - second reject on revised content: `verdict=needs-human disposition=second-reject`.
 - **enforce**: approve → move the ticket to Todo and assign it to the Linear API user. Reject → comment and leave in Triage; the investigating flow gets one bounce. A second reject on the same ticket → comment `needs-human` and stop touching it.
 - **shadow**: comment only, prefixed `[shadow]`, marker `mode=shadow`.
 - **needs-human**: comment with what the rubric cannot answer; never move the ticket.
@@ -60,7 +66,40 @@ When you encounter a ticket where a human's action contradicts your prior verdic
 
 ## Phase 5: Digest
 
-End with: tickets reviewed, verdicts by type with rule-ID counts, agreement events observed (human actions vs prior verdicts), amendment proposals drafted, and tickets skipped as already-current. When run as a recurring task, this digest is the run's final message.
+End with: tickets reviewed, verdicts by type with rule-ID counts, agreement events observed (human actions vs prior verdicts), amendment proposals drafted, tickets skipped as already-current, and the monitoring block below. When run as a recurring task, this digest is the run's final message.
+
+### B8 and Prior-attempts monitoring
+
+Monitor the first 14 days after both rules are live: **starting 2026-07-16 and ending before 2026-07-30**. During the window, query Groundcrew `flaky-implementation` tickets updated on or after 2026-07-16 across all states and read their full comments; list results and the current queue are not a cumulative data source. Recompute the window from `mode=enforce` critic markers so the result survives separate sessions.
+
+On or after 2026-07-30, coordinate finalization on STAFF-1818 with deterministic run key `2026-07-16--2026-07-30`:
+
+Only honor coordination markers authored by the configured Linear API user that runs flaky-critic. Read authorship and creation time from Linear's server-provided comment metadata; embedded `claimed-at` and `lease-until` values are descriptive, not authority. Ignore markers from other authors, markers with a mismatched run key, future-dated markers, and leases whose server creation time plus the documented duration has expired. For linked GitHub or Linear actions, also require the expected repository or project, exact title suffix, run key, and an artifact author/creator attributable to the same trusted automation.
+
+1. If a trusted `<!-- flaky-critic: b8-prior-monitoring-finalized run=2026-07-16--2026-07-30 -->` is present, omit this monitoring block.
+2. Otherwise post `<!-- flaky-critic: b8-prior-monitoring-claim run=2026-07-16--2026-07-30 claimed-at=<ISO timestamp> -->`, then re-read all comments. The earliest trusted non-stale claim by server creation time wins; later claimants stop. A claim is stale 60 minutes after its server creation time when no trusted action or finalized marker exists, allowing a later run to take over.
+3. Before creating side effects, search STAFF-1818 comments, Linear tickets, and GitHub PRs for a trusted `<!-- flaky-critic: b8-prior-monitoring-action run=2026-07-16--2026-07-30 -->`. Reuse the validated linked action when present.
+4. When a new action is required, the claim winner posts `<!-- flaky-critic: b8-prior-monitoring-action-started run=2026-07-16--2026-07-30 owner=<claim comment ID> lease-until=<ISO timestamp> -->` with a four-hour lease, then re-reads STAFF-1818. The earliest trusted unexpired action lease wins; later action starters stop. Calculate expiry from the server creation time, require `owner` to reference the winning trusted claim, and treat renewals as new trusted comments from that owner. Renew before invoking `cb-work` and at least every three hours while work continues.
+5. Revalidate trusted ownership and repeat the action search immediately before any PR or ticket creation. Search-and-re-read is a secondary guard, not the uniqueness mechanism: the creator must use a server-enforced idempotency key or unique external ID derived from the run key. For GitHub, use deterministic head branch `flaky-critic-monitoring/2026-07-16--2026-07-30`; if branch or PR creation reports that it already exists, look up and reuse the existing PR. A Linear fallback ticket may be created only when the available creator provides an atomic idempotency or uniqueness facility; otherwise post `needs-human` on STAFF-1818 instead of risking a duplicate. The `cb-work` request must carry these requirements through its shipping step. Any new PR or ticket must carry the action marker, an exact `STAFF-1818 monitoring` title suffix, and the deterministic run key. A takeover is allowed only after the prior trusted action lease expires with no validated action marker.
+6. Run the final aggregation and complete any threshold or amendment action below. Post the final digest and all action links to STAFF-1818 with the finalized marker. Write it only after required actions exist.
+
+- **Denominator — gated plans:** one verdict marker per unique ticket content-state in the window, including approve, reject, and needs-human. Exclude shadow/backtest verdicts, PR-checkpoint comments, and already-current skips. The one-verdict-per-content-state rule makes each marker one denominator event.
+- **Numerator — statement-missing bounces:** gated plans whose marker has `disposition=amend-and-resubmit` and names `B8` or `B5-prior-attempts` in `statement-missing`. Count a plan once in the headline numerator even when both rules fired; also report per-rule attribution, where one plan may count under both.
+- **Substantive rejects:** report `B8` and `B5-prior-attempts` substantive counts separately. When a human later releases one, compare the cases by mechanism. Any systematic substantive false-reject pattern, regardless of rate, requires a concrete rubric amendment proposal for Rocky's approval using `<!-- flaky-critic: amendment-proposal -->`; never edit or weaken the rubric silently.
+- **Harness-contract watch:** record every plan that invokes B8's harness-contract terminus, including whether the critic accepted an evidenced contract or rejected a generic detach, timeout, 401, or retry-success symptom as laundering. Compare later human outcomes for both accepted and rejected cases. Report any repeated stretch pattern in the substantive-pattern finding; do not broaden the exception from monitoring evidence without Rocky's per-PR approval.
+- **Threshold:** report the cumulative statement-missing numerator, denominator, percentage, and per-rule attribution as provisional during the window. When the denominator is zero, report `N=0`, omit the percentage as not evaluable, and do not trigger a breach. At the window close, **more than 25%** triggers a fix-forward change in the narrowest flaky-debug plan-producing source: normally `references/plan.md` for a missing output field, or `SKILL.md` when the investigation flow failed to collect the required evidence. `plan-e2e.md` and `plan-fast-path.md` already converge on the shared output template; do not duplicate requirements across them. Invoke `core:cb-work`/`cb-work` against `ClipboardHealth/core-utils` with the observed omission and required template change; if that workflow is unavailable, create a linked Groundcrew implementation ticket only through the atomic uniqueness path above. Include the resulting PR or ticket link in the final digest before writing the finalization marker. Do not change B8 or the Prior-attempts gate to reduce the rate.
+
+Use this digest shape:
+
+```text
+New-rule monitoring (2026-07-16..<2026-07-30):
+- gated plans: N
+- statement-missing amend-and-resubmit: N — rate: P% | not evaluable (when gated plans=0) — B8: N, B5-prior-attempts: N
+- substantive rejects — B8: N, B5-prior-attempts: N
+- harness-contract terminus watch: N invoked — accepted: N, laundering rejects: N, human reversals: N
+- threshold: provisional | pass | breached — <fix-forward action or "none">
+- substantive false-reject patterns: <amendment proposal links or "none observed">
+```
 
 ## PR checkpoint (when the task description asks for it)
 

package/skills/flaky-critic/references/backtests/2026-07-16-b8-prior-attempts/answer-key.json

@@ -0,0 +1,242 @@
+[
+  {
+    "file": "case-01.md",
+    "source": "STAFF-1674 via ClipboardHealth/cbh-admin-frontend#7373",
+    "outcome": "reject",
+    "basis": "A human reviewer requested changes because the proposed implementation preserved an obsolete magic-link test path instead of moving tests to the supported authentication path.",
+    "labelStrength": "strong",
+    "correlationNotes": "Cognito/auth family; correlated with case-10 and case-15.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7373#issuecomment-4894471097",
+    "planCapturedAt": "2026-07-06T15:18:27Z",
+    "inputSha256": "50bfa6f4e0b4e473e0b9fd3fefa8487d8ff4de218254cd055f2c277953b8ae6f",
+    "preOutcomeVerified": true
+  },
+  {
+    "file": "case-02.md",
+    "source": "STAFF-1541 via ClipboardHealth/clipboard-health#26392",
+    "outcome": "approve",
+    "basis": "The change was integrated after automated review approval; this repository does not apply the GitHub release marker used by the frontend repositories.",
+    "labelStrength": "strong",
+    "correlationNotes": "Non-E2E service-test case; no direct mechanism duplicate in cohort.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/clipboard-health/pull/26392#issuecomment-4812847260",
+    "planCapturedAt": "2026-06-26T19:36:01Z",
+    "inputSha256": "5bcc4d6f5f623cd8c661d5880e8d73d85cdcd6f4235609c53bca0c51c8c11eb1",
+    "preOutcomeVerified": true
+  },
+  {
+    "file": "case-03.md",
+    "source": "STAFF-1789 via ClipboardHealth/cbh-admin-frontend#7559",
+    "outcome": "approve",
+    "basis": "The change was integrated and carried the release marker.",
+    "labelStrength": "strong",
+    "correlationNotes": "Home-health UI family; related surface to cases 08, 14, and 16, but distinct mechanism.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7559#issuecomment-4976477226",
+    "planCapturedAt": "2026-07-15T03:08:31Z",
+    "inputSha256": "4c8cadc7b2ef5cdab0628ed5ab1c0ad5fb46089584a26cb883368ed649efc188",
+    "preOutcomeVerified": true
+  },
+  {
+    "file": "case-04.md",
+    "source": "STAFF-1556 via ClipboardHealth/cbh-mobile-app#12521",
+    "outcome": "reject",
+    "basis": "The change was left inactive, marked wontfix, and closed without integration after concern about its CI-time cost; no final human quality verdict was recorded.",
+    "labelStrength": "weak",
+    "correlationNotes": "Static-asset/bootstrap family; weak stale/wontfix label.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/cbh-mobile-app/pull/12521#issuecomment-4835293549",
+    "planCapturedAt": "2026-06-29T17:31:32Z",
+    "inputSha256": "80c44f23858efe66e4c2b8b42da92d5875ad6dfeab5b86a59f7a4573d37901db",
+    "preOutcomeVerified": true
+  },
+  {
+    "file": "case-05.md",
+    "source": "STAFF-1573 via ClipboardHealth/cbh-admin-frontend#7282",
+    "outcome": "approve",
+    "basis": "The change was integrated and carried the release marker.",
+    "labelStrength": "strong",
+    "correlationNotes": "Worker-creation collision/retry family; correlated with case-20.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7282#issuecomment-4845889284",
+    "planCapturedAt": "2026-06-30T16:47:14Z",
+    "inputSha256": "132a76ebf62035c21a70660b8c7c8e829ed5f2dc8e366f38066c44dedc28d6f0",
+    "preOutcomeVerified": true
+  },
+  {
+    "file": "case-06.md",
+    "source": "STAFF-1689 via ClipboardHealth/cbh-mobile-app#12635",
+    "outcome": "approve",
+    "basis": "The change was integrated and carried the release marker.",
+    "labelStrength": "strong",
+    "correlationNotes": "Third-party SDK degradation mechanism; otherwise distinct.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/cbh-mobile-app/pull/12635#issuecomment-4897738607",
+    "planCapturedAt": "2026-07-06T21:23:38Z",
+    "inputSha256": "3d1bba0c70995fc74766927b4a2db256d3d6a3d14d532b6fa26cd4b267476321",
+    "preOutcomeVerified": true
+  },
+  {
+    "file": "case-07.md",
+    "source": "STAFF-1320 via ClipboardHealth/cbh-admin-frontend#6824",
+    "outcome": "reject",
+    "basis": "The change was closed without integration; rubric provenance identifies this historical case as superseded rather than an approach rejection.",
+    "labelStrength": "weak",
+    "correlationNotes": "CI/deployment guard family; weak supersession label.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6824#issuecomment-4624163640",
+    "planCapturedAt": "2026-06-04T16:32:07Z",
+    "inputSha256": "863a3093900c7e1eb2da6b96ad76eeb5f2df8907d2657c64b4dfe9690f4997b2",
+    "preOutcomeVerified": true
+  },
+  {
+    "file": "case-08.md",
+    "source": "STAFF-1685 via ClipboardHealth/cbh-admin-frontend#7390",
+    "outcome": "approve",
+    "basis": "The change was integrated and carried the release marker.",
+    "labelStrength": "strong",
+    "correlationNotes": "Home-health UI family; related surface to cases 03, 14, and 16.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7390#issuecomment-4898990709",
+    "planCapturedAt": "2026-07-07T00:51:18Z",
+    "inputSha256": "365c42ecc41cc10786329a658789113b26442496379c1ddbaeb3a18c50bfe453",
+    "preOutcomeVerified": true
+  },
+  {
+    "file": "case-09.md",
+    "source": "STAFF-1534 via ClipboardHealth/cbh-mobile-app#12496",
+    "outcome": "approve",
+    "basis": "The change was integrated and carried the release marker.",
+    "labelStrength": "strong",
+    "correlationNotes": "Mobile E2E time-wheel interaction case; distinct.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/cbh-mobile-app/pull/12496#issuecomment-4812134730",
+    "planCapturedAt": "2026-06-26T18:04:49Z",
+    "inputSha256": "e5692e7bc92c4b4998828b50d16001daa42fa3ab9a0b730e95f5d327bf0f37fc",
+    "preOutcomeVerified": true
+  },
+  {
+    "file": "case-10.md",
+    "source": "STAFF-1672 via ClipboardHealth/cbh-admin-frontend#7380",
+    "outcome": "reject",
+    "basis": "The change was closed by a human without integration and without an explicit public rejection rationale; a later replacement removed the alias-lookup mechanism.",
+    "labelStrength": "weak",
+    "correlationNotes": "Cognito/auth family; correlated with cases 01 and 15; weak supersession/closing label.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7380#issuecomment-4895415943",
+    "planCapturedAt": "2026-07-06T16:51:19Z",
+    "inputSha256": "25acba0e96950f929cb14f645f95d3be27cad2c41b0955f47e62948f8187c150",
+    "preOutcomeVerified": true
+  },
+  {
+    "file": "case-11.md",
+    "source": "STAFF-1647 via ClipboardHealth/cbh-admin-frontend#7336",
+    "outcome": "approve",
+    "basis": "The change was integrated and carried the release marker.",
+    "labelStrength": "strong",
+    "correlationNotes": "Product state/URL synchronization case; distinct.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7336#issuecomment-4868335128",
+    "planCapturedAt": "2026-07-02T16:59:23Z",
+    "inputSha256": "60c53588e9b5d1f979de58fad72908bd6920e77215ed7a62e33fb0cc661d9ef0",
+    "preOutcomeVerified": true
+  },
+  {
+    "file": "case-12.md",
+    "source": "STAFF-1690 via ClipboardHealth/cbh-mobile-app#12633",
+    "outcome": "approve",
+    "basis": "The change was integrated and carried the release marker.",
+    "labelStrength": "strong",
+    "correlationNotes": "Mobile interaction state case; distinct.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/cbh-mobile-app/pull/12633#issuecomment-4897732328",
+    "planCapturedAt": "2026-07-06T21:22:49Z",
+    "inputSha256": "c83156c5abace6719c6baf823add55bc34ac41ec10937337845c4dcdae15e1c3",
+    "preOutcomeVerified": true
+  },
+  {
+    "file": "case-13.md",
+    "source": "SYN-2344 via ClipboardHealth/cbh-admin-frontend#6974",
+    "outcome": "reject",
+    "basis": "The change was closed without integration. An over-broad retry concern was corrected before closing, and no explicit human rejection rationale remained.",
+    "labelStrength": "weak",
+    "correlationNotes": "Readiness/retry classification family; weak closed-without-judgment label.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6974#issuecomment-4671371883",
+    "planCapturedAt": "2026-06-10T14:36:01Z",
+    "inputSha256": "bf60a29876b31b9679100ea126791a428c513ff890c0622142bc379bb53a8d9c",
+    "preOutcomeVerified": true
+  },
+  {
+    "file": "case-14.md",
+    "source": "STAFF-1673 via ClipboardHealth/cbh-admin-frontend#7374",
+    "outcome": "approve",
+    "basis": "The change was integrated and carried the release marker.",
+    "labelStrength": "strong",
+    "correlationNotes": "Home-health UI dialog family; related surface to cases 03, 08, and 16.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7374#issuecomment-4894648209",
+    "planCapturedAt": "2026-07-06T15:33:36Z",
+    "inputSha256": "e19ac6b00bd72dfc9490436960ce39507f5dbfe70047ed66978f461270a29c11",
+    "preOutcomeVerified": true
+  },
+  {
+    "file": "case-15.md",
+    "source": "STAFF-1813 via ClipboardHealth/cbh-admin-frontend#7596",
+    "outcome": "approve",
+    "basis": "The change was integrated and carried the release marker.",
+    "labelStrength": "strong",
+    "correlationNotes": "Cognito/auth family; later replacement mechanism related to cases 01 and 10.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7596#issuecomment-4993894763",
+    "planCapturedAt": "2026-07-16T15:50:43Z",
+    "inputSha256": "81c8ec5b3cdc65b98895ce1cd08664952fbc950e7d538a56c4a5bf46ff8ad6ef",
+    "preOutcomeVerified": true
+  },
+  {
+    "file": "case-16.md",
+    "source": "STAFF-1450 via ClipboardHealth/cbh-mobile-app#12270",
+    "outcome": "reject",
+    "basis": "The change was closed by a human without integration after its own CI exposed unresolved environment/read-path behavior; no explicit final reviewer quality verdict was recorded.",
+    "labelStrength": "weak",
+    "correlationNotes": "Home-health data-readiness family; related product surface to cases 03, 08, and 14; weak label.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/cbh-mobile-app/pull/12270#issuecomment-4674532468",
+    "planCapturedAt": "2026-06-10T21:07:36Z",
+    "inputSha256": "81b4a3b8146f4a1a743936f4b2879e12aff5b447f949b6d8d387802245be7294",
+    "preOutcomeVerified": true
+  },
+  {
+    "file": "case-17.md",
+    "source": "STAFF-1555 via ClipboardHealth/cbh-admin-frontend#7250",
+    "outcome": "approve",
+    "basis": "The change was integrated and carried the release marker.",
+    "labelStrength": "strong",
+    "correlationNotes": "Classified auth/setup retry case; distinct from the Cognito alias family.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7250#issuecomment-4835342687",
+    "planCapturedAt": "2026-06-29T17:38:07Z",
+    "inputSha256": "f64498cbfd9e887b95311774adf59ae084f0138297b9b4a82b14d03e17fd2035",
+    "preOutcomeVerified": true
+  },
+  {
+    "file": "case-18.md",
+    "source": "STAFF-1677 via ClipboardHealth/cbh-admin-frontend#7370",
+    "outcome": "approve",
+    "basis": "The change was integrated and carried the release marker.",
+    "labelStrength": "strong",
+    "correlationNotes": "Locator/interaction stabilization case; distinct.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7370#issuecomment-4894317546",
+    "planCapturedAt": "2026-07-06T15:04:15Z",
+    "inputSha256": "9565765afa497f331e747af5b90dfbbebedd08c062b8d82846a8521ac46bbd34",
+    "preOutcomeVerified": true
+  },
+  {
+    "file": "case-19.md",
+    "source": "STAFF-1603 via ClipboardHealth/cbh-admin-frontend#7320",
+    "outcome": "approve",
+    "basis": "The change was integrated and carried the release marker.",
+    "labelStrength": "strong",
+    "correlationNotes": "Magic-link setup retry case, but a different immediate mechanism from cases 01, 10, and 15.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7320#issuecomment-4861122681",
+    "planCapturedAt": "2026-07-02T00:26:10Z",
+    "inputSha256": "8358cc0f6f6228b905c7fbd85f89a27e21d0c086f69c23b85a40ed4126d30ac3",
+    "preOutcomeVerified": true
+  },
+  {
+    "file": "case-20.md",
+    "source": "STAFF-1688 via ClipboardHealth/cbh-admin-frontend#7386",
+    "outcome": "approve",
+    "basis": "The change was integrated and carried the release marker.",
+    "labelStrength": "strong",
+    "correlationNotes": "Worker-creation collision/retry family; correlated with case-05.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7386#issuecomment-4897676448",
+    "planCapturedAt": "2026-07-06T21:15:10Z",
+    "inputSha256": "fd5ae1a438f24ca899dc24e89727bcef8125359888f8920adf84b77b321fea92",
+    "preOutcomeVerified": true
+  }
+]

package/skills/flaky-critic/references/backtests/2026-07-16-b8-prior-attempts/cohort.md

@@ -0,0 +1,37 @@
+# STAFF-1818 blind-backtest cohort
+
+## Cohort definition
+
+- Date range: 2026-06-01 through 2026-07-16, inclusive.
+- Fixed size: 20 historical flaky-test-fix changes.
+- Outcome balance: 14 approve, 6 reject.
+- Selection: every one of the six labeled changes that ended without integration in the interval, plus 14 integrated changes selected for browser E2E, mobile E2E, service/unit-test, product-race, setup/auth, retry-classification, third-party, time-boundary, locator, and CI/config variety.
+- Order: deterministically shuffled into neutral case filenames; outcome classes are not grouped.
+
+## Plan-snapshot source
+
+Each input is the issue-description payload from the timestamped GitHub comment created by the Linear integration when the implementation issue was linked to its change. This captures the contemporaneous plan and embedded evidence at link time, before later human review or closing context. The outer integration HTML, linked issue heading, review link, and all source identifiers are excluded.
+
+Snapshot completeness: all 20 linkback comments contained a readable, non-truncated issue-description payload.
+
+## Audit fields
+
+Each answer-key entry records the exact GitHub permalink of the timestamped Linear-linkback comment (planCommentUrl), its capture timestamp (planCapturedAt), the SHA-256 of the corresponding sanitized case input (inputSha256), and preOutcomeVerified=true. Pre-outcome verification compares the plan-comment timestamp with the integration timestamp for approve cases, the human requested-change review timestamp where one exists, and otherwise the human closure timestamp for weak reject labels. The independently consumable hashes are also listed in input-hashes.txt, sorted by neutral filename.
+
+## Outcome-label caveats
+
+- Thirteen approve cases were verified as integrated and carrying the frontend repositories' release marker.
+- One non-E2E service-test approve case was verified as integrated and reviewed, but its repository does not use that release marker; this is a repository-convention caveat, not a negative signal.
+- Only one reject case has a strong contemporaneous human quality judgment: a requested-change review rejecting the implementation direction.
+- Five reject labels are weak: one stale/not-pursued case, two supersession-like cases, and two human closures without a final public quality rationale. These must not be treated as equally strong ground truth.
+
+## Correlated families
+
+- Cases 01, 10, and 15 share a Cognito/auth identity family; case 15 is a later replacement direction.
+- Cases 05 and 20 share worker-creation collision/retry mechanics.
+- Cases 03, 08, 14, and 16 share a home-health product surface but use distinct immediate mechanisms.
+- Case 19 is also auth-related but concerns a different setup retry mechanism.
+
+## Sanitization
+
+Inputs contain only the historical Linear issue-description snapshot and its contemporaneous embedded evidence. Source repository/change/ticket identifiers, URLs, labels, integration/release language, closing context, session identifiers, attempt-to-fix identifiers, and outcome-revealing comments are excluded. No leak-term survivor is justified; the required scans must return zero matches.

package/skills/flaky-critic/references/backtests/2026-07-16-b8-prior-attempts/input-hashes.txt

@@ -0,0 +1,20 @@
+50bfa6f4e0b4e473e0b9fd3fefa8487d8ff4de218254cd055f2c277953b8ae6f  case-01.md
+5bcc4d6f5f623cd8c661d5880e8d73d85cdcd6f4235609c53bca0c51c8c11eb1  case-02.md
+4c8cadc7b2ef5cdab0628ed5ab1c0ad5fb46089584a26cb883368ed649efc188  case-03.md
+80c44f23858efe66e4c2b8b42da92d5875ad6dfeab5b86a59f7a4573d37901db  case-04.md
+132a76ebf62035c21a70660b8c7c8e829ed5f2dc8e366f38066c44dedc28d6f0  case-05.md
+3d1bba0c70995fc74766927b4a2db256d3d6a3d14d532b6fa26cd4b267476321  case-06.md
+863a3093900c7e1eb2da6b96ad76eeb5f2df8907d2657c64b4dfe9690f4997b2  case-07.md
+365c42ecc41cc10786329a658789113b26442496379c1ddbaeb3a18c50bfe453  case-08.md
+e5692e7bc92c4b4998828b50d16001daa42fa3ab9a0b730e95f5d327bf0f37fc  case-09.md
+25acba0e96950f929cb14f645f95d3be27cad2c41b0955f47e62948f8187c150  case-10.md
+60c53588e9b5d1f979de58fad72908bd6920e77215ed7a62e33fb0cc661d9ef0  case-11.md
+c83156c5abace6719c6baf823add55bc34ac41ec10937337845c4dcdae15e1c3  case-12.md
+bf60a29876b31b9679100ea126791a428c513ff890c0622142bc379bb53a8d9c  case-13.md
+e19ac6b00bd72dfc9490436960ce39507f5dbfe70047ed66978f461270a29c11  case-14.md
+81c8ec5b3cdc65b98895ce1cd08664952fbc950e7d538a56c4a5bf46ff8ad6ef  case-15.md
+81b4a3b8146f4a1a743936f4b2879e12aff5b447f949b6d8d387802245be7294  case-16.md
+f64498cbfd9e887b95311774adf59ae084f0138297b9b4a82b14d03e17fd2035  case-17.md
+9565765afa497f331e747af5b90dfbbebedd08c062b8d82846a8521ac46bbd34  case-18.md
+8358cc0f6f6228b905c7fbd85f89a27e21d0c086f69c23b85a40ed4126d30ac3  case-19.md
+fd5ae1a438f24ca899dc24e89727bcef8125359888f8920adf84b77b321fea92  case-20.md

package/skills/flaky-critic/references/backtests/2026-07-16-b8-prior-attempts/inputs/case-01.md

@@ -0,0 +1,141 @@
+# Historical plan snapshot
+
+## Summary
+
+Fix the shared Cognito magic-link auth failure behind storm `8e9b7951b284`.
+
+The storm is not 26 independent locator flakes. The retained Playwright LLM reports for runs `28788442064` and `28788960641` show the same setup/auth failure: workplace login submits the magic-link form, Cognito `SignUp` returns the expected PreSignUp rejection with a canonical `from_admin_*` identifier, then the app initiates CUSTOM_AUTH with the email anyway and Cognito returns `NotAuthorizedException`. The UI never reaches `Verification link sent`, so unrelated specs fail in their shared sign-in helper.
+
+## Fix confidence
+
+- Fix class: `A5` — product fix for a real auth failure surfaced by tests.
+- Confidence: `4/5`.
+
+The evidence strongly ties this storm to the magic-link path using the submitted email after `ensureCognitoUser` resolves a `from_admin_*` Cognito username. Remaining uncertainty is limited to proving the changed auth flow resolves the staging Cognito path end-to-end for both magic-link request and redeem flows.
+
+## Observability to reach 5/5
+
+- Frontend auth signal: in a focused staging Playwright auth run, magic-link request reaches `Verification link sent`, and the retained network trace shows `InitiateAuth` uses the canonical `from_admin_*` username after the expected PreSignUp alias rejection.
+- Backend/Cognito signal: Datadog or CloudTrail for the same run shows the expected `SignUp` / `UserLambdaValidationException` record, followed by a successful `InitiateAuth` challenge flow for the canonical username, with no corresponding `NotAuthorizedException` for the submitted email.
+- E2E setup signal: password-mode setup either resolves the Cognito username before `AdminSetUserPassword` or provisions users through a path where email alias lookup succeeds; no `UserNotFoundException` remains for the focused auth spec.
+
+## Storm details
+
+- Shared error signature: `Error: expect(locator).toBeVisible() failed`
+- Tests bundled: 26
+- Repos affected: 1
+- CI runs affected: 2
+- First failure: `2026-07-06T11:36:16Z`
+- Last failure: `2026-07-06T11:42:41Z`
+- Investigation ticket: `[ticket redacted]`
+- Affected commit in sightings: `40b3535bdcea`
+- Current `origin/main` checked during investigation: `56b4dfa7131`; auth code path still exists.
+
+## Contained fingerprints
+
+`64980ce0bdd2`, `2c198abdc78b`, `a33392713e47`, `4f6cfcf736b8`, `fb9e31bd68c5`, `d11f1d2b4512`, `c951cd25352a`, `dd738066715a`, `e5474edb897c`, `94decf0326cf`, `65e9ea7238bc`, `9b1da545aaae`, `63641ab33750`, `694b9cafccab`, `faa38bc87438`, `371f41fa08e0`, `89ca2423c39c`, `28e20640d57b`, `f23030f9f53b`, `35b4b5e72a89`, `2d93cd67b48a`, `af4a361ad72d`, `d8a8744ac24b`, `51322020c02f`, `52c0e81a218c`, `3ecad6b809df`.
+
+## Evidence
+
+### Playwright artifacts
+
+Downloaded `playwright-llm-report` artifacts from:
+
+- [[external evidence reference]]([external evidence reference])
+- [[external evidence reference]]([external evidence reference])
+
+Both reports show shard `e2e-playwright-1` failing while the other shards were stopped by the workflow after the auth/setup failure.
+
+Representative retained network body from `auth.spec.ts:69`, first failed attempt:
+
+```json
+{
+  "__type": "UserLambdaValidationException",
+  "message": "PreSignUp failed with error Signup is not available for this account type. Identifier: \"from_admin_334c80ff-9ff0-4fa2-8820-657fb3bbd028\"."
+}
+```
+
+Immediately after that, the same attempt sends CUSTOM_AUTH with `USERNAME` set to the email address and Cognito returns:
+
+```json
+{
+  "__type": "NotAuthorizedException",
+  "message": "Incorrect username or password."
+}
+```
+
+The password-mode auth test also fails before browser login with:
+
+```text
+Cognito user 'playwright-...' does not exist in pool 'us-west-2_HV1ibP3I6' after 5 attempts.
+```
+
+### Datadog / CloudTrail
+
+Environment: SDLC/staging Cognito pool `us-west-2_HV1ibP3I6`, client ID `6v67adqpeaf6bsh0v8o26lrdjm`.
+
+Queries run through `pup logs`:
+
+- `service:cloudtrail environment:sdlc eventSource:cognito-idp.amazonaws.com eventName:SignUp errorCode:UserLambdaValidationException requestParameters.clientId:6v67adqpeaf6bsh0v8o26lrdjm "Signup is not available for this account type"`
+- `service:cloudtrail environment:sdlc eventSource:cognito-idp.amazonaws.com eventName:InitiateAuth errorCode:NotAuthorizedException requestParameters.clientId:6v67adqpeaf6bsh0v8o26lrdjm "Incorrect username or password"`
+- `service:cloudtrail environment:sdlc eventSource:cognito-idp.amazonaws.com eventName:AdminSetUserPassword errorCode:UserNotFoundException`
+
+Counts observed for `2026-07-06T11:00:00Z` to `2026-07-06T12:00:00Z`:
+
+- `SignUp` / `UserLambdaValidationException`: `1838`
+- `InitiateAuth` / `NotAuthorizedException`: `1489`
+- `AdminSetUserPassword` / `UserNotFoundException`: `635`
+
+Samples came from the GitHub runner IP `35.160.3.213` and the `sdlc-github/GitHubActions` role for AdminSetUserPassword, matching CI rather than an end-user-specific outage. Similar `SignUp` expected-rejection records existed before the storm window; the problematic part is the follow-up auth using the email instead of the canonical Cognito username.
+
+No evidence found for this being a schedule/reporting/chat backend outage, CI runner-pool outage, database/cache/queue issue, or independent per-test locator drift. There were routine staging deploy/change events in the broader window, but the failure signature is specific to Cognito identity resolution and the shared login helper.
+
+## Code path to inspect
+
+- `src/appV2/Auth/cognito/cognitoAuth.ts`
+  - `ensureCognitoUser(email)` parses the `Identifier: "from_admin_*"` value and can return the canonical Cognito username.
+  - `requestMagicLink(email)` currently initiates custom auth with the string it receives and stores the same string in the magic-link cache.
+  - `completeMagicLinkSignIn({ payload, email })` re-initiates custom auth with the cached/prompted string.
+- `src/mobile/Login/sendLoginEmail.ts`
+  - The magic-link request path calls `await ensureCognitoUser(email)` and discards the returned username, then calls `requestMagicLink(email)`.
+- `src/appV2/Auth/cognito/emailOtpAuth.ts`
+  - Email OTP already has the correct pattern: resolve username from `ensureCognitoUser`, initiate auth with that username, and carry it for resend.
+- `playwright/helpers/scheduleSeed.ts`
+  - Most affected specs sign in through `signInAsWorkplaceUser`, which waits for `Verification link sent` after clicking `Get Sign-in Link`.
+- `playwright/helpers/cognitoPassword.ts`
+  - Password-mode tests call `AdminSetUserPassword` by email and hit `UserNotFoundException` when the user’s auth identifier is not the email alias.
+
+## Expected implementation direction
+
+Implement the fix in this ticket using the `cb-work` skill, or if unavailable, the `core:go` skill.
+
+Suggested direction:
+
+1. Make the magic-link request/redeem path mirror the canonical-username handling already used by email OTP.
+2. Preserve the user-facing email for UI copy and remember-email behavior, but use the canonical Cognito username for `initiateCustomAuth` when PreSignUp returns a `from_admin_*` identifier.
+3. Consider evolving the magic-link cache from a single email string to a small structured value containing the display/request email plus auth username, with a backward-compatible read path for existing cache entries.
+4. Handle the cache-miss prompt path deliberately: if the user types an email on the redeem page, resolve the Cognito username before re-initiating auth, while avoiding creation of an extra magic-link challenge during redeem.
+5. Add targeted unit coverage for `sendLoginEmail`, `requestMagicLink`/`completeMagicLinkSignIn`, and AuthRedirect cache/redeem behavior. Include coverage for unverified alias users where PreSignUp returns `Identifier: "from_admin_*"`.
+6. Check whether `setCognitoPasswordForUser` should resolve by returned Cognito username for password-mode E2E setup, or whether those password tests should create users through a path that guarantees email alias lookup works.
+
+Do not fix this by increasing Playwright timeouts or adding another retry around `Verification link sent`; the retained network and Datadog evidence show deterministic Cognito auth rejection.
+
+## Required change / commit instructions
+
+- Include this implementation ticket ID in the change body.
+
+```bash
+
+```
+
+## Validation plan
+
+Run the smallest relevant checks first, then broaden as needed:
+
+```bash
+npm run test:v2 -- src/mobile/Login/sendLoginEmail.test.ts src/appV2/Auth/cognito/emailOtpAuth.test.ts src/appV2/Auth/AuthRedirect/AuthRedirect.test.tsx src/appV2/redesign/Login/Login.test.tsx
+npm run lint:fast
+npm run typecheck
+```
+
+If credentials/environment are available, run a focused Playwright auth spec against staging after the unit-level fix.

package/skills/flaky-critic/references/backtests/2026-07-16-b8-prior-attempts/inputs/case-02.md

@@ -0,0 +1,115 @@
+# Historical plan snapshot
+
+## Flake Details
+
+```json
+{
+  "repo": "[backend monolith]",
+  "test": "PUT /api/agentProfile/stage when worker is deactivated should route suspension through workplace-shift-unassignment when revamp flag is on for regular worker",
+  "file": "src/service-test/endpoints/api/agentProfile/stage/put.test.ts:1752",
+  "framework": "jest",
+  "isNewFlaky": false,
+  "failures": [
+    {
+      "error": "expected workplace-shift-unassignment-digest recipient trigger_data shouldSendImmediately=false, received shouldSendImmediately=true with digestSendAt=2026-06-19T23:00:00.000Z",
+      "stack": "at toStrictEqual (/opt/actions-runner/_work/clipboard-health/clipboard-health/src/service-test/endpoints/api/agentProfile/stage/put.test.ts:1894:31)",
+      "branch": "gh-readonly-queue/main/change-[reference redacted]-57af82a5a209cb71849ca6ecbe939a4f0b0f1215",
+      "pipeline": "[external evidence reference]",
+      "commit": "cf1bb0ac5c2e40c60d6285f716f708c7d011c3d4",
+      "durationMs": 2471,
+      "shard": "8/12",
+      "timestamp": "2026-06-18T23:00:23Z"
+    },
+    {
+      "error": "same assertion diff; digestSendAt=2026-06-19T23:00:00.000Z and shouldSendImmediately=true",
+      "pipeline": "[external evidence reference]",
+      "commit": "cf1bb0ac5c2e40c60d6285f716f708c7d011c3d4",
+      "durationMs": 2230,
+      "shard": "8/12",
+      "timestamp": "2026-06-18T23:00:20Z"
+    },
+    {
+      "error": "same assertion diff; digestSendAt=2026-06-19T23:00:00.000Z and shouldSendImmediately=true",
+      "pipeline": "[external evidence reference]",
+      "commit": "cf1bb0ac5c2e40c60d6285f716f708c7d011c3d4",
+      "durationMs": 2339,
+      "shard": "8/12",
+      "timestamp": "2026-06-18T23:00:18Z"
+    }
+  ]
+}
+```
+
+Related investigation: [ticket redacted]
+
+## Plan Output
+
+**Test ID:** ace0c5711269 / [ticket redacted]
+
+**Confidence:** 5/5. The CI failure timestamp, assertion diff, and current code path line up exactly: the test’s earliest shift can land just before the next digest send time, and `shouldSendImmediately` is designed to return `true` in that case.
+
+**Failure surface:** Test setup/data. The service test uses time-sensitive fixture data that does not always satisfy the assertion it makes about digest batching.
+
+**Current main status:** The code path still exists on current `origin/main` (`82e3b188bb`). Recent commit `77276bb1f8` / change [reference redacted] touched this test for push-link data but did not change the `+24h` earliest shift or the `shouldSendImmediately: false` assertion. `gh pr list` found no open `flaky-test-fix` PRs matching the test file, `workplace-shift-unassignment-digest`, or `shouldSendImmediately`.
+
+**Symptom:** The test expects the Knock digest recipient trigger data to contain `shouldSendImmediately: false`, but CI received `shouldSendImmediately: true` with `digestSendAt: "2026-06-19T23:00:00.000Z"`.
+
+**Root cause:** In `src/service-test/endpoints/api/agentProfile/stage/put.test.ts`, the test creates the earliest shift with `addHours(TIME_NOW, 24)`, where `TIME_NOW` is captured at module import. The notification path computes `digestSendAt` later from real `new Date()` in `RecipientResolverService.computeDigestSendAt`, then `ShiftUnassignmentNotificationService` calls `shouldSendImmediately({ digestSendAt, shiftStart: earliestShiftStart })`. Around the 16:00 America/Los_Angeles digest boundary, `TIME_NOW` can be just before 16:00 PT while notification execution is just after 16:00 PT. That makes the next digest `tomorrow 16:00 PT` (`2026-06-19T23:00:00Z` in the failing PDT run), while the test shift is `TIME_NOW + 24h`, a few seconds earlier than that. The production condition `!isBefore(new Date(digestSendAt), shiftStart)` is then true, so the immediate send flag is correct and the test expectation is brittle.
+
+**Evidence:**
+
+- Failure artifact: all three CI failures happened at `2026-06-18T23:00:18Z` through `23:00:23Z`, which is exactly 16:00 PT during PDT.
+- Failure diff: received `digestSendAt: "2026-06-19T23:00:00.000Z"` and `shouldSendImmediately: true`.
+- Test code: `src/service-test/endpoints/api/agentProfile/stage/put.test.ts:1776` creates the earliest shift at `addHours(TIME_NOW, 24)`, and line 1854 expects `shouldSendImmediately: false`.
+- Production code: `src/modules/facility-notifications/logic/recipient-resolver.service.ts` computes the next digest from `new Date()`; `src/modules/facility-notifications/logic/batching.utils.ts` returns true when the digest send time is not before the shift start; `src/modules/facility-notifications/logic/shift-unassignment-notification.service.ts` uses the earliest sorted shift for that decision.
+- Deterministic boundary calculation: if import-time `TIME_NOW` is `2026-06-18T22:59:50Z` and notification execution is `2026-06-18T23:00:23Z`, the shift starts `2026-06-19T22:59:50Z`; the next digest is `2026-06-19T23:00:00Z`; therefore `digestSendAt >= shiftStart` and `shouldSendImmediately` is true.
+
+**Proposed fix:** Test harness/data fix only.
+
+Update `src/service-test/endpoints/api/agentProfile/stage/put.test.ts` in the “when worker is deactivated” test so the shifts used for the digest-batching assertion are far enough in the future for the next digest window to close before the earliest shift. Prefer matching nearby tests by using the existing `getShiftStartTimeAfterThreeDaysInUtc()` / `getShiftEndTimeAfterThreeDaysInUtc()` helpers for the first shift, derive the second shift from those values, and reuse the same variables in `expectedData` instead of recomputing from `TIME_NOW`.
+
+Suggested shape:
+
+```ts
+const firstShiftStart = getShiftStartTimeAfterThreeDaysInUtc();
+const firstShiftEnd = getShiftEndTimeAfterThreeDaysInUtc();
+const secondShiftStart = addHours(new Date(firstShiftStart), 24).toISOString();
+const secondShiftEnd = addHours(new Date(firstShiftEnd), 24).toISOString();
+```
+
+Then create the two shifts with those values and assert `expectedData.shifts` with `firstShiftStart` and `secondShiftStart`. Update the nearby comment from “24h out” to “3 days out”. Do not change production code: the received `true` value is correct when the digest send time is after the shift start.
+
+**Observability to reach 5/5:** N/A -- confidence is 5/5.
+
+**Sibling candidates:** No obvious sibling requiring the same change. A repo search found this is the only facility-notification digest assertion in the inspected service-test set that combines `shouldSendImmediately: false` with `addHours(TIME_NOW, 24)`. Nearby digest tests already use 3-day shifts or explicitly expect `true` for urgent/near-term shifts.
+
+**Validation plan:**
+
+1. Start service-test dependencies:
+
+```sh
+npx tsx src/service-test/service-test-docker.ts up
+```
+
+2. Run the targeted service test:
+
+```sh
+node --run service-test:dir -- src/service-test/endpoints/api/agentProfile/stage/put.test.ts --testNamePattern="should route suspension through workplace-shift-unassignment when revamp flag is on for regular worker"
+```
+
+3. If the implementation touches only this test, no broader suite should be necessary. If shared helpers are changed, also run affected facility-notification digest tests.
+
+**Open questions:** None.
+
+**Residual risk:** This fixes the known boundary flake for this test. Other tests that assert digest batching with fixture times close to the next digest boundary could still be flaky, but the searched sibling candidates already use farther-out shifts or assert the immediate path intentionally.
+
+## Implementation Instructions
+
+- Include this implementation ticket ID in the change body.
+
+```sh
+
+```
+
+- Implement the plan in this ticket using the `cb-work` skill, or if unavailable, the `core:go` skill.
+- Label the change `flaky-test-fix`.