@clipboard-health/ai-rules 2.41.1 → 2.42.0

package/skills/flaky-critic/references/backtests/2026-07-16-b8-prior-attempts/answer-key.json

@@ -0,0 +1,242 @@
+[
+  {
+    "file": "case-01.md",
+    "source": "STAFF-1674 via ClipboardHealth/cbh-admin-frontend#7373",
+    "outcome": "reject",
+    "basis": "A human reviewer requested changes because the proposed implementation preserved an obsolete magic-link test path instead of moving tests to the supported authentication path.",
+    "labelStrength": "strong",
+    "correlationNotes": "Cognito/auth family; correlated with case-10 and case-15.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7373#issuecomment-4894471097",
+    "planCapturedAt": "2026-07-06T15:18:27Z",
+    "inputSha256": "50bfa6f4e0b4e473e0b9fd3fefa8487d8ff4de218254cd055f2c277953b8ae6f",
+    "preOutcomeVerified": true
+  },
+  {
+    "file": "case-02.md",
+    "source": "STAFF-1541 via ClipboardHealth/clipboard-health#26392",
+    "outcome": "approve",
+    "basis": "The change was integrated after automated review approval; this repository does not apply the GitHub release marker used by the frontend repositories.",
+    "labelStrength": "strong",
+    "correlationNotes": "Non-E2E service-test case; no direct mechanism duplicate in cohort.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/clipboard-health/pull/26392#issuecomment-4812847260",
+    "planCapturedAt": "2026-06-26T19:36:01Z",
+    "inputSha256": "5bcc4d6f5f623cd8c661d5880e8d73d85cdcd6f4235609c53bca0c51c8c11eb1",
+    "preOutcomeVerified": true
+  },
+  {
+    "file": "case-03.md",
+    "source": "STAFF-1789 via ClipboardHealth/cbh-admin-frontend#7559",
+    "outcome": "approve",
+    "basis": "The change was integrated and carried the release marker.",
+    "labelStrength": "strong",
+    "correlationNotes": "Home-health UI family; related surface to cases 08, 14, and 16, but distinct mechanism.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7559#issuecomment-4976477226",
+    "planCapturedAt": "2026-07-15T03:08:31Z",
+    "inputSha256": "4c8cadc7b2ef5cdab0628ed5ab1c0ad5fb46089584a26cb883368ed649efc188",
+    "preOutcomeVerified": true
+  },
+  {
+    "file": "case-04.md",
+    "source": "STAFF-1556 via ClipboardHealth/cbh-mobile-app#12521",
+    "outcome": "reject",
+    "basis": "The change was left inactive, marked wontfix, and closed without integration after concern about its CI-time cost; no final human quality verdict was recorded.",
+    "labelStrength": "weak",
+    "correlationNotes": "Static-asset/bootstrap family; weak stale/wontfix label.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/cbh-mobile-app/pull/12521#issuecomment-4835293549",
+    "planCapturedAt": "2026-06-29T17:31:32Z",
+    "inputSha256": "80c44f23858efe66e4c2b8b42da92d5875ad6dfeab5b86a59f7a4573d37901db",
+    "preOutcomeVerified": true
+  },
+  {
+    "file": "case-05.md",
+    "source": "STAFF-1573 via ClipboardHealth/cbh-admin-frontend#7282",
+    "outcome": "approve",
+    "basis": "The change was integrated and carried the release marker.",
+    "labelStrength": "strong",
+    "correlationNotes": "Worker-creation collision/retry family; correlated with case-20.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7282#issuecomment-4845889284",
+    "planCapturedAt": "2026-06-30T16:47:14Z",
+    "inputSha256": "132a76ebf62035c21a70660b8c7c8e829ed5f2dc8e366f38066c44dedc28d6f0",
+    "preOutcomeVerified": true
+  },
+  {
+    "file": "case-06.md",
+    "source": "STAFF-1689 via ClipboardHealth/cbh-mobile-app#12635",
+    "outcome": "approve",
+    "basis": "The change was integrated and carried the release marker.",
+    "labelStrength": "strong",
+    "correlationNotes": "Third-party SDK degradation mechanism; otherwise distinct.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/cbh-mobile-app/pull/12635#issuecomment-4897738607",
+    "planCapturedAt": "2026-07-06T21:23:38Z",
+    "inputSha256": "3d1bba0c70995fc74766927b4a2db256d3d6a3d14d532b6fa26cd4b267476321",
+    "preOutcomeVerified": true
+  },
+  {
+    "file": "case-07.md",
+    "source": "STAFF-1320 via ClipboardHealth/cbh-admin-frontend#6824",
+    "outcome": "reject",
+    "basis": "The change was closed without integration; rubric provenance identifies this historical case as superseded rather than an approach rejection.",
+    "labelStrength": "weak",
+    "correlationNotes": "CI/deployment guard family; weak supersession label.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6824#issuecomment-4624163640",
+    "planCapturedAt": "2026-06-04T16:32:07Z",
+    "inputSha256": "863a3093900c7e1eb2da6b96ad76eeb5f2df8907d2657c64b4dfe9690f4997b2",
+    "preOutcomeVerified": true
+  },
+  {
+    "file": "case-08.md",
+    "source": "STAFF-1685 via ClipboardHealth/cbh-admin-frontend#7390",
+    "outcome": "approve",
+    "basis": "The change was integrated and carried the release marker.",
+    "labelStrength": "strong",
+    "correlationNotes": "Home-health UI family; related surface to cases 03, 14, and 16.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7390#issuecomment-4898990709",
+    "planCapturedAt": "2026-07-07T00:51:18Z",
+    "inputSha256": "365c42ecc41cc10786329a658789113b26442496379c1ddbaeb3a18c50bfe453",
+    "preOutcomeVerified": true
+  },
+  {
+    "file": "case-09.md",
+    "source": "STAFF-1534 via ClipboardHealth/cbh-mobile-app#12496",
+    "outcome": "approve",
+    "basis": "The change was integrated and carried the release marker.",
+    "labelStrength": "strong",
+    "correlationNotes": "Mobile E2E time-wheel interaction case; distinct.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/cbh-mobile-app/pull/12496#issuecomment-4812134730",
+    "planCapturedAt": "2026-06-26T18:04:49Z",
+    "inputSha256": "e5692e7bc92c4b4998828b50d16001daa42fa3ab9a0b730e95f5d327bf0f37fc",
+    "preOutcomeVerified": true
+  },
+  {
+    "file": "case-10.md",
+    "source": "STAFF-1672 via ClipboardHealth/cbh-admin-frontend#7380",
+    "outcome": "reject",
+    "basis": "The change was closed by a human without integration and without an explicit public rejection rationale; a later replacement removed the alias-lookup mechanism.",
+    "labelStrength": "weak",
+    "correlationNotes": "Cognito/auth family; correlated with cases 01 and 15; weak supersession/closing label.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7380#issuecomment-4895415943",
+    "planCapturedAt": "2026-07-06T16:51:19Z",
+    "inputSha256": "25acba0e96950f929cb14f645f95d3be27cad2c41b0955f47e62948f8187c150",
+    "preOutcomeVerified": true
+  },
+  {
+    "file": "case-11.md",
+    "source": "STAFF-1647 via ClipboardHealth/cbh-admin-frontend#7336",
+    "outcome": "approve",
+    "basis": "The change was integrated and carried the release marker.",
+    "labelStrength": "strong",
+    "correlationNotes": "Product state/URL synchronization case; distinct.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7336#issuecomment-4868335128",
+    "planCapturedAt": "2026-07-02T16:59:23Z",
+    "inputSha256": "60c53588e9b5d1f979de58fad72908bd6920e77215ed7a62e33fb0cc661d9ef0",
+    "preOutcomeVerified": true
+  },
+  {
+    "file": "case-12.md",
+    "source": "STAFF-1690 via ClipboardHealth/cbh-mobile-app#12633",
+    "outcome": "approve",
+    "basis": "The change was integrated and carried the release marker.",
+    "labelStrength": "strong",
+    "correlationNotes": "Mobile interaction state case; distinct.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/cbh-mobile-app/pull/12633#issuecomment-4897732328",
+    "planCapturedAt": "2026-07-06T21:22:49Z",
+    "inputSha256": "c83156c5abace6719c6baf823add55bc34ac41ec10937337845c4dcdae15e1c3",
+    "preOutcomeVerified": true
+  },
+  {
+    "file": "case-13.md",
+    "source": "SYN-2344 via ClipboardHealth/cbh-admin-frontend#6974",
+    "outcome": "reject",
+    "basis": "The change was closed without integration. An over-broad retry concern was corrected before closing, and no explicit human rejection rationale remained.",
+    "labelStrength": "weak",
+    "correlationNotes": "Readiness/retry classification family; weak closed-without-judgment label.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6974#issuecomment-4671371883",
+    "planCapturedAt": "2026-06-10T14:36:01Z",
+    "inputSha256": "bf60a29876b31b9679100ea126791a428c513ff890c0622142bc379bb53a8d9c",
+    "preOutcomeVerified": true
+  },
+  {
+    "file": "case-14.md",
+    "source": "STAFF-1673 via ClipboardHealth/cbh-admin-frontend#7374",
+    "outcome": "approve",
+    "basis": "The change was integrated and carried the release marker.",
+    "labelStrength": "strong",
+    "correlationNotes": "Home-health UI dialog family; related surface to cases 03, 08, and 16.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7374#issuecomment-4894648209",
+    "planCapturedAt": "2026-07-06T15:33:36Z",
+    "inputSha256": "e19ac6b00bd72dfc9490436960ce39507f5dbfe70047ed66978f461270a29c11",
+    "preOutcomeVerified": true
+  },
+  {
+    "file": "case-15.md",
+    "source": "STAFF-1813 via ClipboardHealth/cbh-admin-frontend#7596",
+    "outcome": "approve",
+    "basis": "The change was integrated and carried the release marker.",
+    "labelStrength": "strong",
+    "correlationNotes": "Cognito/auth family; later replacement mechanism related to cases 01 and 10.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7596#issuecomment-4993894763",
+    "planCapturedAt": "2026-07-16T15:50:43Z",
+    "inputSha256": "81c8ec5b3cdc65b98895ce1cd08664952fbc950e7d538a56c4a5bf46ff8ad6ef",
+    "preOutcomeVerified": true
+  },
+  {
+    "file": "case-16.md",
+    "source": "STAFF-1450 via ClipboardHealth/cbh-mobile-app#12270",
+    "outcome": "reject",
+    "basis": "The change was closed by a human without integration after its own CI exposed unresolved environment/read-path behavior; no explicit final reviewer quality verdict was recorded.",
+    "labelStrength": "weak",
+    "correlationNotes": "Home-health data-readiness family; related product surface to cases 03, 08, and 14; weak label.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/cbh-mobile-app/pull/12270#issuecomment-4674532468",
+    "planCapturedAt": "2026-06-10T21:07:36Z",
+    "inputSha256": "81b4a3b8146f4a1a743936f4b2879e12aff5b447f949b6d8d387802245be7294",
+    "preOutcomeVerified": true
+  },
+  {
+    "file": "case-17.md",
+    "source": "STAFF-1555 via ClipboardHealth/cbh-admin-frontend#7250",
+    "outcome": "approve",
+    "basis": "The change was integrated and carried the release marker.",
+    "labelStrength": "strong",
+    "correlationNotes": "Classified auth/setup retry case; distinct from the Cognito alias family.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7250#issuecomment-4835342687",
+    "planCapturedAt": "2026-06-29T17:38:07Z",
+    "inputSha256": "f64498cbfd9e887b95311774adf59ae084f0138297b9b4a82b14d03e17fd2035",
+    "preOutcomeVerified": true
+  },
+  {
+    "file": "case-18.md",
+    "source": "STAFF-1677 via ClipboardHealth/cbh-admin-frontend#7370",
+    "outcome": "approve",
+    "basis": "The change was integrated and carried the release marker.",
+    "labelStrength": "strong",
+    "correlationNotes": "Locator/interaction stabilization case; distinct.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7370#issuecomment-4894317546",
+    "planCapturedAt": "2026-07-06T15:04:15Z",
+    "inputSha256": "9565765afa497f331e747af5b90dfbbebedd08c062b8d82846a8521ac46bbd34",
+    "preOutcomeVerified": true
+  },
+  {
+    "file": "case-19.md",
+    "source": "STAFF-1603 via ClipboardHealth/cbh-admin-frontend#7320",
+    "outcome": "approve",
+    "basis": "The change was integrated and carried the release marker.",
+    "labelStrength": "strong",
+    "correlationNotes": "Magic-link setup retry case, but a different immediate mechanism from cases 01, 10, and 15.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7320#issuecomment-4861122681",
+    "planCapturedAt": "2026-07-02T00:26:10Z",
+    "inputSha256": "8358cc0f6f6228b905c7fbd85f89a27e21d0c086f69c23b85a40ed4126d30ac3",
+    "preOutcomeVerified": true
+  },
+  {
+    "file": "case-20.md",
+    "source": "STAFF-1688 via ClipboardHealth/cbh-admin-frontend#7386",
+    "outcome": "approve",
+    "basis": "The change was integrated and carried the release marker.",
+    "labelStrength": "strong",
+    "correlationNotes": "Worker-creation collision/retry family; correlated with case-05.",
+    "planCommentUrl": "https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7386#issuecomment-4897676448",
+    "planCapturedAt": "2026-07-06T21:15:10Z",
+    "inputSha256": "fd5ae1a438f24ca899dc24e89727bcef8125359888f8920adf84b77b321fea92",
+    "preOutcomeVerified": true
+  }
+]

package/skills/flaky-critic/references/backtests/2026-07-16-b8-prior-attempts/cohort.md

@@ -0,0 +1,37 @@
+# STAFF-1818 blind-backtest cohort
+
+## Cohort definition
+
+- Date range: 2026-06-01 through 2026-07-16, inclusive.
+- Fixed size: 20 historical flaky-test-fix changes.
+- Outcome balance: 14 approve, 6 reject.
+- Selection: every one of the six labeled changes that ended without integration in the interval, plus 14 integrated changes selected for browser E2E, mobile E2E, service/unit-test, product-race, setup/auth, retry-classification, third-party, time-boundary, locator, and CI/config variety.
+- Order: deterministically shuffled into neutral case filenames; outcome classes are not grouped.
+
+## Plan-snapshot source
+
+Each input is the issue-description payload from the timestamped GitHub comment created by the Linear integration when the implementation issue was linked to its change. This captures the contemporaneous plan and embedded evidence at link time, before later human review or closing context. The outer integration HTML, linked issue heading, review link, and all source identifiers are excluded.
+
+Snapshot completeness: all 20 linkback comments contained a readable, non-truncated issue-description payload.
+
+## Audit fields
+
+Each answer-key entry records the exact GitHub permalink of the timestamped Linear-linkback comment (planCommentUrl), its capture timestamp (planCapturedAt), the SHA-256 of the corresponding sanitized case input (inputSha256), and preOutcomeVerified=true. Pre-outcome verification compares the plan-comment timestamp with the integration timestamp for approve cases, the human requested-change review timestamp where one exists, and otherwise the human closure timestamp for weak reject labels. The independently consumable hashes are also listed in input-hashes.txt, sorted by neutral filename.
+
+## Outcome-label caveats
+
+- Thirteen approve cases were verified as integrated and carrying the frontend repositories' release marker.
+- One non-E2E service-test approve case was verified as integrated and reviewed, but its repository does not use that release marker; this is a repository-convention caveat, not a negative signal.
+- Only one reject case has a strong contemporaneous human quality judgment: a requested-change review rejecting the implementation direction.
+- Five reject labels are weak: one stale/not-pursued case, two supersession-like cases, and two human closures without a final public quality rationale. These must not be treated as equally strong ground truth.
+
+## Correlated families
+
+- Cases 01, 10, and 15 share a Cognito/auth identity family; case 15 is a later replacement direction.
+- Cases 05 and 20 share worker-creation collision/retry mechanics.
+- Cases 03, 08, 14, and 16 share a home-health product surface but use distinct immediate mechanisms.
+- Case 19 is also auth-related but concerns a different setup retry mechanism.
+
+## Sanitization
+
+Inputs contain only the historical Linear issue-description snapshot and its contemporaneous embedded evidence. Source repository/change/ticket identifiers, URLs, labels, integration/release language, closing context, session identifiers, attempt-to-fix identifiers, and outcome-revealing comments are excluded. No leak-term survivor is justified; the required scans must return zero matches.

package/skills/flaky-critic/references/backtests/2026-07-16-b8-prior-attempts/input-hashes.txt

@@ -0,0 +1,20 @@
+50bfa6f4e0b4e473e0b9fd3fefa8487d8ff4de218254cd055f2c277953b8ae6f  case-01.md
+5bcc4d6f5f623cd8c661d5880e8d73d85cdcd6f4235609c53bca0c51c8c11eb1  case-02.md
+4c8cadc7b2ef5cdab0628ed5ab1c0ad5fb46089584a26cb883368ed649efc188  case-03.md
+80c44f23858efe66e4c2b8b42da92d5875ad6dfeab5b86a59f7a4573d37901db  case-04.md
+132a76ebf62035c21a70660b8c7c8e829ed5f2dc8e366f38066c44dedc28d6f0  case-05.md
+3d1bba0c70995fc74766927b4a2db256d3d6a3d14d532b6fa26cd4b267476321  case-06.md
+863a3093900c7e1eb2da6b96ad76eeb5f2df8907d2657c64b4dfe9690f4997b2  case-07.md
+365c42ecc41cc10786329a658789113b26442496379c1ddbaeb3a18c50bfe453  case-08.md
+e5692e7bc92c4b4998828b50d16001daa42fa3ab9a0b730e95f5d327bf0f37fc  case-09.md
+25acba0e96950f929cb14f645f95d3be27cad2c41b0955f47e62948f8187c150  case-10.md
+60c53588e9b5d1f979de58fad72908bd6920e77215ed7a62e33fb0cc661d9ef0  case-11.md
+c83156c5abace6719c6baf823add55bc34ac41ec10937337845c4dcdae15e1c3  case-12.md
+bf60a29876b31b9679100ea126791a428c513ff890c0622142bc379bb53a8d9c  case-13.md
+e19ac6b00bd72dfc9490436960ce39507f5dbfe70047ed66978f461270a29c11  case-14.md
+81c8ec5b3cdc65b98895ce1cd08664952fbc950e7d538a56c4a5bf46ff8ad6ef  case-15.md
+81b4a3b8146f4a1a743936f4b2879e12aff5b447f949b6d8d387802245be7294  case-16.md
+f64498cbfd9e887b95311774adf59ae084f0138297b9b4a82b14d03e17fd2035  case-17.md
+9565765afa497f331e747af5b90dfbbebedd08c062b8d82846a8521ac46bbd34  case-18.md
+8358cc0f6f6228b905c7fbd85f89a27e21d0c086f69c23b85a40ed4126d30ac3  case-19.md
+fd5ae1a438f24ca899dc24e89727bcef8125359888f8920adf84b77b321fea92  case-20.md

package/skills/flaky-critic/references/backtests/2026-07-16-b8-prior-attempts/inputs/case-01.md

@@ -0,0 +1,141 @@
+# Historical plan snapshot
+
+## Summary
+
+Fix the shared Cognito magic-link auth failure behind storm `8e9b7951b284`.
+
+The storm is not 26 independent locator flakes. The retained Playwright LLM reports for runs `28788442064` and `28788960641` show the same setup/auth failure: workplace login submits the magic-link form, Cognito `SignUp` returns the expected PreSignUp rejection with a canonical `from_admin_*` identifier, then the app initiates CUSTOM_AUTH with the email anyway and Cognito returns `NotAuthorizedException`. The UI never reaches `Verification link sent`, so unrelated specs fail in their shared sign-in helper.
+
+## Fix confidence
+
+- Fix class: `A5` — product fix for a real auth failure surfaced by tests.
+- Confidence: `4/5`.
+
+The evidence strongly ties this storm to the magic-link path using the submitted email after `ensureCognitoUser` resolves a `from_admin_*` Cognito username. Remaining uncertainty is limited to proving the changed auth flow resolves the staging Cognito path end-to-end for both magic-link request and redeem flows.
+
+## Observability to reach 5/5
+
+- Frontend auth signal: in a focused staging Playwright auth run, magic-link request reaches `Verification link sent`, and the retained network trace shows `InitiateAuth` uses the canonical `from_admin_*` username after the expected PreSignUp alias rejection.
+- Backend/Cognito signal: Datadog or CloudTrail for the same run shows the expected `SignUp` / `UserLambdaValidationException` record, followed by a successful `InitiateAuth` challenge flow for the canonical username, with no corresponding `NotAuthorizedException` for the submitted email.
+- E2E setup signal: password-mode setup either resolves the Cognito username before `AdminSetUserPassword` or provisions users through a path where email alias lookup succeeds; no `UserNotFoundException` remains for the focused auth spec.
+
+## Storm details
+
+- Shared error signature: `Error: expect(locator).toBeVisible() failed`
+- Tests bundled: 26
+- Repos affected: 1
+- CI runs affected: 2
+- First failure: `2026-07-06T11:36:16Z`
+- Last failure: `2026-07-06T11:42:41Z`
+- Investigation ticket: `[ticket redacted]`
+- Affected commit in sightings: `40b3535bdcea`
+- Current `origin/main` checked during investigation: `56b4dfa7131`; auth code path still exists.
+
+## Contained fingerprints
+
+`64980ce0bdd2`, `2c198abdc78b`, `a33392713e47`, `4f6cfcf736b8`, `fb9e31bd68c5`, `d11f1d2b4512`, `c951cd25352a`, `dd738066715a`, `e5474edb897c`, `94decf0326cf`, `65e9ea7238bc`, `9b1da545aaae`, `63641ab33750`, `694b9cafccab`, `faa38bc87438`, `371f41fa08e0`, `89ca2423c39c`, `28e20640d57b`, `f23030f9f53b`, `35b4b5e72a89`, `2d93cd67b48a`, `af4a361ad72d`, `d8a8744ac24b`, `51322020c02f`, `52c0e81a218c`, `3ecad6b809df`.
+
+## Evidence
+
+### Playwright artifacts
+
+Downloaded `playwright-llm-report` artifacts from:
+
+- [[external evidence reference]]([external evidence reference])
+- [[external evidence reference]]([external evidence reference])
+
+Both reports show shard `e2e-playwright-1` failing while the other shards were stopped by the workflow after the auth/setup failure.
+
+Representative retained network body from `auth.spec.ts:69`, first failed attempt:
+
+```json
+{
+  "__type": "UserLambdaValidationException",
+  "message": "PreSignUp failed with error Signup is not available for this account type. Identifier: \"from_admin_334c80ff-9ff0-4fa2-8820-657fb3bbd028\"."
+}
+```
+
+Immediately after that, the same attempt sends CUSTOM_AUTH with `USERNAME` set to the email address and Cognito returns:
+
+```json
+{
+  "__type": "NotAuthorizedException",
+  "message": "Incorrect username or password."
+}
+```
+
+The password-mode auth test also fails before browser login with:
+
+```text
+Cognito user 'playwright-...' does not exist in pool 'us-west-2_HV1ibP3I6' after 5 attempts.
+```
+
+### Datadog / CloudTrail
+
+Environment: SDLC/staging Cognito pool `us-west-2_HV1ibP3I6`, client ID `6v67adqpeaf6bsh0v8o26lrdjm`.
+
+Queries run through `pup logs`:
+
+- `service:cloudtrail environment:sdlc eventSource:cognito-idp.amazonaws.com eventName:SignUp errorCode:UserLambdaValidationException requestParameters.clientId:6v67adqpeaf6bsh0v8o26lrdjm "Signup is not available for this account type"`
+- `service:cloudtrail environment:sdlc eventSource:cognito-idp.amazonaws.com eventName:InitiateAuth errorCode:NotAuthorizedException requestParameters.clientId:6v67adqpeaf6bsh0v8o26lrdjm "Incorrect username or password"`
+- `service:cloudtrail environment:sdlc eventSource:cognito-idp.amazonaws.com eventName:AdminSetUserPassword errorCode:UserNotFoundException`
+
+Counts observed for `2026-07-06T11:00:00Z` to `2026-07-06T12:00:00Z`:
+
+- `SignUp` / `UserLambdaValidationException`: `1838`
+- `InitiateAuth` / `NotAuthorizedException`: `1489`
+- `AdminSetUserPassword` / `UserNotFoundException`: `635`
+
+Samples came from the GitHub runner IP `35.160.3.213` and the `sdlc-github/GitHubActions` role for AdminSetUserPassword, matching CI rather than an end-user-specific outage. Similar `SignUp` expected-rejection records existed before the storm window; the problematic part is the follow-up auth using the email instead of the canonical Cognito username.
+
+No evidence found for this being a schedule/reporting/chat backend outage, CI runner-pool outage, database/cache/queue issue, or independent per-test locator drift. There were routine staging deploy/change events in the broader window, but the failure signature is specific to Cognito identity resolution and the shared login helper.
+
+## Code path to inspect
+
+- `src/appV2/Auth/cognito/cognitoAuth.ts`
+  - `ensureCognitoUser(email)` parses the `Identifier: "from_admin_*"` value and can return the canonical Cognito username.
+  - `requestMagicLink(email)` currently initiates custom auth with the string it receives and stores the same string in the magic-link cache.
+  - `completeMagicLinkSignIn({ payload, email })` re-initiates custom auth with the cached/prompted string.
+- `src/mobile/Login/sendLoginEmail.ts`
+  - The magic-link request path calls `await ensureCognitoUser(email)` and discards the returned username, then calls `requestMagicLink(email)`.
+- `src/appV2/Auth/cognito/emailOtpAuth.ts`
+  - Email OTP already has the correct pattern: resolve username from `ensureCognitoUser`, initiate auth with that username, and carry it for resend.
+- `playwright/helpers/scheduleSeed.ts`
+  - Most affected specs sign in through `signInAsWorkplaceUser`, which waits for `Verification link sent` after clicking `Get Sign-in Link`.
+- `playwright/helpers/cognitoPassword.ts`
+  - Password-mode tests call `AdminSetUserPassword` by email and hit `UserNotFoundException` when the user’s auth identifier is not the email alias.
+
+## Expected implementation direction
+
+Implement the fix in this ticket using the `cb-work` skill, or if unavailable, the `core:go` skill.
+
+Suggested direction:
+
+1. Make the magic-link request/redeem path mirror the canonical-username handling already used by email OTP.
+2. Preserve the user-facing email for UI copy and remember-email behavior, but use the canonical Cognito username for `initiateCustomAuth` when PreSignUp returns a `from_admin_*` identifier.
+3. Consider evolving the magic-link cache from a single email string to a small structured value containing the display/request email plus auth username, with a backward-compatible read path for existing cache entries.
+4. Handle the cache-miss prompt path deliberately: if the user types an email on the redeem page, resolve the Cognito username before re-initiating auth, while avoiding creation of an extra magic-link challenge during redeem.
+5. Add targeted unit coverage for `sendLoginEmail`, `requestMagicLink`/`completeMagicLinkSignIn`, and AuthRedirect cache/redeem behavior. Include coverage for unverified alias users where PreSignUp returns `Identifier: "from_admin_*"`.
+6. Check whether `setCognitoPasswordForUser` should resolve by returned Cognito username for password-mode E2E setup, or whether those password tests should create users through a path that guarantees email alias lookup works.
+
+Do not fix this by increasing Playwright timeouts or adding another retry around `Verification link sent`; the retained network and Datadog evidence show deterministic Cognito auth rejection.
+
+## Required change / commit instructions
+
+- Include this implementation ticket ID in the change body.
+
+```bash
+
+```
+
+## Validation plan
+
+Run the smallest relevant checks first, then broaden as needed:
+
+```bash
+npm run test:v2 -- src/mobile/Login/sendLoginEmail.test.ts src/appV2/Auth/cognito/emailOtpAuth.test.ts src/appV2/Auth/AuthRedirect/AuthRedirect.test.tsx src/appV2/redesign/Login/Login.test.tsx
+npm run lint:fast
+npm run typecheck
+```
+
+If credentials/environment are available, run a focused Playwright auth spec against staging after the unit-level fix.

package/skills/flaky-critic/references/backtests/2026-07-16-b8-prior-attempts/inputs/case-02.md

@@ -0,0 +1,115 @@
+# Historical plan snapshot
+
+## Flake Details
+
+```json
+{
+  "repo": "[backend monolith]",
+  "test": "PUT /api/agentProfile/stage when worker is deactivated should route suspension through workplace-shift-unassignment when revamp flag is on for regular worker",
+  "file": "src/service-test/endpoints/api/agentProfile/stage/put.test.ts:1752",
+  "framework": "jest",
+  "isNewFlaky": false,
+  "failures": [
+    {
+      "error": "expected workplace-shift-unassignment-digest recipient trigger_data shouldSendImmediately=false, received shouldSendImmediately=true with digestSendAt=2026-06-19T23:00:00.000Z",
+      "stack": "at toStrictEqual (/opt/actions-runner/_work/clipboard-health/clipboard-health/src/service-test/endpoints/api/agentProfile/stage/put.test.ts:1894:31)",
+      "branch": "gh-readonly-queue/main/change-[reference redacted]-57af82a5a209cb71849ca6ecbe939a4f0b0f1215",
+      "pipeline": "[external evidence reference]",
+      "commit": "cf1bb0ac5c2e40c60d6285f716f708c7d011c3d4",
+      "durationMs": 2471,
+      "shard": "8/12",
+      "timestamp": "2026-06-18T23:00:23Z"
+    },
+    {
+      "error": "same assertion diff; digestSendAt=2026-06-19T23:00:00.000Z and shouldSendImmediately=true",
+      "pipeline": "[external evidence reference]",
+      "commit": "cf1bb0ac5c2e40c60d6285f716f708c7d011c3d4",
+      "durationMs": 2230,
+      "shard": "8/12",
+      "timestamp": "2026-06-18T23:00:20Z"
+    },
+    {
+      "error": "same assertion diff; digestSendAt=2026-06-19T23:00:00.000Z and shouldSendImmediately=true",
+      "pipeline": "[external evidence reference]",
+      "commit": "cf1bb0ac5c2e40c60d6285f716f708c7d011c3d4",
+      "durationMs": 2339,
+      "shard": "8/12",
+      "timestamp": "2026-06-18T23:00:18Z"
+    }
+  ]
+}
+```
+
+Related investigation: [ticket redacted]
+
+## Plan Output
+
+**Test ID:** ace0c5711269 / [ticket redacted]
+
+**Confidence:** 5/5. The CI failure timestamp, assertion diff, and current code path line up exactly: the test’s earliest shift can land just before the next digest send time, and `shouldSendImmediately` is designed to return `true` in that case.
+
+**Failure surface:** Test setup/data. The service test uses time-sensitive fixture data that does not always satisfy the assertion it makes about digest batching.
+
+**Current main status:** The code path still exists on current `origin/main` (`82e3b188bb`). Recent commit `77276bb1f8` / change [reference redacted] touched this test for push-link data but did not change the `+24h` earliest shift or the `shouldSendImmediately: false` assertion. `gh pr list` found no open `flaky-test-fix` PRs matching the test file, `workplace-shift-unassignment-digest`, or `shouldSendImmediately`.
+
+**Symptom:** The test expects the Knock digest recipient trigger data to contain `shouldSendImmediately: false`, but CI received `shouldSendImmediately: true` with `digestSendAt: "2026-06-19T23:00:00.000Z"`.
+
+**Root cause:** In `src/service-test/endpoints/api/agentProfile/stage/put.test.ts`, the test creates the earliest shift with `addHours(TIME_NOW, 24)`, where `TIME_NOW` is captured at module import. The notification path computes `digestSendAt` later from real `new Date()` in `RecipientResolverService.computeDigestSendAt`, then `ShiftUnassignmentNotificationService` calls `shouldSendImmediately({ digestSendAt, shiftStart: earliestShiftStart })`. Around the 16:00 America/Los_Angeles digest boundary, `TIME_NOW` can be just before 16:00 PT while notification execution is just after 16:00 PT. That makes the next digest `tomorrow 16:00 PT` (`2026-06-19T23:00:00Z` in the failing PDT run), while the test shift is `TIME_NOW + 24h`, a few seconds earlier than that. The production condition `!isBefore(new Date(digestSendAt), shiftStart)` is then true, so the immediate send flag is correct and the test expectation is brittle.
+
+**Evidence:**
+
+- Failure artifact: all three CI failures happened at `2026-06-18T23:00:18Z` through `23:00:23Z`, which is exactly 16:00 PT during PDT.
+- Failure diff: received `digestSendAt: "2026-06-19T23:00:00.000Z"` and `shouldSendImmediately: true`.
+- Test code: `src/service-test/endpoints/api/agentProfile/stage/put.test.ts:1776` creates the earliest shift at `addHours(TIME_NOW, 24)`, and line 1854 expects `shouldSendImmediately: false`.
+- Production code: `src/modules/facility-notifications/logic/recipient-resolver.service.ts` computes the next digest from `new Date()`; `src/modules/facility-notifications/logic/batching.utils.ts` returns true when the digest send time is not before the shift start; `src/modules/facility-notifications/logic/shift-unassignment-notification.service.ts` uses the earliest sorted shift for that decision.
+- Deterministic boundary calculation: if import-time `TIME_NOW` is `2026-06-18T22:59:50Z` and notification execution is `2026-06-18T23:00:23Z`, the shift starts `2026-06-19T22:59:50Z`; the next digest is `2026-06-19T23:00:00Z`; therefore `digestSendAt >= shiftStart` and `shouldSendImmediately` is true.
+
+**Proposed fix:** Test harness/data fix only.
+
+Update `src/service-test/endpoints/api/agentProfile/stage/put.test.ts` in the “when worker is deactivated” test so the shifts used for the digest-batching assertion are far enough in the future for the next digest window to close before the earliest shift. Prefer matching nearby tests by using the existing `getShiftStartTimeAfterThreeDaysInUtc()` / `getShiftEndTimeAfterThreeDaysInUtc()` helpers for the first shift, derive the second shift from those values, and reuse the same variables in `expectedData` instead of recomputing from `TIME_NOW`.
+
+Suggested shape:
+
+```ts
+const firstShiftStart = getShiftStartTimeAfterThreeDaysInUtc();
+const firstShiftEnd = getShiftEndTimeAfterThreeDaysInUtc();
+const secondShiftStart = addHours(new Date(firstShiftStart), 24).toISOString();
+const secondShiftEnd = addHours(new Date(firstShiftEnd), 24).toISOString();
+```
+
+Then create the two shifts with those values and assert `expectedData.shifts` with `firstShiftStart` and `secondShiftStart`. Update the nearby comment from “24h out” to “3 days out”. Do not change production code: the received `true` value is correct when the digest send time is after the shift start.
+
+**Observability to reach 5/5:** N/A -- confidence is 5/5.
+
+**Sibling candidates:** No obvious sibling requiring the same change. A repo search found this is the only facility-notification digest assertion in the inspected service-test set that combines `shouldSendImmediately: false` with `addHours(TIME_NOW, 24)`. Nearby digest tests already use 3-day shifts or explicitly expect `true` for urgent/near-term shifts.
+
+**Validation plan:**
+
+1. Start service-test dependencies:
+
+```sh
+npx tsx src/service-test/service-test-docker.ts up
+```
+
+2. Run the targeted service test:
+
+```sh
+node --run service-test:dir -- src/service-test/endpoints/api/agentProfile/stage/put.test.ts --testNamePattern="should route suspension through workplace-shift-unassignment when revamp flag is on for regular worker"
+```
+
+3. If the implementation touches only this test, no broader suite should be necessary. If shared helpers are changed, also run affected facility-notification digest tests.
+
+**Open questions:** None.
+
+**Residual risk:** This fixes the known boundary flake for this test. Other tests that assert digest batching with fixture times close to the next digest boundary could still be flaky, but the searched sibling candidates already use farther-out shifts or assert the immediate path intentionally.
+
+## Implementation Instructions
+
+- Include this implementation ticket ID in the change body.
+
+```sh
+
+```
+
+- Implement the plan in this ticket using the `cb-work` skill, or if unavailable, the `core:go` skill.
+- Label the change `flaky-test-fix`.

package/skills/flaky-critic/references/backtests/2026-07-16-b8-prior-attempts/inputs/case-03.md

@@ -0,0 +1,24 @@
+# Historical plan snapshot
+
+## Groundcrew
+
+Repository: [admin frontend]
+Implementation workflow: use the `core:cb-work`/`cb-work` skill when available. If that skill is unavailable, follow this repo's [AGENTS.md/CLAUDE.md]([external evidence reference]) implementation workflow and run the documented verification.
+
+## Task
+
+A deep dive on the chronic Home Health fullLifecycle flake (fingerprint family 33deef731a10; [ticket family redacted] — recurred after 6+ times) diagnosed the root cause at 4/5 confidence:
+
+Every HH action dialog (`CaseClosureDialog`, `CancelVisitDialog`, etc.) renders as a React descendant of a react-query list item (`CaseCard` from `useAgencyCases`, `VisitCard` from `useCaseVisits`). Those lists are invalidated by the dialogs' own mutations (`useUpdateVisit.ts:66`, `useUpdateCase.ts:44-56`) and also refetch on a 5-min `staleTime` / 10-min `refetchInterval` / window-focus cadence. When a refetch resolves while a dialog is open, the list item reconciles ("element is not stable") or drops out of the filtered result ("element was detached from the DOM") — exactly the signatures in [ticket redacted] (detached Close button at `submitCloseCase`) and [ticket redacted] (flapping cancel-dialog title).
+
+This ticket is the phase-1 mitigation: while any HH action dialog is open, suppress `refetchOnWindowFocus`/`refetchInterval` on the backing list queries and defer query invalidation until the dialog closes (flush on close). Do NOT add waits, retries, or locator changes to the specs — the 12 prior stabilization commits did that and the flake recurred each time (rubric B4).
+
+## Acceptance Criteria
+
+- [ ] While an HH dialog is open, the backing list queries neither interval/focus-refetch nor process invalidations; deferred invalidations flush on dialog close.
+- [ ] Fault-injection validation: force a list refetch while a dialog is open — before the change this reproduces the detached/unstable signature, after it does not. If the environment cannot run e2e against staging, cover with a component-level test that simulates refetch-while-open.
+- [ ] Existing HH unit/component tests and lint/typecheck pass.
+
+## Notes
+
+The durable fix (hoisting dialogs into a stable dialog host above `cases.map`) and fullLifecycle spec decomposition are a separate ticket. Full deep-dive document: ask Rocky for `deep-dives/hh-full-lifecycle.md`.