@clipboard-health/ai-rules 2.41.0 → 2.41.2

package/skills/flaky-critic/references/backtests/2026-07-16-b8-prior-attempts/inputs/case-14.md

@@ -0,0 +1,111 @@
+# Historical plan snapshot
+
+## Groundcrew
+
+Repository: [admin frontend]
+Implementation workflow: use the `cb-work` skill. If unavailable, use the `core:go` skill.
+
+Include this implementation ticket ID in the change body.
+
+```bash
+
+```
+
+## Flake Details
+
+```json
+{
+  "repo": "[admin frontend]",
+  "test": "Home Health Full Lifecycle complete home health workflow: create case, manage visits, verify and reject occurrences, close case",
+  "file": "playwright/e2e/homeHealth/fullLifecycle.spec.ts:876",
+  "framework": "playwright",
+  "isNewFlaky": false,
+  "failures": [
+    {
+      "error": "Error: expect(locator).toBeVisible() failed\n\nLocator: getByRole('dialog').getByText('Cancel Regular visit')\nExpected: visible\nTimeout: 30000ms\nError: element(s) not found\n\nCall log:\n  - Expect \"toBeVisible\" with timeout 30000ms\n  - waiting for getByRole('dialog').getByText('Cancel Regular visit')\n",
+      "stack": "Error: expect(locator).toBeVisible() failed\n\nLocator: getByRole('dialog').getByText('Cancel Regular visit')\nExpected: visible\nTimeout: 30000ms\nError: element(s) not found\n\nCall log:\n  - Expect \"toBeVisible\" with timeout 30000ms\n  - waiting for getByRole('dialog').getByText('Cancel Regular visit')\n\n    at /opt/actions-runner/_work/[admin frontend]/[admin frontend]/playwright/e2e/homeHealth/fullLifecycle.spec.ts:1272:81\n    at /opt/actions-runner/_work/[admin frontend]/[admin frontend]/playwright/e2e/homeHealth/fullLifecycle.spec.ts:1252:7\n    at WorkerMain._runTest (/opt/actions-runner/_work/[admin frontend]/[admin frontend]/node_modules/dd-trace/packages/datadog-instrumentations/src/playwright.js:1692:5)",
+      "branch": "main",
+      "pipeline": "[external evidence reference]",
+      "commit": "c0fc2cd743575ada27d544cea455e13cd45caf28",
+      "durationMs": 51423,
+      "shard": "2/4",
+      "timestamp": "2026-07-05T01:07:09Z"
+    }
+  ]
+}
+```
+
+## Investigation Plan Output
+
+**Test ID:** [ticket redacted] / sighting-key `a5a77ac290d27dc9`
+
+**Confidence:** 4/5. The trace and failure screenshot directly show the cancel button click completed, no dialog was present afterward, and the case visits section was collapsed at failure. The exact React state transition that collapsed/unmounted the visits subtree is inferred rather than directly instrumented.
+
+**Failure surface:** User action no-op / test harness race. The action target existed and Playwright completed the click, but the expected dialog did not start/render before the visit subtree was collapsed/unmounted. No cancellation backend request was emitted.
+
+**Current main status:** Still present on current `origin/main` / HEAD `56b4dfa7131b6f6b84989145d8f0c2657811c252`. The failing path in `playwright/e2e/homeHealth/fullLifecycle.spec.ts` still does a single click followed by `page.getByRole("dialog").getByText("Cancel Regular visit")`. `CaseCard` still stores expansion state locally with `useModalState(ModalStatus.ended)` and `Collapse unmountOnExit`, and `CancelVisitDialog` still lives under `VisitActions` inside that collapsible visit subtree.
+
+**Symptom:** In Step e, `findVisitRowByWorker` found the booked worker row and the row's `Cancel Visit` button. The test clicked it, then timed out for 30s waiting for `getByRole('dialog').getByText('Cancel Regular visit')` at `playwright/e2e/homeHealth/fullLifecycle.spec.ts:1272`.
+
+**Root Cause:** The cancel-visit path is less resilient than the other hardened home-health dialog helpers. It assumes the visit subtree remains expanded and that one button click will synchronously expose the cancel dialog. In the failed attempt, the subtree was collapsed by the time the dialog assertion began, so the `CancelVisitDialog` mounted under `VisitActions` was not present. Because the helper returns a `Locator` after finding the row but does not re-check expansion or retry opening the dialog, the test can lose the row/dialog state between row lookup and assertion.
+
+**Evidence:**
+
+- LLM report artifact `playwright-llm-report` for run [reference redacted]` has schema v3 summary: 52 total tests, 49 passed, 1 flaky, 0 final failures. This test failed once and passed on retry.
+- Failed attempt error: `getByRole('dialog').getByText('Cancel Regular visit')` timed out after 30000ms at line 1272.
+- Failed attempt trace: `findVisitRowByWorker` reloaded the page, expanded case visits, observed the booked worker link `Playwright Iiuwfojg`, observed that row's `Cancel Visit` button, then completed the click without a click-level error.
+- Trace call evidence: click `call@1019` canonical to `<button aria-label="Cancel Visit" ...>`, completed at point `{x:433.12,y:516.25}`, and had no Playwright error. The next assertion `call@1023` waited 30000ms and timed out.
+- Failure screenshot and `error-context` page snapshot show the target case card visible but collapsed with `Show Case Visits`, and no dialog in the DOM.
+- Network evidence around the failed click shows no `PATCH /[home-health service]/api/v1/*/visits/*` cancellation request after the click. No backend cancellation request started, so this is not a backend cancellation failure.
+- Existing fix check: no open change with `flaky-test-fix` matched `fullLifecycle.spec.ts` or home-health E2E. Recent main commits touched this spec for other modal/row lookup flakes, but none directly address this cancel-dialog open path.
+
+**Proposed Fix:** Test harness fix in `playwright/e2e/homeHealth/fullLifecycle.spec.ts`.
+
+1. Add a helper for opening the cancel-visit dialog that mirrors the hardened patterns already used for verify/reject dialogs:
+   - Accept `page`, the case/row lookup inputs, the target visit id when available, and the expected dialog title.
+   - In a bounded `expect(async () => ...).toPass(...)`, ensure the case visits section is expanded, ensure the target row/action button is visible and enabled, click the button only when the expected dialog is not already open, and assert the cancel dialog title plus confirmation text are visible.
+   - Treat only these states as retryable: the case visits section is collapsed after lookup, the target row or action button detaches before the click, the target action is temporarily hidden/disabled, or the expected dialog title plus confirmation text are not visible after the open click while no cancellation `PATCH` has started.
+   - Treat these states as non-retryable: a cancel dialog opens for a different visit, the helper has already clicked the dialog confirmation button, a cancellation `PATCH /visits/{id}` request has started, or the selected visit id no longer matches the expected visit.
+   - Re-clicking `Cancel Visit` is idempotent only before confirmation because the row action opens a client-side dialog; no backend cancellation mutation starts until the dialog's confirmation button is clicked. If the matching dialog is already open, the helper should return/assert it instead of clicking again.
+   - There is no single deterministic completion signal to wait on for dialog open. The row click only changes React modal state inside `VisitActions`, and `CancelVisitDialog` is mounted under the collapsible `CaseVisits` subtree with `Collapse unmountOnExit`; the cancellation network request happens later on confirmation. A bounded web-first assertion on the specific dialog title and confirmation text is therefore the narrowest observable signal.
+   - Attach diagnostics on failure: whether `Show Case Visits` or `Hide Case Visits` is visible, visible dialog texts, visible action button labels, target worker/link count, target visit id if known, and visible visit IDs.
+2. Use the helper for Step e's booked-visit cancellation. Consider using it for Step d's unbooked cancellation too, since Step d has the same single-click/single-dialog-assertion anti-pattern.
+3. Keep the cancellation response wait scoped to the expected visit id for Step e, as it already is. For Step d, if practical, scope the `PATCH /visits` wait to the selected CNA visit id or add enough diagnostics to prove which visit was patched.
+4. Confirm whether modification reason is required under this test user. If required in local/staging auth context, fill `Reason for change` before clicking the dialog's `Cancel Visit`; if not required, leave unchanged. The field is conditional via `useIsModificationReasonRequired()`.
+
+**Observability to Reach 5/5:**
+
+- Add test diagnostics in the new helper for dialog-open failures: expanded/collapsed case visits state, target row count, target button count, dialog count/text, visible action labels, and a short case-card text snapshot.
+- Add reporter or helper-level network notes for whether a `GET /visits/{id}` or `PATCH /visits/{id}` was observed after the open click. This would distinguish “dialog never mounted” from “dialog mounted loading/error state and then unmounted.”
+- Product telemetry is not required for the immediate fix, but if this reappears outside tests, add a lightweight analytics/log event for cancel dialog open failures or visit-action click-to-dialog transitions in the Home Health case dashboard.
+
+**Sibling Candidates:**
+
+- Step d in `playwright/e2e/homeHealth/fullLifecycle.spec.ts` (`Cancel an unbooked CNA regular visit`) uses the same direct cancel click and broad `page.getByRole("dialog")` assertion.
+- Other `VisitActions` dialogs under the collapsible `CaseVisits` subtree are already partially hardened for edit/verify/reject/remove flows; review any remaining direct `page.getByRole("dialog")` open assertions in this spec before stopping.
+
+**Validation Plan:**
+
+- `npm run lint:fast -- playwright/e2e/homeHealth/fullLifecycle.spec.ts` if the script accepts file args; otherwise `npm run lint:fast`.
+- Run the targeted deployed-env Playwright test if available in this repo/environment, or the repo's documented Playwright command for `playwright/e2e/homeHealth/fullLifecycle.spec.ts` against staging.
+- At minimum, run the relevant TypeScript/lint check after editing: `npm run typecheck` if a full check is acceptable; otherwise the narrowest documented check the repo supports.
+
+**Open Questions:** None blocking. The implementation can proceed with the conservative test-harness fix.
+
+**Residual Risk:** The full lifecycle spec remains long and has many backend/test-data dependencies. This fix should stabilize the cancel-dialog open race, but separate flakes in auth, backend eventual consistency, case visit row lookup, or occurrence verification can still surface in the same test.
+
+## Acceptance Criteria
+
+- [ ] Step e opens the booked-visit cancel dialog through a bounded deterministic helper rather than a one-shot click/assertion.
+- [ ] The helper documents and enforces retryable vs non-retryable states before retrying an open click.
+- [ ] The helper only re-clicks `Cancel Visit` while no matching dialog is open, no confirmation click has happened, and no cancellation `PATCH` has started.
+- [ ] The helper uses the specific dialog title and confirmation text as the bounded web-first assertion because there is no earlier deterministic open-completion signal.
+- [ ] Step d either uses the same helper or is explicitly ruled out with evidence.
+- [ ] Failure diagnostics identify whether the case visits section is collapsed, the target row/action is missing, a dialog is loading/erroring, a different visit dialog is open, or no dialog exists.
+- [ ] The change body includes this implementation ticket ID.
+
+## Related Tickets
+
+Investigation source: [ticket redacted]
+
+Prior related tickets from the source task: [ticket redacted], [ticket redacted], [ticket redacted], [ticket redacted], [ticket redacted], [ticket redacted], [ticket redacted], [ticket redacted], [ticket redacted], [ticket redacted].

package/skills/flaky-critic/references/backtests/2026-07-16-b8-prior-attempts/inputs/case-15.md

@@ -0,0 +1,27 @@
+# Historical plan snapshot
+
+## Parent
+
+[ticket redacted] (flakiness program spec), Workstream C.
+
+## Groundcrew
+
+Repository: [admin frontend]
+Implementation workflow: use the `core:cb-work`/`cb-work` skill when available. If that skill is unavailable, follow this repo's [AGENTS.md/CLAUDE.md]([external evidence reference]) implementation workflow and run the documented verification.
+
+## What to build
+
+[ticket redacted] (available) made the backend return the Cognito username from admin user creation. Username-based Cognito operations are strongly consistent; the harness's current resolve-by-email path goes through the eventually consistent alias index, which is the root cause of the "Cognito user does not exist" storm family. Switch the Playwright harness to consume the returned username for all post-creation Cognito operations, then **delete the armor that only existed to absorb the lookup race**: the PreSignUp readiness probe, oversized readiness budgets (the ~90s window), and any throttle/retry logic whose sole purpose was surviving alias-index lag. Keep armor that guards genuinely different hazards (e.g. cross-process rate-limit throttling for call volume).
+
+First step: verify the username field is actually present in the staging environment's responses (the contract available; deployment matters). If absent, stop and report rather than building against a phantom.
+
+## Acceptance criteria
+
+- [ ] Harness user setup consumes the username from the creation response; no post-creation resolve-by-email remains on the setup path.
+- [ ] Obsolete probe/budget/retry code is deleted, each deletion named in the change body with the hazard it guarded now being impossible.
+- [ ] Auth-dependent spec pass rate is unchanged or better across repeated CI runs (state the runs checked).
+- [ ] A classification note distinguishes deleted armor (lookup-lag) from retained armor (rate-limit volume) so reviewers can check the boundary.
+
+## Notes
+
+Evidence: staging-seed-reliability deep dive; [ticket redacted] storm; consumer-side mitigations 152fe5ea8d7/32c31abeeef. Mobile mirror is a separate ticket that adopts this ticket's approach.

package/skills/flaky-critic/references/backtests/2026-07-16-b8-prior-attempts/inputs/case-16.md

@@ -0,0 +1,46 @@
+# Historical plan snapshot
+
+## Implementation repositories
+
+Primary investigation touched:
+
+- [mobile frontend]
+- ClipboardHealth/[worker gateway]
+- ClipboardHealth/[home-health service]
+
+The final fix may be either test setup hardening in `[mobile frontend]` or staging config/service alignment for the worker-app BFF home-health API target.
+
+## Related investigation
+
+- [ticket redacted]: `Worker accepts a visit invite > should accept a visit invite from the invites page`
+
+## Evidence
+
+Run [reference redacted]` failed after the invite and booking APIs succeeded:
+
+- `GET /[home-health service]/api/v1/visits/invites` returned a valid invite for worker `6a28d0f9eb110465a75bf4a4`, workplace `Playwright Facility MPR sch8dgis`, visit `2c9d0d13-01f8-4bda-bb7a-a1656e7ed4a2`.
+- `PATCH /[home-health service]/api/v1/visits/2c9d0d13-01f8-4bda-bb7a-a1656e7ed4a2` returned 200 with `status=FILLED` and `bookedWorkerId=6a28d0f9eb110465a75bf4a4`.
+- The subsequent worker-app request returned 200 but empty data twice:
+  `GET /worker-app/in-home/case/visit/?filter[booked]=true&filter[qualifications]=RN&include[]=case&include[]=workplace` => `{"data":[],"included":[]}`.
+
+The same CI run also had an unticketed sibling failure in `Worker visit lifecycle`: the open visits list returned empty repeatedly for `filter[booked]=false` before any booking.
+
+Datadog trace evidence:
+
+- Worker-app BFF served `GET /worker-app/in-home/case/visit` using worker id `6a28d0f9eb110465a75bf4a4` and commit `531ec7f1f42caa6ef348784194b9acf56be81a51`.
+- BFF called `cbh-[home-health service]-api.staging.service.local:3000/[home-health service]/api/v1/in-home-cases?filter[status]=OPEN&filter[booked]=true&filter[qualifications]=RN`.
+- [home-health service] ran the raw `Cases`/`Visits` query with `c.status = OPEN`, `v.booked_worker_id = workerId`, and `LOWER(v.worker_req) = RN`, then returned no rows before workplace hydration.
+- The public invite/booking requests in the browser are served as `hh-[home-health service]`, while the BFF read goes through internal `cbh-[home-health service]-api.staging.service.local`. They share the same home-health commit in the trace, but the read path did not see the newly created/updated visit.
+
+## Suggested fix
+
+- Confirm whether staging public Home Health API writes and worker-app BFF internal reads point at the same service/database and use the same route prefix. If not, align `HOME_HEALTH_API_URL` or the E2E setup base URL so read/write paths are consistent.
+- In `[mobile frontend]` Home Health Playwright setup, add a worker-visible readiness check after creating a visit/invite or after booking:
+  - poll `/worker-app/in-home/case/visit` as the worker until the seeded visit appears
+  - fail during setup with worker id, visit id, case id, workplace id, BFF response body, and trace URL/details if it never appears
+- Add regression coverage at the service boundary if feasible: create/seed an RN Home Health visit, then assert `/in-home-cases?filter[status]=OPEN&filter[booked]=false&filter[qualifications]=RN` returns it before booking and `filter[booked]=true` returns it after booking.
+
+## Verification
+
+- Re-run `playwright/e2e/homeHealth/workerAcceptsVisitInvite.spec.ts` against staging.
+- Re-run `playwright/e2e/homeHealth/workerVisitLifecycle.spec.ts` because the same empty-list symptom appeared in the same run.

package/skills/flaky-critic/references/backtests/2026-07-16-b8-prior-attempts/inputs/case-17.md

@@ -0,0 +1,101 @@
+# Historical plan snapshot
+
+## Groundcrew
+
+Repository: [admin frontend]
+Implementation workflow: use the `cb-work` skill. If that skill is unavailable, use the `core:go` skill.
+
+## Task
+
+Stabilize the shared Playwright worker setup path used by the license E2E test when a transient license-manager/auth response is surfaced as HTTP 401 during `worker_license_creation`.
+
+Do not change the license UI assertions for this failure. The failing attempt never reached the browser flow; it failed in `Agent.createAgentWithRetry()` while provisioning the worker.
+
+## Required change / Commit Notes
+
+- Include this implementation ticket ID in the change body.
+
+## Flake Details
+
+```json
+{
+  "repo": "[admin frontend]",
+  "test": "Licenses should be able to approve and reject license as admin and verify change license logs",
+  "file": "playwright/e2e/licenses.spec.ts:15",
+  "framework": "playwright",
+  "isNewFlaky": false,
+  "failures": [
+    {
+      "error": "Error: Failed to create agent after 2 attempts.\nDiagnostics: {\"historicalCorrelationKey\":\"[correlation redacted]\",\"correlationId\":\"[correlation redacted]\",\"githubRunId\":\"28306756185\",\"githubRunAttempt\":\"1\",\"githubJob\":\"e2e-tests-staging\",\"shardIndex\":\"2\",\"shardTotal\":\"4\",\"workerIndex\":\"1\",\"parallelIndex\":\"1\",\"pid\":57618,\"status\":401,\"responseShape\":{\"message\":\"string\",\"statusCode\":\"number\"},\"apiGatewayRequestId\":\"fpZk4jo8vHcEJOQ=\",\"sanitizedResponseMessage\":\"Unauthorized\",\"errorName\":\"AxiosError\",\"errorMessage\":\"Request failed with status code 401\",\"errorCode\":\"ERR_BAD_REQUEST\"}",
+      "stack": "Error: Failed to create agent after 2 attempts.\nDiagnostics: {\"historicalCorrelationKey\":\"[correlation redacted]\",\"correlationId\":\"[correlation redacted]\",\"githubRunId\":\"28306756185\",\"githubRunAttempt\":\"1\",\"githubJob\":\"e2e-tests-staging\",\"shardIndex\":\"2\",\"shardTotal\":\"4\",\"workerIndex\":\"1\",\"parallelIndex\":\"1\",\"pid\":57618,\"status\":401,\"responseShape\":{\"message\":\"string\",\"statusCode\":\"number\"},\"apiGatewayRequestId\":\"fpZk4jo8vHcEJOQ=\",\"sanitizedResponseMessage\":\"Unauthorized\",\"errorName\":\"AxiosError\",\"errorMessage\":\"Request failed with status code 401\",\"errorCode\":\"ERR_BAD_REQUEST\"}\n    at /opt/actions-runner/_work/[admin frontend]/[admin frontend]/playwright/api/createAgent.ts:160:13\n    at /opt/actions-runner/_work/[admin frontend]/[admin frontend]/playwright/e2e/licenses.spec.ts:31:21\n    at WorkerMain._runTest (/opt/actions-runner/_work/[admin frontend]/[admin frontend]/node_modules/dd-trace/packages/datadog-instrumentations/src/playwright.js:1692:5)",
+      "branch": "main",
+      "pipeline": "[external evidence reference]",
+      "commit": "94ea9f242c3211186e9613f743b137043eae003d",
+      "durationMs": 21949,
+      "shard": "2/4",
+      "timestamp": "2026-06-28T00:49:44Z"
+    }
+  ]
+}
+```
+
+## Plan Output
+
+**Test ID:** `60d14b1d432fd603aa6b-59e4c2e48b8e7a355f7f` from the downloaded `playwright-llm-report` for run [reference redacted]`.
+
+**Confidence:** 4/5. The CI artifacts directly show the setup step, status code, request ID, and retry behavior. The only inferred link is the exact backend/auth reason why license-manager returned 401 for a token that had just worked elsewhere.
+
+**Failure surface:** Test setup/auth/data. The test failed before app navigation or any license UI action.
+
+**Current main status:** The failing code path still exists on current `origin/main`. `playwright/api/createAgent.ts` still classifies this 401 as `non_retryable_agent_setup_failure`; no open `flaky-test-fix` change was found for `licenses.spec.ts`/`createAgent.ts`. Recent setup/auth fixes (`[ticket redacted]`, `[ticket redacted]`, `[ticket redacted]`) improve nearby setup paths but do not cover this exact license-manager 401 classification.
+
+**Symptom:** `playwright/e2e/licenses.spec.ts:31` failed in `agentInstance.createAgentWithRetry()` with `Failed to create agent after 2 attempts`. The final error was an Axios 401 from `POST [external evidence reference] with API Gateway request id `fpZk4jo8vHcEJOQ=`.
+
+**Root cause:** `Agent.createAgentWithRetry()` retries whole worker setup for known transient setup failures, but its local retry classifier treats a 401 from setup provisioning as non-retryable. In this run, the admin token was freshly minted and validated, HCP creation succeeded, worker token acquisition succeeded, and another admin-token setup call (`stripe_setup`) succeeded. The 401 was isolated to `worker_license_creation` on attempt 2 and the Playwright retry passed with the same test scenario, so the frontend setup helper is bailing on a transient backend/auth failure instead of letting the existing fresh-identity retry boundary recover.
+
+**Evidence:**
+
+- LLM report summary: 53 tests total, 49 passed, 2 flaky, 0 failed. The license test was flaky: attempt 1 failed after 21.9s, attempt 2 passed after 19.4s.
+- Attempt 1 steps show only setup/fixtures and teardown; browser network summary has 0 retained requests, confirming no UI flow ran before failure.
+- Job log for correlation `[correlation redacted]`:
+  - `admin_token_acquisition` success at 00:49:04.
+  - `admin_token_validation` success at 00:49:04.
+  - attempt 1 `hcp_creation` failed with retryable 504 (`apiGatewayRequestId=fpZiYh88vHcEJeg=`), then the helper retried.
+  - attempt 2 `hcp_creation` success, `hcp_auth_token_acquisition` success, and `contract_signing` success.
+  - attempt 2 `worker_license_creation` failed after 5040ms with status 401, response `{"message":"Unauthorized","statusCode":401}`, `apiGatewayRequestId=fpZk4jo8vHcEJOQ=`.
+- Current code evidence:
+  - `playwright/api/createAgent.ts` runs `worker_license_creation` and `contract_signing` inside `Promise.all` and then classifies 401 as non-retryable.
+  - `playwright/api/facility.ts` uses the same `worker_license_creation` endpoint but its outer helper delegates to shared `isRetryableE2eSetupError`, which treats non-403 setup errors as retryable. `createAgent.ts` is stricter and inconsistent for this setup-auth failure.
+
+**Proposed fix:**
+
+1. Preserve the failing setup step name in the thrown diagnostics, not only in console logs. A pragmatic implementation is to add a small `E2eSetupStepError` wrapper in `playwright/helpers/e2eSetupDiagnostics.ts` (or equivalent structured metadata) that carries `stepName`, `stepStatus`, HTTP diagnostics, and the original cause from `runE2eSetupStep()`.
+2. Update `playwright/api/createAgent.ts` retry classification so a 401 from `worker_license_creation` is retryable at the existing `createAgentWithRetry()` whole-agent boundary. Keep 403 non-retryable. Avoid making every 401 globally retryable unless the preserved diagnostics prove it is from an E2E setup provisioning step with a valid prior admin token.
+3. Prefer retrying the whole `createAgentWithRetry()` attempt with a fresh generated worker identity over retrying the individual create-license POST. That keeps the retry idempotent if a downstream service ever writes state before returning an auth-shaped error.
+4. Add focused tests around the retry classifier / diagnostics helper if the repo has a nearby test harness pattern. At minimum, cover: 401 `worker_license_creation` is retryable, 403 remains non-retryable, missing admin token remains non-retryable, and existing 408/429/5xx behavior is unchanged.
+5. In `[admin frontend]`, include `stepName`, request method, sanitized URL/service name, HTTP status, `apiGatewayRequestId`, and `traceparent` in the final `Failed to create agent after ...` diagnostics, not just in preceding console logs.
+
+**Sibling candidates:**
+
+- `playwright/api/facility.ts`: same `worker_license_creation` endpoint. It appears already more permissive via `isRetryableE2eSetupError`, but verify it still behaves correctly after any diagnostics wrapper change.
+- `playwright/api/agent.ts`: legacy worker setup helper creates licenses and retries setup broadly; audit for diagnostic consistency, but avoid unrelated refactors.
+- [ticket redacted] is a duplicate cluster member for the same `createAgent`/token-guard family and should be covered by the same implementation.
+
+**Validation plan:**
+
+- `npm run lint:fast -- playwright/api/createAgent.ts playwright/helpers/e2eSetupDiagnostics.ts playwright/api/facility.ts playwright/api/agent.ts`
+- `npm run typecheck`
+- If staging credentials are available, run the focused Playwright spec against staging: `npx playwright test playwright/e2e/licenses.spec.ts --grep "should be able to approve and reject license as admin and verify change license logs" --project "Desktop Chrome"`.
+- If the implementation only changes classifier/helper logic and a unit test is added, run the smallest relevant Jest/Vitest command for that test file as well.
+
+**Open questions:** None blocking. The implementation should be conservative and retry only the setup-auth case supported by the artifacts.
+
+**Residual risk:** A real persistent license-manager authorization regression would still fail after the retry budget. Without backend auth reason telemetry, future 401s may still require API Gateway/Datadog correlation to determine whether the backend or test setup should own the fix.
+
+## Acceptance Criteria
+
+- [ ] `Agent.createAgentWithRetry()` retries the observed transient `worker_license_creation` 401 at the whole-agent setup boundary.
+- [ ] Persistent permission failures such as 403 still bail immediately.
+- [ ] Final setup failure diagnostics include the failing setup step name.
+- [ ] [ticket redacted] and duplicate cluster member [ticket redacted] are referenced in the change body or implementation notes.
+- [ ] Validation commands from the plan are run, or blockers are documented in the change.

package/skills/flaky-critic/references/backtests/2026-07-16-b8-prior-attempts/inputs/case-18.md

@@ -0,0 +1,88 @@
+# Historical plan snapshot
+
+## Implementation task
+
+Implement the flaky-test-debugger plan from [ticket redacted] for this Playwright E2E flake.
+
+## Required change / commit instructions
+
+- Include this implementation ticket ID in the change body.
+- Implement this plan using the `cb-work` skill or, if unavailable, the `core:go` skill.
+
+## Flake details
+
+```json
+{
+  "repo": "[admin frontend]",
+  "test": "Schedule daily view (redesign) — desktop workplace user posts a single shift from the Add Shift side panel (SCHEDULE_VIEW_POST_LAUNCH on)",
+  "file": "playwright/e2e/dailyView.spec.ts:420",
+  "framework": "playwright",
+  "isNewFlaky": false,
+  "failures": [
+    {
+      "error": "TimeoutError: locator.click: Timeout 30000ms exceeded. Call log: waiting for getByRole('combobox', { name: 'Role' }); locator canonical; element was not stable; element was detached from the DOM, retrying",
+      "branch": "main",
+      "pipeline": "[external evidence reference]",
+      "commit": "b568dcdc304406da26be6128e183cae35390fa64",
+      "durationMs": 57388,
+      "shard": "2/4",
+      "timestamp": "2026-07-02T19:06:56Z"
+    }
+  ]
+}
+```
+
+## Flaky-test-debugger plan
+
+**Test ID:** `b03b0307e93e39946710-71ce947e36caa7d4ae78`
+
+**Confidence:** 4/5. The Playwright LLM report and screenshot show the panel/Role field is transient after opening, but the report does not retain a close-state RUM event or DOM snapshot at the exact detach moment, so the precise close/remount trigger is inferred.
+
+**Failure surface:** User action no-op / test harness synchronization.
+
+**Current main status:** Still relevant. The failing helper still performs `await page.getByRole("combobox", { name: "Role" }).click()` immediately after visible checks in `playwright/e2e/dailyView.spec.ts`. Recent main contains prior stabilizations from [ticket redacted]/[ticket redacted] and a later roster Add Shift panel change (`aaa7a899d7e`), but no current open `flaky-test-fix` change and no later change that waits for the Role combobox to be actionable after the side-panel transition/remount window.
+
+**Symptom:** Failed attempt opened the Add Shift panel, saw the `Add Shift` title and visible Role combobox, then timed out for 30s clicking the Role combobox. Playwright reported the element was not stable and was detached from the DOM. The failure screenshot shows the schedule page without the Add Shift side panel.
+
+**Root cause:** The test starts Role selection as soon as the panel title and Role field are visible. Visibility is too early for this surface: the desktop Add Shift side panel is animated and its form can still re-render/reset while route/day data and panel-related queries settle. The first destructive operation is not the role click, so the test can safely make the panel-open/role-selection phase idempotent before posting, instead of relying on one immediate Role click.
+
+**Evidence:**
+
+- LLM report summary: 52 total tests, 49 passed, 1 flaky, 2 skipped; this test failed attempt 1 and passed attempt 2.
+- Failed attempt timeline: `openAddShiftPanel` clicked `Add` at ~26.25s, `Add Shift` was visible at ~26.37s, Role was visible at ~26.386s, and the Role click started at ~26.391s then timed out after 30s.
+- Error artifact: `locator.click` on `getByRole('combobox', { name: 'Role' })` hit unstable element and DOM detachment.
+- Screenshot artifact: final failure image shows the daily schedule with no Add Shift side panel open.
+- Network/lifecycle: no `POST /v3/shifts` was emitted. The failed attempt had 333 observed network instances with no group/instance-cap drops; background schedule/workplace/default-time requests were still active around the panel-open window. `GET /api/shiftTimes/get/...qualification=CNA` returned 200.
+- RUM artifact: retained custom action `HCF Admin Web App (V2): Schedule add shift panel state changed` with `isOpen: true`; no retained matching close event.
+- Code path: `postSingleShiftViaAddPanel` in `playwright/e2e/dailyView.spec.ts` waits for title/Role visibility, then immediately clicks Role. `openAddShiftPanel` retries the safe Add click but does not wait for the side-panel transition or Role actionability.
+
+**Proposed fix:** Test harness fix in `playwright/e2e/dailyView.spec.ts`.
+
+1. Add a small helper for the safe pre-post phase, for example `selectRoleInAddShiftPanel({ page, roleName: "CNA" })`.
+2. In that helper, use `expect(...).toPass()` around only idempotent actions:
+   - ensure the Add Shift panel is open; if it disappeared before any post, call `openAddShiftPanel` again;
+   - locate the Role combobox;
+   - use `roleCombobox.click({ trial: true, timeout: 5_000 })` to wait for Playwright actionability/stability without changing state;
+   - if the combobox is not already the desired role, click it and choose the option;
+   - assert the combobox contains the chosen role before leaving the retry block.
+3. Keep the destructive post flow outside any retry: wait for date/default times/Post enabled, create the `waitForResponse` promise, click Post once, handle optional rush-fee dialog, then assert the panel closes.
+4. Reuse the same helper for the mobile bottom-sheet test because it calls the same `postSingleShiftViaAddPanel` path. Do not add `waitForTimeout`; use actionability/trial click and assertions as deterministic signals.
+
+**Observability to reach 5/5:**
+
+- Add a close-reason field to the existing Add Shift panel RUM state event (`closeButton`, `postSuccess`, `inviteSuccess`, `hostOverlayReplacement`, `unmount`, etc.) so the next occurrence proves whether the panel ended, remounted, or only moved during transition.
+- Extend the Playwright reporter or test helper to attach a targeted screenshot/DOM snapshot when a locator actionability failure mentions `detached from the DOM`, especially for panel/dialog surfaces.
+- Consider adding a test-only `data-testid` or stable panel root role/name for Add Shift panel scoping if the helper still has to use global locators.
+
+**Sibling candidates:** The mobile Add Shift bottom-sheet E2E in the same file shares `postSingleShiftViaAddPanel`; update it automatically through the shared helper. No other E2E currently performs this exact Add Shift panel Role selection.
+
+**Validation plan:**
+
+- `npx oxfmt --check playwright/e2e/dailyView.spec.ts`
+- `npm run lint -- playwright/e2e/dailyView.spec.ts`
+- If deployed E2E env credentials are available: `npm run playwright:run -- --grep "posts a single shift from the Add Shift"`
+- If local Playwright cannot start because `TEST_HELPER_API_KEY` / deployed API env is missing, state that in the change and rely on CI for the E2E run.
+
+**Open questions:** None.
+
+**Residual risk:** If the panel is genuinely closing for real users without any outside click, the test-harness fix may mask a product bug. The close-reason telemetry above would make that distinguishable in the next occurrence.

package/skills/flaky-critic/references/backtests/2026-07-16-b8-prior-attempts/inputs/case-19.md

@@ -0,0 +1,98 @@
+# Historical plan snapshot
+
+## Groundcrew
+
+Repository: [admin frontend]
+Implementation workflow: use the `cb-work` skill when available. If that skill is unavailable, use the `core:go`/`go` skill. Include this implementation ticket ID in the change body.
+
+```bash
+
+```
+
+## Task
+
+Fix the shared workplace magic-link Playwright sign-in harness so successful but slow auth completion does not cause `retryUntilPassOrTimeout` to retry on the same page and then wait for the login Email field after the session is already authenticated.
+
+## Acceptance Criteria
+
+- [ ] `signInAsWorkplaceUser` no longer times out filling `getByLabel("Email")` when a previous magic-link attempt completes auth after the redirect wait budget.
+- [ ] The helper still retries promptly when the magic link is expired/already used or the auth error screen is visible.
+- [ ] The fix is applied to sibling workplace magic-link/OTP retry paths that navigate to `/login` and fill Email inside `retryUntilPassOrTimeout`.
+- [ ] The change body includes this implementation ticket ID.
+
+## Flake Details
+
+```json
+{
+  "repo": "[admin frontend]",
+  "test": "Reporting (redesign) — desktop page chrome and all four KPI cards render on landing",
+  "file": "playwright/e2e/reporting.spec.ts:15",
+  "framework": "playwright",
+  "isNewFlaky": true,
+  "failures": [
+    {
+      "error": "Error: locator.fill: Timeout 30000ms exceeded.\nCall log:\n  - waiting for getByLabel('Email')\n\n\nCall Log:\n- Timeout 90000ms exceeded while waiting on the predicate",
+      "stack": "Error: locator.fill: Timeout 30000ms exceeded.\nCall log:\n  - waiting for getByLabel('Email')\n\n\nCall Log:\n- Timeout 90000ms exceeded while waiting on the predicate\n    at retryUntilPassOrTimeout (/opt/actions-runner/_work/[admin frontend]/[admin frontend]/playwright/helpers/common.ts:6:26)\n    at signInAsWorkplaceUser (/opt/actions-runner/_work/[admin frontend]/[admin frontend]/playwright/helpers/scheduleSeed.ts:97:32)\n    at /opt/actions-runner/_work/[admin frontend]/[admin frontend]/playwright/e2e/reporting.spec.ts:20:7\n    at WorkerMain._runTest (/opt/actions-runner/_work/[admin frontend]/[admin frontend]/node_modules/dd-trace/packages/datadog-instrumentations/src/playwright.js:1692:5)",
+      "branch": "main",
+      "pipeline": "[external evidence reference]",
+      "commit": "4804367a1bf5dc5f55e097868ac06decc8fc175c",
+      "durationMs": [attempt reference redacted],
+      "shard": "3/4",
+      "timestamp": "2026-07-01T13:51:35Z"
+    }
+  ]
+}
+```
+
+## Plan Output
+
+**Test ID:** c5fb8ef2d524
+
+**Confidence:** 5/5. The LLM report timeline, network events, and screenshot directly show the helper timing out a successful auth attempt, retrying, and then ending on the authenticated Schedule page while waiting for the login Email field.
+
+**Failure surface:** test setup/auth/data. The reporting page was never reached in the failed attempt.
+
+**Current main status:** the failing commit `4804367a1bf5dc5f55e097868ac06decc8fc175c` is still represented by current main/current branch. The shared `signInAsWorkplaceUser` path and `waitForWorkplaceMagicLinkSignIn` helper still exist. There is no open `flaky-test-fix` change for `reporting.spec.ts`, `scheduleSeed.ts`, or `workplaceSignIn.ts`. Prior commit `766c7575fbd` added the 15s bounded redirect wait, but this flake occurred after that change.
+
+**Symptom:** `playwright/e2e/reporting.spec.ts:15` failed in `signInAsWorkplaceUser` before `gotoAppRoute({ url: "/facility/invoice/reporting" })`. The outer `retryUntilPassOrTimeout` expired after the retry callback hit `locator.fill` timeout waiting for `getByLabel('Email')`.
+
+**Root cause:** `waitForWorkplaceMagicLinkSignIn` treats a workplace magic-link redirect that takes more than 15s as a failed attempt. In this run, the first magic-link Continue click was valid and Cognito returned success, but authenticated app bootstrap did not finish until roughly 20.7s after the click. The helper timed out at 15s, the retry loop navigated back to `/login` on the same page, and the late success from the first attempt established the session and redirected to `/schedule`. The retry then waited for an Email field on an authenticated Schedule page. Later retries repeated the same pattern because the page was already signed in.
+
+**Evidence:**
+
+- LLM report summary: 52 total tests, 1 flaky, 0 failed after retry. The flaky entry is exactly `Reporting (redesign) — desktop > page chrome and all four KPI cards render on landing`.
+- Failed attempt timeline: first `/login` navigation at 12.106s, Email fill at 13.324s, magic-link page navigation at 17.336s, Continue click at 18.181s, then `Expect "toHaveURL"` failed after 15.003s.
+- The same failed attempt retried at 34.231s with `Navigate to "/login"`, then `Fill "playwright-fmuepswefho@playwright-hcf.com" getByLabel('Email')` started at 38.854s and timed out after 30.012s.
+- Authenticated workplace traffic began immediately after that retry fill started: `/api/facilityUser/findByEmail` returned 200 at 38.859s and `/api/user/get/...` returned 200 at 38.926s.
+- The failure screenshot shows the authenticated Schedule page for `test facility mvsij0epxt`, not the login page.
+- No reporting API/navigation/assertion ran in the failed attempt; the retry passed and reached `/facility/invoice/reporting` normally.
+
+**Proposed fix:** test harness fix in `playwright/helpers/workplaceSignIn.ts` and the magic-link callers.
+
+1. Replace the fixed 15s schedule URL wait with an outcome-based wait: allow a successful `/schedule` redirect for up to the same order as other Cognito login flows (the password helper uses 60s), while racing/polling for visible expired/already-used/auth-error text so bad links still fail promptly and request a fresh link.
+2. Make retry attempts idempotent when a previous auth attempt finishes late. Before filling Email after `page.goto("/login")`, wait for either the login Email field or an authenticated schedule URL. If schedule is reached, return success instead of filling the form.
+3. Apply the same already-authenticated/form-ready guard to sibling retry helpers that call `/login` and fill Email inside `retryUntilPassOrTimeout`, notably the local `submitMagicLinkForm` in `playwright/e2e/auth.spec.ts` and the email OTP request helper in `playwright/e2e/cognito/cognitoEmailOtpAuth.spec.ts` if it shares the same retry hazard.
+4. Keep the diagnostics from `workplaceSignIn.ts`: sanitized current URL, visible auth error text, elapsed time since link request, schedule reached boolean, and magic-link marker.
+
+**Observability to reach 5/5:** N/A -- confidence is 5/5 for the test-harness root cause. Optional improvement: add a structured stdout event when workplace magic-link sign-in times out or resolves as already authenticated after a retry, including elapsed time and sanitized URL, so future auth setup flakes can be grouped without opening the screenshot.
+
+**Sibling candidates:** all specs using `signInAsWorkplaceUser` from `playwright/helpers/scheduleSeed.ts`; direct magic-link auth retries in `playwright/e2e/auth.spec.ts`; OTP retry flow in `playwright/e2e/cognito/cognitoEmailOtpAuth.spec.ts`.
+
+**Validation plan:**
+
+- `npm run lint:fast -- playwright/helpers/workplaceSignIn.ts playwright/helpers/scheduleSeed.ts playwright/e2e/auth.spec.ts playwright/e2e/cognito/cognitoEmailOtpAuth.spec.ts`
+- `npx oxfmt --check playwright/helpers/workplaceSignIn.ts playwright/helpers/scheduleSeed.ts playwright/e2e/auth.spec.ts playwright/e2e/cognito/cognitoEmailOtpAuth.spec.ts`
+- `npx cspell --no-must-find-files playwright/helpers/workplaceSignIn.ts playwright/helpers/scheduleSeed.ts playwright/e2e/auth.spec.ts playwright/e2e/cognito/cognitoEmailOtpAuth.spec.ts`
+- `npm run typecheck`
+- With staging E2E credentials available: `npx playwright test playwright/e2e/reporting.spec.ts --project="Desktop Chrome" --grep "page chrome and all four KPI cards render on landing"`
+- Prefer also running the desktop auth magic-link test if credentials are available: `npx playwright test playwright/e2e/auth.spec.ts --project="Desktop Chrome" --grep "clicking the emailed verification link signs the user in"`
+
+**Open questions:** none blocking. Use the smallest harness change that preserves fast retry for explicit auth errors while preventing a slow success from poisoning same-page retries.
+
+**Residual risk:** this fixes the retry/idempotency race. It does not explain why successful Cognito workplace magic-link completion sometimes takes more than 15s in staging CI; if that latency grows past the new success window, the helper should fail with diagnostics rather than a misleading Email locator timeout.
+
+## Investigation Notes
+
+- The repository did not contain `scripts/fetch-llm-report.sh`, so the LLM report was downloaded with `gh run download 28522523810 --name playwright-llm-report`.
+- Artifact path used for investigation: `/tmp/playwright-llm-report-28522523810/llm-report.json`.
+- Failure screenshot decoded locally to `/tmp/reporting-flake-attempt0.png`; it showed the authenticated Schedule page.

package/skills/flaky-critic/references/backtests/2026-07-16-b8-prior-attempts/inputs/case-20.md

@@ -0,0 +1,91 @@
+# Historical plan snapshot
+
+## Implementation instructions
+
+Implement the plan below in `[admin frontend]` using the `cb-work` skill or, if unavailable, the `core:go` skill.
+
+Include this implementation ticket ID in the change body.
+
+```bash
+
+```
+
+## Flake details
+
+```json
+{
+  "repo": "[admin frontend]",
+  "test": "Schedule rate negotiations (redesign) — desktop workplace user accepts one rate proposal and ends another from the shift details drawer",
+  "file": "playwright/e2e/rateNegotiation.spec.ts:40",
+  "framework": "playwright",
+  "isNewFlaky": false,
+  "failures": [
+    {
+      "error": "Error: Failed to create agent after 1 attempt.\nDiagnostics: {\"historicalCorrelationKey\":\"[correlation redacted]\",\"correlationId\":\"[correlation redacted]\",\"githubRunId\":\"28798037729\",\"githubRunAttempt\":\"1\",\"githubJob\":\"e2e-tests-staging\",\"shardIndex\":\"3\",\"shardTotal\":\"4\",\"workerIndex\":\"1\",\"parallelIndex\":\"1\",\"pid\":[attempt reference redacted],\"stepName\":\"hcp_creation\",\"stepStatus\":\"failure\",\"durationMs\":166,\"requestMethod\":\"POST\",\"requestUrl\":\"[external evidence reference]",\"requestService\":\"api\",\"status\":400,\"responseShape\":{\"error\":\"string\",\"message\":\"string\",\"statusCode\":\"number\"},\"traceparent\":\"00-00000000000000001732fff1044f982e-197732b055021fe4-01\",\"apiGatewayRequestId\":\"AFm5ki1zvHcESJw=\",\"sanitizedResponseMessage\":\"Error while creating user\\nBad Request\",\"errorName\":\"AxiosError\",\"errorMessage\":\"Request failed with status code 400\",\"errorCode\":\"ERR_BAD_REQUEST\"}",
+      "stack": "Error: Failed to create agent after 1 attempt.\nDiagnostics: {\"historicalCorrelationKey\":\"[correlation redacted]\",\"correlationId\":\"[correlation redacted]\",\"githubRunId\":\"28798037729\",\"githubRunAttempt\":\"1\",\"githubJob\":\"e2e-tests-staging\",\"shardIndex\":\"3\",\"shardTotal\":\"4\",\"workerIndex\":\"1\",\"parallelIndex\":\"1\",\"pid\":[attempt reference redacted],\"stepName\":\"hcp_creation\",\"stepStatus\":\"failure\",\"durationMs\":166,\"requestMethod\":\"POST\",\"requestUrl\":\"[external evidence reference]",\"requestService\":\"api\",\"status\":400,\"responseShape\":{\"error\":\"string\",\"message\":\"string\",\"statusCode\":\"number\"},\"traceparent\":\"00-00000000000000001732fff1044f982e-197732b055021fe4-01\",\"apiGatewayRequestId\":\"AFm5ki1zvHcESJw=\",\"sanitizedResponseMessage\":\"Error while creating user\\nBad Request\",\"errorName\":\"AxiosError\",\"errorMessage\":\"Request failed with status code 400\",\"errorCode\":\"ERR_BAD_REQUEST\"}\n    at /opt/actions-runner/_work/[admin frontend]/[admin frontend]/playwright/api/createAgent.ts:152:13\n    at /opt/actions-runner/_work/[admin frontend]/[admin frontend]/playwright/e2e/rateNegotiation.spec.ts:84:24\n    at WorkerMain._runTest (/opt/actions-runner/_work/[admin frontend]/[admin frontend]/node_modules/dd-trace/packages/datadog-instrumentations/src/playwright.js:1692:5)",
+      "branch": "main",
+      "pipeline": "[external evidence reference]",
+      "commit": "1983e526d20ac66dc2c52102caeef8d6bba92cd2",
+      "durationMs": 8074,
+      "shard": "3/4",
+      "timestamp": "2026-07-06T14:14:13Z"
+    }
+  ]
+}
+```
+
+## Plan output
+
+**Test ID:** `2f8cacf23bff`
+
+**Confidence:** 4/5. The failing artifact shows `createAgentWithRetry()` bailed after one `hcp_creation` 400, and Datadog logs for the same trace show a Cognito `ConflictException` during backend user creation. The only inferred link is which identifier collided, because the API response intentionally returned a generic message.
+
+**Failure surface:** test setup/auth/data.
+
+**Current main status:** The failing code path still exists on current `origin/main`. Prior fixes are already available: `[reference redacted]` / `dcd544114bc` classified visible `Phone already in use` 400s, and `[reference redacted]` / `6d3f71348c7` scoped that retry to worker creation. This occurrence is not covered because the response body was only `Error while creating user\nBad Request`.
+
+**Symptom:** In `playwright/e2e/rateNegotiation.spec.ts:84`, the first worker setup call fails before any browser user-flow assertion. `Agent.createAgentWithRetry()` reports `Failed to create agent after 1 attempt` even though its default max is 5, because `getAgentSetupRetryClassification()` returned `non_retryable_agent_setup_failure` and called `bail()`.
+
+**Root cause:** The E2E worker setup retry classifier only treats `hcp_creation` HTTP 400 responses as retryable when the sanitized response includes `Phone already in use`. For this run, `POST /api/user/create` returned a generic 400. Datadog APM/logs for trace id `1671679822332401710` show the backend span `cbh-user POST /api/user/create`, HTTP 400, `BadRequestException: Error while creating user`, and a log error from `/app/src/modules/auth/services/cognito.service.ts` with `ConflictException` in `ensureIdentifierAvailableForEmployeeUpdate`. That is an identity collision/setup-data failure, but the frontend harness cannot see the precise backend conflict reason and therefore does not retry with the next generated worker identity.
+
+**Evidence:**
+
+- LLM report `/tmp/playwright-llm-report-28798037729/llm-report.json`: 52 tests, 1 flaky, failed attempt error `Failed to create agent after 1 attempt`, passed retry duration 28.475s.
+- Failure diagnostics: `stepName=hcp_creation`, `POST [external evidence reference] status 400, `traceparent=00-00000000000000001732fff1044f982e-197732b055021fe4-01`, `apiGatewayRequestId=AFm5ki1zvHcESJw=`, sanitized response `Error while creating user\nBad Request`.
+- Datadog APM: W3C trace low-64 id `1671679822332401710`; span `6264525095033341259`, service `cbh-user`, resource `POST /api/user/create`, HTTP status code 400, duration 113.5ms, `exception.message=Error while creating user`, `exception.type=BadRequestException`.
+- Datadog logs for that trace: `Error while creating user`; error kind `ConflictException`; stack includes `CognitoService.throwRealWorkerConflict`, `classifyOrphanCandidates`, `ensureIdentifierAvailableForEmployeeUpdate`, `UserManipulationService.handleNewAgentInIdentityProviders`, and `UserManipulationService.createAgent`.
+- Code path: `playwright/api/createAgent.ts` generates a fresh phone/email inside the retry callback, but `playwright/helpers/createAgentRetryClassification.ts` bails on this generic 400 shape.
+- Existing regression coverage is in `scripts/createAgentRetryClassification.test.ts` and currently only covers the visible `Phone already in use` 400 case.
+
+**Proposed fix:** Test harness fix.
+
+- In `playwright/helpers/createAgentRetryClassification.ts`, extend the worker-creation 400 classifier narrowly. Treat this shape as retryable only when all of these hold: status is 400, diagnostics `stepName` is `hcp_creation`, `requestUrl` or `requestService` identifies `/api/user/create`, and `sanitizedResponseMessage` matches the generic user-creation bad request/conflict shape (`Error while creating user\nBad Request`, and optionally `Conflict Exception` if present).
+- Return the existing `retryable_phone_collision` classification to minimize churn, or rename it to `retryable_worker_identity_collision` if the implementer wants the log label to match email/phone/Cognito identifier conflicts more accurately. If renamed, update the union type and tests in the same change.
+- Do not classify all `hcp_creation` 400s as retryable; request-shape/schema failures should still fail fast.
+- Add/extend Jest coverage in `scripts/createAgentRetryClassification.test.ts` for the generic `/api/user/create` 400 response and for a negative 400 case from an unrelated setup step or endpoint.
+
+**Observability to reach 5/5:** Add a safe machine-readable reason to setup diagnostics or backend response/log tags, such as `worker_identity_conflict`, without exposing phone/email. Best options: have `cbh-user` tag/log the conflict classification on the `POST /api/user/create` span, and have `runE2eSetupStep` preserve a non-PII error code when the response includes one. That would remove the need to infer from backend stack names.
+
+**Sibling candidates:**
+
+- `playwright/api/createAgent.ts` is the failing class helper and should get the benefit through the shared classifier.
+- `playwright/api/facility.ts` already documents the same backend generic `Error while creating user / Conflict Exception` collision shape around its HCP setup path; confirm whether its retry wrapper already converges or needs the same classification behavior.
+- `playwright/api/agent.ts` has an older `createAgentWithRetry()` helper that retries all failures without this classifier; no immediate change required, but it is related setup surface.
+
+**Validation plan:**
+
+```bash
+npx jest scripts/createAgentRetryClassification.test.ts --runInBand
+npm run lint:fast -- playwright/helpers/createAgentRetryClassification.ts scripts/createAgentRetryClassification.test.ts
+npm run typecheck
+```
+
+If the implementer can run staging E2E safely, also run the target spec shard or the single spec:
+
+```bash
+npx playwright test playwright/e2e/rateNegotiation.spec.ts --project=chromium
+```
+
+**Open questions:** None blocking. The implementation should use the current sanitized response shape and avoid relying on PII from Datadog logs.
+
+**Residual risk:** If staging has enough retained test users that the 9M generated phone space produces frequent collisions, retries will reduce flake rate but not eliminate it forever. A later stronger fix could expand the valid generated phone space or add a test-only reserved identity namespace in the backend.