@clipboard-health/ai-rules 2.38.2 โ†’ 2.39.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@clipboard-health/ai-rules",
3
- "version": "2.38.2",
3
+ "version": "2.39.0",
4
4
  "description": "Pre-built AI agent rules for consistent coding standards.",
5
5
  "keywords": [
6
6
  "ai",
@@ -19,9 +19,9 @@ Resolve bundled "./scripts" paths relative to SKILL.md.
19
19
 
20
20
  The skill uses two sentinels with visible footer lines.
21
21
 
22
- **Addressed sentinel**: `<sub>๐Ÿค– <code>cb-babysit:addressed v1 core@3.17.1</code></sub>`. Appended on its own line at the end of every reply the skill posts so re-runs know which threads and review-body comments are already handled. Dedupe also recognizes the legacy `babysit-pr:addressed v1` prefix from before this skill was renamed.
22
+ **Addressed sentinel**: `<sub>๐Ÿค– <code>cb-babysit:addressed v1 core@3.18.0</code></sub>`. Appended on its own line at the end of every reply the skill posts so re-runs know which threads and review-body comments are already handled. Dedupe also recognizes the legacy `babysit-pr:addressed v1` prefix from before this skill was renamed.
23
23
 
24
- **Follow-up sentinel**: `<sub>๐Ÿค– <code>cb-babysit:followup v1 core@3.17.1</code></sub>`. Attached to replies that defer an out-of-scope comment as a tracked follow-up. The sentinel is additive: the post-reply scripts still append the `addressed` sentinel at the end.
24
+ **Follow-up sentinel**: `<sub>๐Ÿค– <code>cb-babysit:followup v1 core@3.18.0</code></sub>`. Attached to replies that defer an out-of-scope comment as a tracked follow-up. The sentinel is additive: the post-reply scripts still append the `addressed` sentinel at the end.
25
25
 
26
26
  **Sentinel recency rules.** The script emits a per-thread `activityState` with three values. Step 6a owns the handling rules for each state.
27
27
 
@@ -247,7 +247,7 @@ Body templates (the script appends the `addressed` sentinel if missing):
247
247
  - **Agree**: `Addressed in <commit-url>. <one-line what-changed>.`
248
248
  - **Disagree**: `Leaving current behavior. <reasoning>.`
249
249
  - **Already fixed**: `Already handled by <commit-url-or-file:line>. <brief pointer>.`
250
- - **Defer**: `Out of scope for this PR; this looks like follow-up work rather than something introduced or required by this change. <one-line rationale or pointer if useful>.\n\n<sub>๐Ÿค– <code>cb-babysit:followup v1 core@3.17.1</code></sub>`
250
+ - **Defer**: `Out of scope for this PR; this looks like follow-up work rather than something introduced or required by this change. <one-line rationale or pointer if useful>.\n\n<sub>๐Ÿค– <code>cb-babysit:followup v1 core@3.18.0</code></sub>`
251
251
 
252
252
  For Defer replies, include the follow-up sentinel on its own line as shown. The script will append the `addressed` sentinel after it on its own line, so the final body ends with the follow-up sentinel followed by a blank line followed by the `addressed` sentinel. `grep cb-babysit:followup` finds the deferral and `grep cb-babysit:addressed` still marks the thread handled for dedupe.
253
253
 
@@ -263,7 +263,7 @@ The PR-level summary should:
263
263
 
264
264
  - Group by source. Use `## Review-body findings` for step-7 work and `## Conversation-tab comments` for step-6b work. Omit a section if its list is empty.
265
265
  - Inside each section, group verdicts under **Agree / Disagree / Already fixed / Deferred (out of scope)** subheadings. Omit a subheading if its list is empty.
266
- - Under **Deferred (out of scope)**, list each deferred item as a bullet, followed on its own line by `<sub>๐Ÿค– <code>cb-babysit:followup v1 core@3.17.1</code></sub>` so grep catches them individually.
266
+ - Under **Deferred (out of scope)**, list each deferred item as a bullet, followed on its own line by `<sub>๐Ÿค– <code>cb-babysit:followup v1 core@3.18.0</code></sub>` so grep catches them individually.
267
267
  - Include the commit URL for fixes.
268
268
  - End with a fenced fingerprint block listing every current fingerprint (addressed and deferred) one per line. Include both `reviewBodyComments[].fingerprint` (whole-body, one per automated review) and `activeIssueComments[].fingerprint` (per Conversation-tab comment). Future runs dedupe by matching these against `priorBabysitSentinels`.
269
269
 
@@ -10,7 +10,7 @@
10
10
 
11
11
  SENTINEL_PREFIX='cb-babysit:addressed v1 '
12
12
  LEGACY_SENTINEL_PREFIX='babysit-pr:addressed v1 '
13
- SENTINEL='<sub>๐Ÿค– <code>cb-babysit:addressed v1 core@3.17.1</code></sub>'
13
+ SENTINEL='<sub>๐Ÿค– <code>cb-babysit:addressed v1 core@3.18.0</code></sub>'
14
14
 
15
15
  # Bot author allowlist (JSON array literal). Used by unresolvedPrComments.sh
16
16
  # as a fallback when GraphQL's `author.__typename == "Bot"` misses a GitHub
@@ -17,5 +17,5 @@ Concisely explain the user intent from session history and the meaningful behavi
17
17
 
18
18
  Optional, don't fabricate: ticket links, rollout plan, residual risk, or specific areas for reviewers to focus.
19
19
 
20
- <sub>๐Ÿค– <code>cb-ship:created v1 core@3.17.1</code></sub>
20
+ <sub>๐Ÿค– <code>cb-ship:created v1 core@3.18.0</code></sub>
21
21
  ```
@@ -8,7 +8,7 @@ This rubric turns Rocky's observed review standards into checkable rules for fla
8
8
 
9
9
  Findings from automated reviewers (Mendral, CodeRabbit, etc.) are **advisory inputs, not binding rules**: the critic evaluates each on rubric merits and may decline one with stated reasoning, exactly as cb-babysit's Disagree replies do.
10
10
 
11
- **Statement-missing vs substance-missing (backtest amendment, 2026-06-11):** B5/B6/B7/B8 each require an explicit statement in the plan (dedup-check results, confidence score plus observability section, current-main status, causal-chain section). When the _statement_ is absent but the violation is not demonstrated โ€” no actual duplicate found, no actual low-confidence code fix, no actual staleness, no demonstrably symptom-terminated diagnosis โ€” the verdict is reject with disposition **amend-and-resubmit**: one bounce asking for exactly the missing sections, never escalation. A demonstrated violation (a real duplicate, a real โ‰ค2/5 code fix, a diagnosis that demonstrably stops at a symptom) is a substantive reject. The 2026-06-11 backtest showed 6 of 7 false rejections were statement-missing on plans predating these conventions; live plans carry the sections by template, but the distinction stays load-bearing for the agreement metric.
11
+ **Statement-missing vs substance-missing (backtest amendment, 2026-06-11):** B5/B6/B7/B8 each require an explicit statement in the plan (dedup-check results, Prior attempts section when applicable, confidence score plus observability section, current-main status, causal-chain section). When the _statement_ is absent but the violation is not demonstrated โ€” no actual duplicate found, no actual prior implementation tickets for the fingerprint family, no actual low-confidence code fix, no actual staleness, no demonstrably symptom-terminated diagnosis โ€” the verdict is reject with disposition **amend-and-resubmit**: one bounce asking for exactly the missing sections, never escalation. A demonstrated violation (a real duplicate, a mechanism already tried by a prior merged fix, a real โ‰ค2/5 code fix, a diagnosis that demonstrably stops at a symptom) is a substantive reject. The 2026-06-11 backtest showed 6 of 7 false rejections were statement-missing on plans predating these conventions; live plans carry the sections by template, but the distinction stays load-bearing for the agreement metric.
12
12
 
13
13
  ## 2. Banned patterns
14
14
 
@@ -54,6 +54,8 @@ Opening one plan/PR per failing test when the failures share one mechanism (same
54
54
 
55
55
  **Prior-sibling signal (backtest amendment, 2026-06-11):** sibling PRs for the same mechanism that were closed _unmerged_ are a strong reject signal, not mere context. A plan citing closed-unmerged siblings must explain what is materially different this time โ€” narrower scope alone is not an answer (the backtest's one genuine false approval cited prior wontfix siblings and explained them away).
56
56
 
57
+ **Prior attempts (amendment, 2026-07-16):** when a fingerprint family has prior implementation tickets, the plan MUST contain a **Prior attempts** section listing each prior ticket/PR, what it blamed, what it changed, and the recurrence evidence showing its diagnosis was wrong or incomplete. A missing section is statement-missing and gets amend-and-resubmit per ยง1. A plan that re-proposes a mechanism a prior merged fix already tried is a substantive reject. A family with **โ‰ฅ2 failed merged fixes** โ€” merged fixes followed by recurrence โ€” is disqualified from the normal fix path: verdict = reject and escalate as needs-human. Apply the `chronic` label and route the family to the credential-checked, dossier-first `flaky-deep-dive` skill. The โ‰ฅ2 threshold is intentionally different from flaky-triage's โ‰ฅ3-prior-implementation-tickets screen: triage counts tickets cheaply before any diagnosis exists, while the critic can see merge and recurrence outcomes and applies the sharper failed-merged-fix test. Either stage flagging chronic routes the same way; the counts are staged by the evidence available at each stage, not a contradiction.
58
+
57
59
  ### B6 โ€” Code fix on low-confidence diagnosis
58
60
 
59
61
  Proposing a code change when root-cause confidence is โ‰ค2/5, or any plan below 5/5 that omits observability follow-ups.
@@ -58,15 +58,28 @@ If the type is ambiguous, check the test file extension and imports to confirm.
58
58
 
59
59
  If the type is E2E, also classify where the failure surfaced in the lifecycle -- see [Classify the E2E Failure Surface](./references/plan-e2e.md#classify-the-e2e-failure-surface) in `plan-e2e.md`. The failure surface dictates how broadly to investigate before reading or editing the test.
60
60
 
61
- ## Phase 1b: Check for Existing Fixes
61
+ ## Phase 1b: Build the Dossier and Check for Existing Fixes
62
62
 
63
- Before investigating, check whether someone (or another agent) has already fixed this flake.
63
+ Before diagnosing, build the fingerprint family's dossier, then check whether someone (or another agent) has already fixed this flake. Do not limit the dossier to the failing file or recent activity.
64
64
 
65
- 1. **Search open PRs with the `flaky-test-fix` label** that touch the failing test file or its surrounding code. Use GitHub search scoped to the repo:
65
+ 1. **Search Linear for the fingerprint family and exact test title.** Derive the fingerprint family from the ticket title or Flake details and the full test title from `testName`. Run these four `list_issues` queries, then union and deduplicate the results by ticket ID:
66
+
67
+ ```text
68
+ project: "Groundcrew", label: "flaky-investigation", query: "<fingerprint-family>"
69
+ project: "Groundcrew", label: "flaky-investigation", query: "<exact full test title>"
70
+ project: "Groundcrew", label: "flaky-implementation", query: "<fingerprint-family>"
71
+ project: "Groundcrew", label: "flaky-implementation", query: "<exact full test title>"
72
+ ```
73
+
74
+ For all four queries, omit the state filter so every state, including terminal states, is searched. Omit created/updated date filters; there is no 14-day restriction. The `query` field must search issue titles and descriptions, not the file name alone.
75
+ - Fetch every matching issue with `get_issue`, then read its relations, comments, and linked PRs so closed and merged attempts are not lost to truncated list results
76
+
77
+ 2. **Build the `Prior attempts` table before diagnosis.** Add one row for each prior implementation ticket and linked PR found through either query. The table columns are `Prior ticket/PR`, `What it blamed`, `What it changed`, and `Recurrence evidence`. The table must list each prior ticket/PR, what it blamed, what it changed, and the recurrence evidence showing its diagnosis was wrong or incomplete. Use later investigation sightings and post-merge failures as recurrence evidence; do not infer failure merely because an attempt is old. If there are no prior implementation tickets, record `None found` plus the fingerprint and test-title searches run.
78
+ 3. **Search open PRs with the `flaky-test-fix` label** that touch the failing test file or its surrounding code. Use GitHub search scoped to the repo:
66
79
  - Search PRs labeled `flaky-test-fix` for the test file name or test directory
67
80
  - Review the PR's changes to assess whether they address the same flake pattern with reasonable confidence โ€” if so, stop and report it to the user rather than opening a duplicate fix
68
81
  - If the PR only partially addresses the flake or targets a different root cause, note it and proceed with investigation
69
- 2. **Check recent commits on `main`** that touch the failing test file or its surrounding code:
82
+ 4. **Check recent commits on `main`** that touch the failing test file or its surrounding code:
70
83
  - `git log --oneline -20 origin/main -- <test-file-path>` and also check the parent directory or related source files
71
84
  - Read the commit messages โ€” if one clearly fixes the same flake pattern, stop and report it to the user
72
85
 
@@ -75,6 +88,7 @@ If an existing fix is found, report:
75
88
  - The PR number/URL or commit hash
76
89
  - A brief summary of what it addresses
77
90
  - Whether it fully covers the current flake or only partially
91
+ - The `Prior attempts` table, including recurrence evidence for any failed prior diagnosis
78
92
 
79
93
  If no existing fix is found, proceed to Phase 2.
80
94
 
@@ -94,6 +94,7 @@ Produce the plan with these fields:
94
94
  - **Confidence:** score (1-5) with brief justification
95
95
  - **Failure surface:** CI/job setup, test setup/auth/data, app bootstrap, user action no-op, backend request, post-success render, assertion/locator, or mixed
96
96
  - **Current main status:** whether the failing commit's code path still exists on current `main`, has already been fixed, or has changed enough that the plan must be adjusted
97
+ - **Prior attempts:** list each prior ticket/PR, what it blamed, what it changed, and the recurrence evidence showing its diagnosis was wrong or incomplete. Use a table with `Prior ticket/PR`, `What it blamed`, `What it changed`, and `Recurrence evidence` columns. If the dossier search found no prior implementation tickets, write `None found` and include the fingerprint-family and exact-test-title searches run.
97
98
  - **Symptom:** what failed and where
98
99
  - **Root cause:** concise technical explanation
99
100
  - **Causal chain:** each link from failing assertion to terminal cause with its evidence, or the explicit break point and the observability that would extend it (see [Causal Chain](#causal-chain))
@@ -0,0 +1,44 @@
1
+ # Flaky Root-Cause Knowledge Base
2
+
3
+ Use this index to match a failure signature to a previously diagnosed mechanism before opening a new implementation path. Entries are indexed by mechanism, never by test. A test name or file is evidence for a mechanism, not the identity of an entry.
4
+
5
+ When adding an entry, keep the same sections used by the seeded entries:
6
+
7
+ 1. Symptom signatures
8
+ 2. Mechanism
9
+ 3. Affected repositories and surfaces
10
+ 4. What fixed it
11
+ 5. What failed and why
12
+ 6. Current status
13
+ 7. Evidence
14
+
15
+ ## Symptom index
16
+
17
+ | Symptom signature or fingerprint | Mechanism cue | Entry |
18
+ | -------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
19
+ | `element was detached from the DOM`, `element is not stable`, open dialog or sheet disappears during a mutation/refetch | Overlay is rendered below a query-driven mapped row that is reconciled or removed | [Query-driven list dialog teardown](./query-driven-list-dialog-teardown.md) |
20
+ | Dialog title or sheet content never becomes stably visible after it opened | The owning list item remounted while the overlay was open | [Query-driven list dialog teardown](./query-driven-list-dialog-teardown.md) |
21
+ | Fingerprints `33deef731a10`, `2d93cd67b48a` | Home Health instance of query-driven overlay ownership | [Query-driven list dialog teardown](./query-driven-list-dialog-teardown.md) |
22
+ | `Cognito user '<email>' does not exist`, `UserNotFoundException` immediately after a successful user-create response | Consumer is rediscovering an opaque Cognito username through an eventually consistent email alias | [Cognito email-alias eventual consistency](./cognito-email-alias-eventual-consistency.md) |
23
+ | Email-based Cognito Admin mutation fails while the same user exists under `from_admin_<uuid>` | Strong username identity was discarded at the producer boundary | [Cognito email-alias eventual consistency](./cognito-email-alias-eventual-consistency.md) |
24
+ | `TooManyRequestsException`, `MAKE_TEST_TOKEN: Too many requests`, clustered Cognito setup `401`/`403` across shards | Parallel workers exceed the shared Cognito Admin API budget | [Cognito API throttling](./cognito-api-throttling.md) |
25
+ | `AdminUpdateUserAttributes` throttle storm, especially from token setup | Stateful Cognito write is on a per-test or per-token hot path | [Cognito API throttling](./cognito-api-throttling.md) |
26
+ | Fresh entity creation succeeds, then consuming UI/API reports `Invalid workplace`, `Worker <id> not found`, opaque `500`, or readiness `422` | CDC/read-model ingestion has not reached the consuming path | [CDC and read-model readiness](./cdc-read-model-readiness.md) |
27
+ | Fixed CDC sleep passes usually but fails under load | Time delay is standing in for a missing readiness signal | [CDC and read-model readiness](./cdc-read-model-readiness.md) |
28
+ | `ChunkLoadError`, `Failed to fetch dynamically imported module`, lazy route never renders, downstream locator timeout after JS asset `5xx` | A lazy import became permanently rejected after a transient asset failure | [Lazy-chunk load recovery](./lazy-chunk-load-recovery.md) |
29
+ | Many test-specific lazy-route fix PRs for different pages in the same incident | Shared product loader lacks centralized recovery | [Lazy-chunk load recovery](./lazy-chunk-load-recovery.md) |
30
+ | Hashed JS/CSS asset returns CloudFront `503 LimitExceeded`; retries replay the same error for about 10 seconds | CloudFront error caching amplifies a short S3-origin burst | [CDN 503 amplification](./cdn-503-amplification.md) |
31
+ | Cold-cache deploy stampede followed by widespread static-asset failures | Hashed assets lack immutable caching and edge errors are pinned | [CDN 503 amplification](./cdn-503-amplification.md) |
32
+ | `LaunchDarkly initialization failed after N attempts`, app/service fails before tests start | Feature-flag bootstrap is a hard startup dependency | [Feature-flag and third-party bootstrap](./feature-flag-and-third-party-bootstrap.md) |
33
+ | Popup/new tab bypasses LaunchDarkly, analytics, monitoring, Maps, or other mocks | Mocks were installed on one page instead of the browser context | [Feature-flag and third-party bootstrap](./feature-flag-and-third-party-bootstrap.md) |
34
+ | Stripe/Maps/vendor SDK request fails, hangs, or escapes mocks before readiness UI renders | Third-party bootstrap is nondeterministic or isolated at the wrong browser scope | [Feature-flag and third-party bootstrap](./feature-flag-and-third-party-bootstrap.md) |
35
+ | Component test hangs after `userEvent.click/type`, especially with fake timers | Async `user-event` work is not awaited or timers cannot advance | [user-event and fake-timer timing](./user-event-and-fake-timer-timing.md) |
36
+ | Test passes but later throws `window is not defined`, act warning, or post-teardown timer error | Pending debounced/async work escaped teardown | [user-event and fake-timer timing](./user-event-and-fake-timer-timing.md) |
37
+ | Failures appeared during `@testing-library/user-event` v13 to v14 or Vitest migration | Previously synchronous assumptions became explicit promises and timer integration requirements | [user-event and fake-timer timing](./user-event-and-fake-timer-timing.md) |
38
+
39
+ ## Lookup discipline
40
+
41
+ - Match the mechanism using the failure surface and causal evidence, not only a shared error string.
42
+ - Read the candidate entry's failed-attempt section before proposing a fix.
43
+ - If the evidence contradicts the entry, continue the investigation and add a new mechanism only when the causal terminus is genuinely different.
44
+ - Update an existing entry when a new repository or test exposes the same mechanism.
@@ -0,0 +1,51 @@
1
+ # CDC and Read-Model Readiness
2
+
3
+ Last reviewed: 2026-07-16.
4
+
5
+ ## Symptom signatures
6
+
7
+ - A create API returns success, then the UI's consuming API reports `Invalid workplace`, `Worker <id> not found`, missing offer/estimate data, or an opaque `500`.
8
+ - A fresh shift, offer, worker, or workplace is visible in the source service but not the downstream read model.
9
+ - A fixed sleep usually works but fails during load or an environment incident.
10
+ - An outer retry re-creates fresh entities on every attempt and never lets one entity finish propagating.
11
+
12
+ ## Mechanism
13
+
14
+ The test creates data in a source-of-truth service and immediately exercises a UI path backed by another service or read model. CDC, event ingestion, or asynchronous projection has not made the new entity visible to the exact consuming path yet.
15
+
16
+ Readiness must be defined by the API/state the UI will consume. A successful producer response, an arbitrary sleep, or readiness in a different store does not prove the user path is ready.
17
+
18
+ ## Affected repositories and surfaces
19
+
20
+ - `cbh-admin-frontend`: fresh shift offers, rate negotiation, Daily View, and other seeded E2E flows.
21
+ - `clipboard-health` and downstream services such as curated shifts: producer-to-read-model ingestion boundaries.
22
+ - Any test helper that creates cross-service data and then navigates immediately to a consumer.
23
+
24
+ ## What fixed it
25
+
26
+ - Poll the same consuming API or deterministic state used by the UI.
27
+ - Bound the wait and fail with the last status, body, entity IDs, trace/request metadata, and attempt count.
28
+ - Retry only classified readiness failures; keep genuine validation errors fast-fail.
29
+ - Reuse or extract one shared per-repository readiness helper for the mechanism.
30
+ - Keep the same entity across readiness attempts. Do not re-seed and reset the CDC clock.
31
+
32
+ [STAFF-1010](https://linear.app/clipboardhealth/issue/STAFF-1010) established the accepted pattern: poll the same offer-estimate path used by the rate-negotiation dialog instead of extending a fixed CDC sleep. [cbh-admin-frontend#6780](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6780) applied the shared bounded helper to shift-offer setup.
33
+
34
+ ## What failed and why
35
+
36
+ - Increasing the fixed CDC sleep encoded a latency guess and provided no useful exhaustion evidence.
37
+ - The outer HCF retry created a new workplace/worker/shift each time; every retry restarted ingestion instead of waiting for the first entity.
38
+ - Matching only a detailed readiness message failed when the downstream service logged the specific cause but returned an opaque `500` to the client.
39
+ - Retrying all `422` responses in [cbh-admin-frontend#6974](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6974) would have made real validation errors wait 90 seconds. Review narrowed the implementation so opaque `500`s retry while unrelated `422`s still fail fast.
40
+ - Adding a new per-spec readiness loop duplicates policy and lets timeout/classification behavior drift. Extend the shared helper instead.
41
+
42
+ ## Current status
43
+
44
+ Known recurring mechanism with a documented A1 shared-helper rule in the flaky-fix rubric. The mechanism is not globally โ€œfixedโ€ because new producer/consumer boundaries continue to appear; the reusable resolution is a consuming-path readiness gate with classified failures and shared ownership.
45
+
46
+ ## Evidence
47
+
48
+ - [STAFF-1010](https://linear.app/clipboardhealth/issue/STAFF-1010): canonical accepted plan for fresh shift-offer readiness.
49
+ - [cbh-admin-frontend#6780](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6780): bounded shared shift-offer readiness helper.
50
+ - [cbh-admin-frontend#6792](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6792): JSON:API readiness-message parsing used by the helper lineage.
51
+ - [cbh-admin-frontend#6974](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6974): opaque `500` classification and reviewer-enforced `422` fast-fail behavior.
@@ -0,0 +1,50 @@
1
+ # CDN 503 Amplification for Hashed Assets
2
+
3
+ Last reviewed: 2026-07-16.
4
+
5
+ ## Symptom signatures
6
+
7
+ - Hashed JavaScript or CSS assets return CloudFront `503`, often `LimitExceeded`.
8
+ - A short S3 website-origin error burst becomes about ten seconds of identical edge failures.
9
+ - Client import retries and one reload all receive the same cached error.
10
+ - Failures cluster immediately after deploy when E2E begins against a cold CloudFront cache.
11
+ - Many routes fail together because their chunks share the same distribution/origin.
12
+
13
+ ## Mechanism
14
+
15
+ Deploy-time cold-cache demand creates a request stampede against the S3 website endpoint. A brief origin `5xx` is cached by CloudFront's default error TTL, so every retry replays the edge-cached error after the origin may already have recovered. Hashed assets without immutable cache headers also miss an opportunity to stay resident in browsers and at the edge.
16
+
17
+ The CDN turns a short origin blip into a wider, longer outage. Client recovery is necessary but cannot succeed while the error itself is pinned.
18
+
19
+ ## Affected repositories and surfaces
20
+
21
+ - `cbh-admin-frontend`: frontend deploy action and CloudFront/S3 Terraform.
22
+ - `cbh-mobile-app`: HCP webapp CloudFront/S3 Terraform.
23
+ - Staging E2E and real browser route loads that fetch content-hashed assets through those distributions.
24
+
25
+ ## What fixed it
26
+
27
+ - Set `error_caching_min_ttl = 0` for transient `500`, `502`, `503`, and `504` responses.
28
+ - Upload content-hashed JavaScript/CSS with `Cache-Control: public, max-age=31536000, immutable`.
29
+ - Preserve short/default caching for stable-named runtime/configuration files.
30
+ - Add per-status CloudFront metrics and access logs so the next burst can distinguish origin latency, cache behavior, and specific error codes.
31
+
32
+ The infrastructure fixes landed in [cbh-admin-frontend#7010](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7010) and [cbh-mobile-app#12288](https://github.com/ClipboardHealth/cbh-mobile-app/pull/12288).
33
+
34
+ ## What failed and why
35
+
36
+ - The central `loadLazily` retry ladder ran three imports and one reload within the period where CloudFront served the same cached error. Correct client recovery was defeated by edge policy.
37
+ - E2E preflight and deployed-asset graph checks from [cbh-admin-frontend#5942](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/5942), [#5957](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/5957), and [#5958](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/5958) improved detection and reduced canary pressure but did not remove error pinning.
38
+ - Selective invalidation in [cbh-admin-frontend#5840](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/5840) reduced deploy churn but did not change the transient-error TTL.
39
+ - Adding more test retries would increase the cold-cache stampede and continue replaying the cached `503`.
40
+
41
+ ## Current status
42
+
43
+ Fixed in Terraform for both frontend distributions. Origin failover remains deliberately deferred; add it only if measured bursts still fail after error caching is disabled. The log-forwarding/pipeline follow-up is operational observability, not required to understand the mechanism.
44
+
45
+ ## Evidence
46
+
47
+ - [STAFF-1470](https://linear.app/clipboardhealth/issue/STAFF-1470): cross-repository CDN amplification work.
48
+ - [cbh-admin-frontend#7010](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7010): error TTL, immutable hashed assets, metrics, and access logs.
49
+ - [cbh-mobile-app#12288](https://github.com/ClipboardHealth/cbh-mobile-app/pull/12288): mobile distribution port of the infrastructure fix.
50
+ - [cbh-mobile-app#11898](https://github.com/ClipboardHealth/cbh-mobile-app/pull/11898): deployed-asset preflight and the linked root-cause analysis used by both Terraform fixes.
@@ -0,0 +1,51 @@
1
+ # Cognito API Throttling Under Parallel Setup Load
2
+
3
+ Last reviewed: 2026-07-16.
4
+
5
+ ## Symptom signatures
6
+
7
+ - `TooManyRequestsException` from `AdminUpdateUserAttributes`, `MakeTestToken`, login, or token-mint setup.
8
+ - `MAKE_TEST_TOKEN: Too many requests`.
9
+ - HCF or auth setup produces clustered `401`/`403` failures across shards while unrelated UI assertions never run.
10
+ - Failures correlate with worker/shard fan-out and pass on retry after the burst subsides.
11
+
12
+ ## Mechanism
13
+
14
+ Parallel Playwright workers and overlapping CI runs share one staging Cognito pool and API quota. Stateful writes or token creation placed on per-test/per-token hot paths multiply across workers. In-process caches do not coordinate separate Node processes, so each worker independently repeats the same setup work.
15
+
16
+ Retries that do not reduce concurrency or call volume amplify the burst. The terminal cause is shared-capacity pressure, not the individual test that happened to request a token when the quota was exhausted.
17
+
18
+ ## Affected repositories and surfaces
19
+
20
+ - `cbh-admin-frontend`: Playwright global setup, token minting, shared admin user types, dynamic worker/HCF setup.
21
+ - `cbh-mobile-app`: shared credential acquisition and admin-auth token caches.
22
+ - Any parallel test harness using the same Cognito pool or backend token endpoint without cross-process coordination.
23
+
24
+ ## What fixed it
25
+
26
+ - Remove stateful Cognito writes from per-token paths and provision invariants at creation/global-setup boundaries.
27
+ - Deduplicate remaining writes by identity and requested attributes.
28
+ - Serialize or cap Cognito Admin/token operations across processes, not only within one worker.
29
+ - Cache shared tokens with a conservative TTL.
30
+ - Cap Playwright worker fan-out when the shared environment cannot support the theoretical concurrency.
31
+ - Preserve structured status/body/request IDs so a throttle is not misdiagnosed as an auth or UI failure.
32
+
33
+ [cbh-admin-frontend#6697](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6697) removed per-token user-type writes; [cbh-admin-frontend#6731](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6731) added cross-process slots, token caching, diagnostics, and a worker cap.
34
+
35
+ ## What failed and why
36
+
37
+ - Frontend waits and Playwright retries did not change Cognito call volume. Repeating setup could create more token and attribute operations and worsen the throttle.
38
+ - Per-process throttles/caches were insufficient because each Playwright worker is a separate process.
39
+ - Retrying every auth/setup failure without classification hid real `401`/`403` contract errors and still amplified rate-limited operations.
40
+ - Treating the Upcoming Charges test as the owner was a failure-surface mistake: the observed error occurred in shared auth setup before the page loaded.
41
+
42
+ ## Current status
43
+
44
+ Mitigated for the known admin staging setup paths. Cross-process coordination and removal of hot-path writes are the reusable fix class. Re-open the capacity question only after measuring call volume with these controls; quota increases or a dedicated E2E pool are secondary options, not the first fix.
45
+
46
+ ## Evidence
47
+
48
+ - [STAFF-1122](https://linear.app/clipboardhealth/issue/STAFF-1122): canonical investigation; CloudTrail evidence included 980 throttled `AdminUpdateUserAttributes` calls.
49
+ - [cbh-admin-frontend#6697](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6697): removed per-token Cognito writes and deduplicated remaining setup mutations.
50
+ - [STAFF-1169](https://linear.app/clipboardhealth/issue/STAFF-1169): follow-up for broader staging auth fan-out.
51
+ - [cbh-admin-frontend#6731](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6731): shared cross-process throttling, caching, worker cap, and diagnostics.
@@ -0,0 +1,50 @@
1
+ # Cognito Email-Alias Eventual Consistency
2
+
3
+ Last reviewed: 2026-07-16.
4
+
5
+ ## Symptom signatures
6
+
7
+ - `Cognito user '<email>' does not exist in pool '<pool>'` immediately after the user-create API returned success.
8
+ - `AdminSetUserPassword` or `AdminUpdateUserAttributes` raises `UserNotFoundException` when called with an email.
9
+ - A bounded retry sometimes passes under light load and fails as a storm under overlapping CI runs.
10
+ - Direct lookup by the opaque username succeeds while lookup or mutation by email still fails.
11
+
12
+ ## Mechanism
13
+
14
+ Backend user creation stores Cognito users under opaque usernames such as `from_admin_<uuid>` and stores the email as an attribute/alias. When the producer discards that username, the test harness must rediscover the identity through Cognito's email-alias index. That lookup path is eventually consistent; the strongly consistent username path is already available but was not returned across the API contract.
15
+
16
+ This mechanism is distinct from Cognito throttling. Throttling can lengthen the propagation tail, but a `UserNotFoundException` from email-based resolution after a proven successful create is an identity-contract mismatch.
17
+
18
+ ## Affected repositories and surfaces
19
+
20
+ - `clipboard-health`: workplace/facility user creation and the backend-main contract.
21
+ - `cbh-admin-frontend`: Playwright password and user-type provisioning for freshly created synthetic users.
22
+ - Other consumers that create a Cognito identity through backend-main and then address Cognito Admin APIs by email.
23
+
24
+ ## What fixed it
25
+
26
+ - Return the opaque Cognito username from the user-creation response and contract.
27
+ - Use that username for subsequent Cognito Admin mutations instead of resolving by email.
28
+ - Fail loudly or emit explicit telemetry when Cognito provisioning fails, rather than returning an apparently usable user with no identity.
29
+ - Keep a readiness probe only as a migration guard for producer paths that cannot yet return the username.
30
+
31
+ The producer-side contract fix landed in [clipboard-health#26862](https://github.com/ClipboardHealth/clipboard-health/pull/26862) for [STAFF-1793](https://linear.app/clipboardhealth/issue/STAFF-1793).
32
+
33
+ ## What failed and why
34
+
35
+ - Five `UserNotFoundException` retries spaced two seconds apart guessed that email propagation would finish within ten seconds. The guess usually held, then failed under load.
36
+ - The 75โ€“90 second readiness implementation in [cbh-admin-frontend#7380](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7380) was useful diagnostic armor but remained consumer-side: it still paid the eventually consistent email-resolution cost and added Cognito Admin calls.
37
+ - Increasing the wait cannot fix a swallowed producer failure where no Cognito identity was created. The producer must distinguish absence from propagation.
38
+ - Treating the whole storm as throttling alone would preserve the incorrect email-based identity contract.
39
+
40
+ ## Current status
41
+
42
+ The workplace-user producer contract now exposes the strongly consistent username. Consumer paths should delete email-resolution probes as they adopt the contract. Keep this entry active until all fresh-user setup paths, including worker/HCP variants, use a returned username or another deterministic identity.
43
+
44
+ ## Evidence
45
+
46
+ - [STAFF-1667](https://linear.app/clipboardhealth/issue/STAFF-1667): storm family with six tests and the shared `Cognito user does not exist` signature.
47
+ - [STAFF-1672](https://linear.app/clipboardhealth/issue/STAFF-1672): consumer-side readiness design.
48
+ - [cbh-admin-frontend#7380](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7380): closed readiness implementation and its bounded-wait diagnostics.
49
+ - [STAFF-1793](https://linear.app/clipboardhealth/issue/STAFF-1793): producer-side root fix.
50
+ - [clipboard-health#26862](https://github.com/ClipboardHealth/clipboard-health/pull/26862): returned username, fail-loud behavior, cleanup, and metrics.
@@ -0,0 +1,55 @@
1
+ # Feature-Flag and Third-Party Bootstrap Nondeterminism
2
+
3
+ Last reviewed: 2026-07-16.
4
+
5
+ ## Symptom signatures
6
+
7
+ - `LaunchDarkly initialization failed after N attempts` aborts application or service startup.
8
+ - A vendor SDK/API request for feature flags, Stripe, Maps, analytics, or monitoring fails or hangs before expected UI renders.
9
+ - A popup/new tab makes real third-party requests even though the original page was mocked.
10
+ - CORS errors or default flag values are followed by unauthorized or wrong-route API calls.
11
+ - The final test failure is a missing button/locator, but the first divergence is an external bootstrap request.
12
+
13
+ ## Mechanism
14
+
15
+ Application readiness depends on third-party initialization whose availability, page scope, or completion timing is nondeterministic.
16
+
17
+ There are two important subtypes:
18
+
19
+ 1. A true runtime dependency such as LaunchDarkly may intentionally be hard. Failing open is safe only when every flag has a verified safe default and the client will recover correctly.
20
+ 2. E2E-only external dependencies should be isolated at the right scope. Page-scoped mocks do not apply to a popup/new tab, and route navigation is not ready merely because the DOM content loaded event fired.
21
+
22
+ ## Affected repositories and surfaces
23
+
24
+ - `clipboard-health`: NestJS feature-flag provider startup.
25
+ - `cbh-admin-frontend`: facility onboarding popups, LaunchDarkly/analytics/monitoring/Places dependencies.
26
+ - `cbh-mobile-app`: vendor-backed onboarding and post-auth route bootstraps when the external integration is the earliest failing dependency.
27
+ - Any browser test that opens another page or relies on external scripts before rendering route readiness UI.
28
+
29
+ ## What fixed it
30
+
31
+ - Install E2E mocks on `browserContext.route()` when every page/tab in the context must inherit them.
32
+ - Register popup/new-page waits before the click, then wait for a product readiness signal on the new page.
33
+ - For vendor bootstraps, wait on an explicit SDK or product readiness signal and report the failed external request.
34
+ - Mock third-party behavior when it is not the system under test, while preserving the real contract shape.
35
+ - Decide LaunchDarkly degraded behavior explicitly from product safety, not from test convenience.
36
+
37
+ [cbh-admin-frontend#5989](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/5989) moved feature-flag and third-party mocks to the browser context for popup coverage.
38
+
39
+ ## What failed and why
40
+
41
+ - The blanket LaunchDarkly fail-open proposal in [clipboard-health#26748](https://github.com/ClipboardHealth/clipboard-health/pull/26748) was closed. Review established that defaults are not reliably safe and an initialized client might not recover in the intended way, so making LaunchDarkly optional could produce a broader incorrect-state outage.
42
+ - Page-scoped `page.route()` mocks passed on the original page but leaked real LaunchDarkly and third-party requests from a newly opened tab.
43
+ - Waiting only for navigation or the DOM content loaded event treated the HTML document as application readiness; a vendor SDK or flag bootstrap could still prevent React from rendering.
44
+ - Longer locator timeouts hid the external request that failed and provided no bootstrap diagnostics.
45
+ - Mocking away the third party is wrong when that integration is the behavior under test; use contract-faithful mocks only for dependencies outside the scenario.
46
+ - Do not classify every Stripe- or Maps-named test as this mechanism. [STAFF-1486](https://linear.app/clipboardhealth/issue/STAFF-1486) and [cbh-mobile-app#12309](https://github.com/ClipboardHealth/cbh-mobile-app/pull/12309) were static-asset `503` failures on a Stripe route, so they belong to the lazy-chunk/CDN entries.
47
+
48
+ ## Current status
49
+
50
+ Mixed by dependency. Browser-context mock scope and explicit vendor readiness signals are the established test patterns. LaunchDarkly remains an intentional hard dependency in backend-main until safe defaults and recovery semantics are proven; do not cite this KB entry as permission to swallow its initialization errors.
51
+
52
+ ## Evidence
53
+
54
+ - [cbh-admin-frontend#5989](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/5989): popup failure caused by page-scoped LaunchDarkly/third-party mocks.
55
+ - [clipboard-health#26748](https://github.com/ClipboardHealth/clipboard-health/pull/26748): rejected LaunchDarkly fail-open attempt and reviewer rationale.
@@ -0,0 +1,51 @@
1
+ # Lazy-Chunk Load Failures Need Product-Side Recovery
2
+
3
+ Last reviewed: 2026-07-16.
4
+
5
+ ## Symptom signatures
6
+
7
+ - `ChunkLoadError`.
8
+ - `Failed to fetch dynamically imported module`.
9
+ - A JavaScript asset request returns `5xx`, then a lazy route never renders.
10
+ - Playwright times out on a page locator even though the earliest failure is a static chunk request.
11
+ - Multiple routes/tests fail with different selectors during the same asset incident.
12
+
13
+ ## Mechanism
14
+
15
+ A transient static-asset failure rejects a dynamic import. The lazy loader or route remains stuck behind the rejected import/error boundary even after the origin recovers. Downstream locator failures are secondary because the route's code never loaded.
16
+
17
+ The recovery belongs in the shared product loader: classify chunk-load failures and perform one guarded reload for the affected app version, commit, page, and session. The test may add diagnostics, but it should not be the only recovery layer.
18
+
19
+ ## Affected repositories and surfaces
20
+
21
+ - `cbh-admin-frontend`: React lazy routes through the central `loadLazily` wrapper.
22
+ - `cbh-mobile-app`: Playwright route-bootstrap and static-asset recovery helpers; these protect tests, not real users.
23
+ - Any deployed SPA using content-hashed chunks that may briefly fail during deploy/origin/CDN incidents.
24
+
25
+ ## What fixed it
26
+
27
+ - Centralize recognition of browser chunk-load/dynamic-import failures.
28
+ - Retry the import as appropriate, then reload at most once using a session guard scoped to app version, commit, and route.
29
+ - Fall through to the normal error boundary after the bounded recovery is exhausted.
30
+ - Log whether recovery was attempted, skipped by the guard, or exhausted.
31
+
32
+ [cbh-admin-frontend#5650](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/5650) is the canonical product fix. Mobile later added Playwright-only static-asset route-bootstrap armor in [cbh-mobile-app#11958](https://github.com/ClipboardHealth/cbh-mobile-app/pull/11958) and reused it on the Stripe-named asset failure in [cbh-mobile-app#12309](https://github.com/ClipboardHealth/cbh-mobile-app/pull/12309).
33
+
34
+ ## What failed and why
35
+
36
+ - Five sibling PRs ([#5645](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/5645), [#5646](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/5646), [#5647](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/5647), [#5648](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/5648), and [#5649](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/5649)) treated individual routes/tests as separate fixes. They were consolidated because the mechanism was shared.
37
+ - Per-test route-ready retries and chunk monitors can improve diagnostics but leave real users stuck when the lazy loader itself has no recovery.
38
+ - Unbounded reloads can create a loop during a persistent deploy or asset mismatch. Recovery must be one-time and version/route scoped.
39
+ - Loader recovery alone could not defeat a CDN that cached the same `503` for every retry; that is the separate [CDN amplification mechanism](./cdn-503-amplification.md).
40
+
41
+ ## Current status
42
+
43
+ Product-side recovery is merged for the known admin loader. Mobile's cited recovery is test-harness-only, so a real-user mobile web failure still needs a product-side loader diagnosis. Future per-route sightings should first verify whether the surface has product recovery, harness armor, or neither, and whether the asset failure is transient. Do not create one implementation per test.
44
+
45
+ ## Evidence
46
+
47
+ - [cbh-admin-frontend#5645](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/5645): representative closed per-surface attempt that mixed product and E2E recovery.
48
+ - [cbh-admin-frontend#5650](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/5650): merged centralized lazy-chunk recovery.
49
+ - [cbh-admin-frontend#5492](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/5492): earlier chunk-load logging and user-facing refresh behavior.
50
+ - [cbh-mobile-app#11958](https://github.com/ClipboardHealth/cbh-mobile-app/pull/11958): Playwright app-bootstrap static-asset retry helper.
51
+ - [cbh-mobile-app#12309](https://github.com/ClipboardHealth/cbh-mobile-app/pull/12309): Stripe-route sighting whose actual cause was CloudFront/static assets, not Stripe bootstrap.
@@ -0,0 +1,52 @@
1
+ # Query-Driven List Dialog Teardown
2
+
3
+ Last reviewed: 2026-07-16.
4
+
5
+ ## Symptom signatures
6
+
7
+ - Playwright reports `element is not stable` followed by `element was detached from the DOM`.
8
+ - A dialog or sheet opens, then its title/content disappears or never becomes stably visible.
9
+ - The failure moves between actions on different rows of the same query-backed list.
10
+ - A mutation succeeds, but the confirmation interaction or post-success assertion loses the overlay.
11
+ - Known families: `33deef731a10` and `2d93cd67b48a`.
12
+
13
+ ## Mechanism
14
+
15
+ The overlay is owned by a component rendered beneath a mapped query result, such as `cases.map(...<CaseCard>)` or `visits.map(...<VisitCard>)`. Background, focus, interval, or mutation-triggered query refreshes reconcile, reorder, or remove the row while the overlay is open. Because the dialog or sheet lifecycle is tied to that row, the overlay remounts or unmounts during interaction.
16
+
17
+ The decisive test is structural: if deleting or replacing the launching row also destroys the open overlay, the product has this mechanism even when the visible failure is a locator timeout.
18
+
19
+ ## Affected repositories and surfaces
20
+
21
+ - `cbh-admin-frontend`: Home Health case and visit action dialogs; other query-driven list dialogs found by the repository sweep.
22
+ - `cbh-mobile-app`: placement sign-on-bonus and badge-history sheets; workplace-review comment and reply action sheets.
23
+ - Any React surface that renders modal state and overlay JSX inside a query-backed `.map()` descendant.
24
+
25
+ ## What fixed it
26
+
27
+ - Hoist overlay ownership above the mapped list into one stable, entity-keyed host. Rows become trigger-only and pass entity identity to the host.
28
+ - Add regression coverage that opens the overlay, removes or refreshes the launching row, and proves the overlay survives.
29
+ - Add an architecture guard for direct dialog/modal/sheet JSX inside `.map()` callbacks, then perform a semantic sweep for indirect descendants the syntax guard cannot see.
30
+ - Split oversized sequential E2E flows after the product fix so one mechanism does not surface as many indistinguishable steps.
31
+
32
+ The durable Home Health fix is [cbh-admin-frontend#7574](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7574). Repository-wide follow-ups landed in [cbh-admin-frontend#7569](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7569), [cbh-mobile-app#12857](https://github.com/ClipboardHealth/cbh-mobile-app/pull/12857), and [cbh-mobile-app#12860](https://github.com/ClipboardHealth/cbh-mobile-app/pull/12860).
33
+
34
+ ## What failed and why
35
+
36
+ - Repeated `toPass` blocks, response waits, locator retries, failure trackers, and expanded timeouts hardened whichever Home Health step failed most recently but left overlay ownership unchanged. The family recurred across [STAFF-1329](https://linear.app/clipboardhealth/issue/STAFF-1329), [STAFF-1369](https://linear.app/clipboardhealth/issue/STAFF-1369), [STAFF-1495](https://linear.app/clipboardhealth/issue/STAFF-1495), [STAFF-1554](https://linear.app/clipboardhealth/issue/STAFF-1554), [STAFF-1582](https://linear.app/clipboardhealth/issue/STAFF-1582), [STAFF-1673](https://linear.app/clipboardhealth/issue/STAFF-1673), and [STAFF-1685](https://linear.app/clipboardhealth/issue/STAFF-1685).
37
+ - Preserving modal state across a remount did not preserve the DOM node Playwright or a user was interacting with. The Team Members recurrence from [STAFF-1422](https://linear.app/clipboardhealth/issue/STAFF-1422) to [STAFF-1459](https://linear.app/clipboardhealth/issue/STAFF-1459) is the precedent.
38
+ - Pausing backing-list observers and deferring invalidations in [cbh-admin-frontend#7559](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7559) reduced the race window but was explicitly an interim mitigation: another ancestor update could still remove the row. The hoisted host later deleted that workaround.
39
+ - Splitting the E2E test alone would reduce blast radius and improve fingerprints, but it would not fix the user-facing overlay teardown.
40
+
41
+ ## Current status
42
+
43
+ Fixed for the known admin and mobile surfaces covered by [STAFF-1789](https://linear.app/clipboardhealth/issue/STAFF-1789), [STAFF-1790](https://linear.app/clipboardhealth/issue/STAFF-1790), [STAFF-1796](https://linear.app/clipboardhealth/issue/STAFF-1796), [STAFF-1797](https://linear.app/clipboardhealth/issue/STAFF-1797), and [STAFF-1806](https://linear.app/clipboardhealth/issue/STAFF-1806). Treat future instances as the same mechanism when overlay ownership is still below a replaceable query row; extend the stable-host pattern and guard rather than adding per-test waits.
44
+
45
+ ## Evidence
46
+
47
+ - [STAFF-1789](https://linear.app/clipboardhealth/issue/STAFF-1789): mitigation and fault-injection proof for Home Health list refreshes.
48
+ - [STAFF-1790](https://linear.app/clipboardhealth/issue/STAFF-1790): durable host design and E2E decomposition.
49
+ - [cbh-admin-frontend#7559](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7559): refetch guard, with residual ownership risk documented.
50
+ - [cbh-admin-frontend#7574](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7574): hoisted host and deletion of the temporary guard.
51
+ - [cbh-mobile-app#12857](https://github.com/ClipboardHealth/cbh-mobile-app/pull/12857): mobile sheet hoists and architecture guard.
52
+ - [cbh-mobile-app#12860](https://github.com/ClipboardHealth/cbh-mobile-app/pull/12860): indirect workplace-review descendants hoisted after the syntactic guard could not detect them.
@@ -0,0 +1,52 @@
1
+ # user-event and Fake-Timer Component-Test Timing
2
+
3
+ Last reviewed: 2026-07-16.
4
+
5
+ ## Symptom signatures
6
+
7
+ - Component test hangs or times out after `userEvent.click`, `type`, `clear`, or keyboard input.
8
+ - Assertions run before event-driven state updates complete.
9
+ - Fake-timer suites stop progressing because `user-event` scheduled work cannot advance.
10
+ - A test passes, then emits an act warning, unhandled rejection, or `window is not defined` after jsdom teardown.
11
+ - Failures appear after a `user-event`, Vitest, React, or Testing Library upgrade exposes previously implicit timing.
12
+
13
+ ## Mechanism
14
+
15
+ `@testing-library/user-event` v14 APIs are asynchronous and model multiple browser events. Tests written with v13-style synchronous calls can assert or tear down before those events finish. When fake timers are active, the configured `user-event` instance must be able to advance the runner's timers; otherwise its internal delays and the component's scheduled work deadlock.
16
+
17
+ A related teardown class occurs when debounced component-library timers remain pending after the DOM environment is destroyed. Timer ownership and cleanup, not a larger timeout, determine correctness.
18
+
19
+ ## Affected repositories and surfaces
20
+
21
+ - `cbh-mobile-app`: Jest and Vitest component tests migrated from `user-event` v13.5 to v14.
22
+ - `cbh-admin-frontend`: component-library debounces and fake-timer teardown patterns.
23
+ - Any component test mixing `user-event`, fake timers, MSW, debounced UI libraries, animations, or scheduled state updates.
24
+
25
+ ## What fixed it
26
+
27
+ - Upgrade to `user-event` v14 and await every interaction.
28
+ - Prefer one `const user = userEvent.setup(...)` instance per test.
29
+ - In fake-timer suites, configure `advanceTimers` with the active runner, such as `vi.advanceTimersByTime` or `jest.advanceTimersByTime`.
30
+ - Flush only the work the scenario owns, restore real timers in `afterEach`, and keep cleanup before jsdom teardown.
31
+ - Enforce `testing-library/await-async-events` so new interactions that are not awaited fail lint.
32
+
33
+ The repository-wide migration landed in [cbh-mobile-app#10310](https://github.com/ClipboardHealth/cbh-mobile-app/pull/10310).
34
+
35
+ ## What failed and why
36
+
37
+ - [cbh-mobile-app#5565](https://github.com/ClipboardHealth/cbh-mobile-app/pull/5565) attempted a broad `fireEvent` to `userEvent` lint conversion before the full async/timer migration. It was closed with unresolved lint and test failures; changing the API name alone did not establish correct timing.
38
+ - Static `userEvent.*` calls mixed with `userEvent.setup()` left inconsistent isolation and made missing awaits easy to reintroduce.
39
+ - Adding per-test or global timeout increases, including the Vitest 4 migration's timeout accommodation in [cbh-mobile-app#11947](https://github.com/ClipboardHealth/cbh-mobile-app/pull/11947), can make slow suites pass but does not fix a deadlocked fake timer or interaction that was not awaited.
40
+ - Fake timers alone are not sufficient. Without `advanceTimers`, `user-event` may wait forever; without teardown flushing/restoration, callbacks may fire after jsdom is gone.
41
+ - The closed [cbh-admin-frontend#6699](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6699) correctly identified a pending MUI debounce but never landed, illustrating that a plausible timer diagnosis still needs a maintained, merged cleanup pattern.
42
+
43
+ ## Current status
44
+
45
+ The mobile v14 migration and lint enforcement are merged. Treat new component-test timing failures as one of three explicit subtypes before editing: interaction not awaited, fake-timer integration, or pending teardown work. Do not default to timeout inflation.
46
+
47
+ ## Evidence
48
+
49
+ - [cbh-mobile-app#10310](https://github.com/ClipboardHealth/cbh-mobile-app/pull/10310): v13.5 to v14 migration, approximately 1,300 awaited calls, fake-timer `advanceTimers` setup, and lint enforcement.
50
+ - [cbh-mobile-app#5565](https://github.com/ClipboardHealth/cbh-mobile-app/pull/5565): incomplete earlier mechanical conversion.
51
+ - [cbh-mobile-app#11947](https://github.com/ClipboardHealth/cbh-mobile-app/pull/11947): Vitest 4 migration showing remaining user-event/fake-timer/MSW timing pressure.
52
+ - [cbh-admin-frontend#6699](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6699): post-teardown MUI debounce signature and proposed timer cleanup.
@@ -0,0 +1,138 @@
1
+ ---
2
+ name: flaky-deep-dive
3
+ description: Investigate chronic flaky-test fingerprint families with a credential-checked, dossier-first, cross-repository causal-chain method and produce a designed fix or observability design for human review. Use when flaky-triage routes a family with at least three prior implementation tickets, flaky-critic routes a family with at least two failed merged fixes, or a user asks for the chronic flake deep-dive track.
4
+ ---
5
+
6
+ # Flaky Deep Dive
7
+
8
+ Run a diagnosis-and-design investigation for a chronic fingerprint family. Produce a document for human review; never edit product code, create a fix branch, commit a fix, or open a fix PR.
9
+
10
+ Read the bundled [Home Health dry run](references/dry-run-home-health-full-lifecycle.md) for the expected dossier and causal narrative depth. Its structure is distilled from Rocky's `hh-full-lifecycle.md` and `staging-seed-reliability.md` reference deep dives; the Phase 6 checklist remains binding where the historical artifact records an evidence limitation.
11
+
12
+ ## Phase 0: Enforce credential preconditions
13
+
14
+ Run the preflight before reading source code or diagnosing:
15
+
16
+ ```bash
17
+ bash "<flaky-deep-dive-skill-dir>/scripts/check-prerequisites.sh" "${AWS_PROFILE:-sdlc}"
18
+ ```
19
+
20
+ The preflight must prove all three:
21
+
22
+ 1. `pup` can query Datadog APM spans and logs.
23
+ 2. The staging API is reachable through the non-production VPN.
24
+ 3. AWS read credentials are active through `aws sts get-caller-identity`.
25
+
26
+ If any check fails, stop the investigation. Do not continue with reduced evidence or silently cap confidence. File a provisioning ticket linked to the chronic ticket with:
27
+
28
+ - the fingerprint family and blocked investigation;
29
+ - the failed command and exact error class;
30
+ - the missing access: Datadog APM/log read, non-production VPN, or AWS `sdlc` read credentials;
31
+ - the causal-chain link or fault-injection step the access blocks;
32
+ - the remediation printed by the preflight.
33
+
34
+ If the environment cannot file Linear tickets, output the exact title and body for a human to file. Do not proceed blind.
35
+
36
+ For a CI-sourced E2E family, also prove access to the exact run and reporter artifact:
37
+
38
+ ```bash
39
+ bash "<flaky-deep-dive-skill-dir>/../flaky-debug/scripts/fetch-llm-report.sh" \
40
+ "<github-actions-url>"
41
+ ```
42
+
43
+ If `gh` authentication fails, run `gh auth login --hostname github.com`. If the artifact expired, rerun the source workflow or obtain the `playwright-llm-report`/Playwright HTML report. An inaccessible source artifact blocks the investigation.
44
+
45
+ ## Phase 1: Build the dossier first
46
+
47
+ Do not diagnose until the dossier is complete.
48
+
49
+ Read and follow `../flaky-debug/SKILL.md` Phase 1b in full. Its four unbounded Linear searches for the fingerprint family and exact test title are the canonical dossier recipe; do not narrow, copy, or alter them.
50
+
51
+ The chronic track adds these requirements for every implementation attempt returned by that recipe:
52
+
53
+ 1. Read the ticket and investigation plan.
54
+ 2. Read the complete PR diff, review discussion, merge state, and fix commit through `gh`.
55
+ 3. Record what the attempt blamed and exactly what it changed.
56
+ 4. Find a later post-merge sighting that proves recurrence. Do not infer failure merely from age.
57
+ 5. Check current `main` in every repository the attempt touched.
58
+
59
+ Create a `Prior attempts` table with these columns:
60
+
61
+ | Prior ticket/PR/commit | What it blamed | What it changed | Recurrence evidence |
62
+ | ---------------------- | -------------- | --------------- | ------------------- |
63
+
64
+ Every prior ticket, PR, and fix commit must appear. Attempt N+1 must rule out attempts 1..N: after the table, list each prior blamed mechanism and the evidence that falsifies it, limits it to one symptom, or proves it incomplete. A new attempt that merely repeats a prior mechanism is not a deep dive.
65
+
66
+ Also search open and closed-unmerged PRs labeled `flaky-test-fix`, plus recent `main` commits touching the test, helper, product surface, and implicated services. Stop and report if a current fix already covers the family.
67
+
68
+ ## Phase 2: Resolve ownership from the workspace
69
+
70
+ Start from evidence, not the repository containing the test.
71
+
72
+ Read and follow the ownership and causal-chain rules in `../flaky-debug/references/plan.md`. The chronic track additionally requires recording the registry result and checkout SHA for every implicated repository.
73
+
74
+ Locate a `ClipboardHealth/groundtruth` checkout. Resolve deployed services through `registry/services.json`, then resolve repositories and owners through `registry/repos.json`:
75
+
76
+ ```bash
77
+ jq --arg service "<service-id>" \
78
+ '.services[] | select(.id == $service)' \
79
+ "<groundtruth>/registry/services.json"
80
+
81
+ jq --arg repo "<repo-id>" \
82
+ '.repos[] | select(.id == $repo)' \
83
+ "<groundtruth>/registry/repos.json"
84
+ ```
85
+
86
+ Read `context/devin-wiki/<repo>/` when it helps locate the owning code. Follow the causal chain into every implicated repository and inspect current `main`; never stop at the repository where the test lives. Record the resolved service, repository, owner, route or responsibility, and code checkout SHA in the deliverable.
87
+
88
+ If the registry is missing or cannot resolve an implicated service, treat ownership as a broken causal-chain link and design the registry/telemetry change needed to resolve it.
89
+
90
+ ## Phase 3: Gather evidence across every link
91
+
92
+ Use all applicable evidence sources:
93
+
94
+ - **LLM reporter and Playwright artifacts:** identify the exact attempt, lifecycle step, network instance, request/response body, trace ID, request ID, screenshot, and stack.
95
+ - **Datadog APM:** follow the per-request backend-selected trace ID. Read `../flaky-debug/references/datadog-apm-traces.md` before querying.
96
+ - **Datadog run-window logs:** query the absolute CI window for seed operations, deploys, CDC lag, queues, background jobs, and other causal events outside the request path. A trace-only search cannot establish these events.
97
+ - **GitHub Actions:** use `gh run view`, job logs, annotations, artifacts, retry history, commit SHA, and deployed-version evidence.
98
+ - **AWS read-only evidence:** use the active staging credentials for service-specific read commands when Datadog and CI artifacts leave an AWS-managed link unresolved.
99
+ - **Current source:** cite file and line, config key, workflow step, or specific log event in the owning repository.
100
+
101
+ Preserve exact commands, absolute time windows, trace/request IDs, run URLs, commit SHAs, and zero-result limitations in the evidence appendix.
102
+
103
+ ## Phase 4: Terminate the causal chain
104
+
105
+ Apply the causal-chain and confidence rules in `../flaky-debug/references/plan.md` without changing their score meanings or valid terminal states.
106
+
107
+ Fault-injection reproduction is mandatory for 5/5. Use a focused lower-level test or harness control that delays, fails, reorders, throttles, or disables the blamed response or step. A normal green rerun is not reproduction.
108
+
109
+ When the chain is broken, do not design a speculative code fix. Convert the deliverable into the observability design that would terminate the chain on the next occurrence, including event names, fields, correlation IDs, service/repository, retention, reporter exposure, and the query that would prove or falsify the hypothesis.
110
+
111
+ ## Phase 5: Design the fix for review
112
+
113
+ For a terminated chain, design the smallest fix at the terminal cause and name:
114
+
115
+ - owning repository, files, and team;
116
+ - behavior and contract changes;
117
+ - migration, rollout, and compatibility constraints;
118
+ - defensive apparatus or prior mitigations to delete after the fix;
119
+ - fault-injection and no-retry validation;
120
+ - residual risks and falsifying outcomes.
121
+
122
+ Separate the root fix from optional resilience or observability follow-ups. Do not turn another service's defect into a local test retry. Do not implement the design.
123
+
124
+ ## Phase 6: Write the deliverable
125
+
126
+ Write Markdown to the user-specified path, or `/tmp/flaky-deep-dive-<fingerprint>.md` by default, with:
127
+
128
+ 1. Title and metadata: family, repositories, status, investigator, current-main SHAs, confidence.
129
+ 2. Credential preflight evidence.
130
+ 3. `Dossier` and `Prior attempts`.
131
+ 4. Latest recurrence signatures.
132
+ 5. Causal chain with ownership resolution.
133
+ 6. Root-cause statement and discriminated alternatives.
134
+ 7. Designed fix, or `Observability design` when the chain is broken.
135
+ 8. Fault-injection reproduction and validation plan.
136
+ 9. Evidence appendix with links, commands, IDs, timestamps, and citations.
137
+
138
+ End with the document path/link and a one-sentence review request. Never open a direct fix PR.
@@ -0,0 +1,177 @@
1
+ # Dry run: Home Health `fullLifecycle.spec.ts` chronic flake
2
+
3
+ - **Repo:** `cbh-admin-frontend`
4
+ - **Spec:** `playwright/e2e/homeHealth/fullLifecycle.spec.ts`
5
+ - **Fingerprint family:** `33deef731a10` with related variants
6
+ - **Status:** Closed as Done at least six times before recurring
7
+ - **Investigator:** Deep-dive track; diagnosis and design only
8
+ - **Confidence:** 4/5; the causal chain terminates, but fault injection remains
9
+ - **Source:** Real chronic-family deep dive completed on 2026-07-07
10
+ - **Historical code SHA:** `fe4e2cd8424`
11
+ - **Current-main SHA checked:** `1bafb36f8f909274b96eef840655ffdc7e15f9d0`
12
+
13
+ This checked-in dry run demonstrates the expected depth and document shape. The original investigation lacked a VPN-capable environment, which correctly held confidence below 5/5. The credential-checked forward rerun below stopped before diagnosis, as the new skill requires.
14
+
15
+ ## 0. Credential preflight and current-main status
16
+
17
+ ### Credential-checked forward rerun on 2026-07-16
18
+
19
+ - Datadog APM: passed a staging span query through `pup`.
20
+ - Datadog logs: passed a staging log query through `pup`.
21
+ - Staging VPN: failed. `curl --connect-timeout 10 --max-time 20 https://apigateway.staging.clipboardhealth.org/api/healthCheck` exited 56 with `CONNECT tunnel failed, response 403`.
22
+ - AWS: failed. `aws sts get-caller-identity --profile sdlc` exited 255 because the `sdlc` profile was not configured.
23
+
24
+ The live forward test stopped at Phase 0 and produced a provisioning-ticket body. It did not reuse the historical diagnosis as if current credentials had passed.
25
+
26
+ ### Current main
27
+
28
+ The historical designed fix has since landed:
29
+
30
+ - [PR #7574](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7574), commit `e6818dabd950d43755c3ec96de0373062a4522ba`, hoisted the Home Health action dialogs and split the 2,096-line mega-spec.
31
+ - [PR #7581](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7581), commit `ed0f1e63e39e24e771b47cbe883f5925be9d45d6`, hardened the stable host and case-dialog loading path.
32
+ - At `origin/main@1bafb36f8f909274b96eef840655ffdc7e15f9d0`, `HomeHealthDialogHost.tsx:67-77` renders the active dialog beside the list children, and `HomeHealthActiveDialog.tsx:63-81` owns every case and visit action dialog.
33
+
34
+ A new investigation must stop as already fixed unless recurrence evidence post-dates those commits. The implementation matching the dry run's design is evidence that the causal document was actionable; it is not permission for this skill to open a PR.
35
+
36
+ ## 1. Dossier
37
+
38
+ The spec was created by consolidating seven Home Health E2E files into one ten-step lifecycle test. Every later fix hardened the step where the failure last surfaced. None removed the product lifecycle churn shared by the failures.
39
+
40
+ | Prior ticket/PR/commit | What it blamed | What it changed | Recurrence evidence |
41
+ | ------------------------------------- | --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------- |
42
+ | STAFF-1329 / PR #6902 / `d6633c656e3` | Occurrence tab readiness | Added waits for occurrence tabs | Failure moved to visit-row lookup in STAFF-1369 |
43
+ | STAFF-1327 / PR #6903 / `f1bf003b6c2` | Visit-verification tab state | Changed product tab state and spec waits | Failure moved to visit-row and ROC lookup |
44
+ | STAFF-1369 / PR #6915 / `3a11b8effc8` | Visit-row render race | Added row-lookup retries and a negotiation tweak | Recurred as ROC visit lookup in STAFF-1495 |
45
+ | STAFF-1377 / PR #6930 / `07491124e72` | Remove-booked-worker assertion | Changed the dialog and assertion | Later failures surfaced in other dialogs |
46
+ | STAFF-1449 / PR #6987 / `53330b9c0ea` | Create transport errors | Added fail-fast handling and more waits | The family recurred; machinery grew without removing the cause |
47
+ | STAFF-1441 / PR #6990 / `51577fa1778` | Response-wait timing | Rewrote lifecycle response waits | STAFF-1495 recurred within days |
48
+ | STAFF-1495 / PR #7050 / `5ebd5ee839a` | ROC visit lookup | Hardened ROC-specific lookup | STAFF-1554 recurred at auth-refresh waits |
49
+ | STAFF-1554 / PR #7251 / `2f942662c37` | Auth refresh during waits | Added a 306-line response helper tolerant of refresh | STAFF-1582 recurred at close case |
50
+ | STAFF-1582 / PR #7316 / `28bdf114626` | Close-case flow | Reworked close-case steps | STAFF-1685 recurred with the identical title |
51
+ | STAFF-1601 / PR #7321 / `b09225ddc35` | Visit-verification modal state | Changed occurrence-table dialog behavior | STAFF-1673 recurred at cancel-visit dialog |
52
+ | STAFF-1673 / PR #7374 / `ac7c1b79ae5` | Cancel-visit dialog open | Added `toPass`, immediate-failure tracking, and diagnostics | STAFF-1682 recurred six days later at close case |
53
+ | STAFF-1685 / PR #7390 / `fe4e2cd8424` | Close case again | Added another large close-case hardening block | STAFF-1659 and STAFF-1682 still failed |
54
+ | STAFF-1789 / PR #7559 / `2b7ebc87c2d` | Backing-list refreshes while dialogs are open | Paused/canceled list observers and deferred invalidations behind `HomeHealthListRefetchGuard` with fault-injection tests | The PR documented that dialogs still lived below mapped cards; PR #7574 removed the 217-line guard and implemented the durable host |
55
+ | STAFF-1790 / PR #7574 / `e6818dabd95` | Dialog lifecycle coupled to mapped cards | Hoisted all desktop action dialogs and split the mega-spec | No post-merge recurrence found in the current-main recheck |
56
+ | PR #7581 / `ed0f1e63e39` | Remaining host loading/history edges | Hardened the stable host and case-dialog loading path | No post-merge recurrence found in the current-main recheck |
57
+
58
+ ### Ruled-out prior hypotheses
59
+
60
+ Before the durable host in PR #7574, the next attempt had to rule out the twelve symptom fixes and the #7559 mitigation:
61
+
62
+ - Readiness and locator timing are incomplete explanations because the same family migrated across tabs, rows, auth waits, and several independent dialogs after those waits landed.
63
+ - Response-wait rewrites are incomplete because later signatures show DOM detachment after the response machinery was centralized.
64
+ - Preserving one dialog's state is incomplete because sibling dialogs under the same list-item lifecycle continued to unmount.
65
+ - More retries are falsified by recurrence after roughly one thousand net lines of retry and diagnostic apparatus accumulated in the spec and helpers.
66
+ - Pausing known list refreshes was incomplete because #7559's own residual-risk statement left the dialogs below mapped cards and exposed them to ancestor updates outside those queries.
67
+
68
+ ### Latest recurrence signatures
69
+
70
+ - STAFF-1682: the close button repeatedly became unstable and detached from the DOM during `locator.click`.
71
+ - STAFF-1659: the cancel-visit dialog title never reached a stable visible state within 30 seconds.
72
+
73
+ Both signatures point to component lifecycle churn, not a slow assertion.
74
+
75
+ ## 2. Causal chain
76
+
77
+ **Terminal cause:** Home Health action dialogs are React descendants of query-backed list items. The dialogs' own mutations invalidate and refresh those lists. Background refreshes and status changes reconcile, reorder, or remove the list item while its dialog is open, destabilizing or unmounting the dialog subtree.
78
+
79
+ ```text
80
+ Playwright click or dialog-title assertion fails
81
+ โ†’ the open dialog becomes unstable or disappears
82
+ โ†’ query invalidation/background refetch returns a new case or visit list
83
+ โ†’ CaseCard or VisitCard reconciles, reorders, or leaves the open list
84
+ โ†’ the dialog subtree, owned by that list item, remounts or unmounts
85
+ โ†’ terminal cause: dialog lifecycle is coupled to invalidated list-item lifecycle
86
+ ```
87
+
88
+ ### Ownership and historical code
89
+
90
+ - `useUpdateVisit.ts:66` invalidates the case-visits query after mutations.
91
+ - `useUpdateCase.ts:44-56` invalidates open and closed agency-case queries plus case-visits data.
92
+ - `useCaseVisits.ts` and `useAgencyCases.ts:69` use five-minute stale and ten-minute interval refresh behavior.
93
+ - `CaseVisits.tsx:42` sorts and maps visits on render.
94
+ - `CaseActionsMenu.tsx:92` renders `CaseCloseDialog` inside the case card's actions.
95
+ - `Visit/VisitActions.tsx:70` renders `CancelVisitDialog` inside the visit card's actions.
96
+
97
+ Groundtruth resolves service `admin-web-app` to repository `cbh-admin-frontend`; its listed owners are Synapse, Platform, Growth, Tempo, Alchemy, and Action. The repository registry owner is Platform. No backend service is needed to terminate this particular chain.
98
+
99
+ ### Close-case chain
100
+
101
+ 1. The test opens the close dialog and clicks Close.
102
+ 2. A list invalidation or background refetch resolves while the dialog is open.
103
+ 3. The cases array changes and the owning card rerenders, producing the observed unstable element.
104
+ 4. The close mutation changes the case to closed.
105
+ 5. The open-cases query returns without that case, unmounting the card and dialog, producing the observed detached element.
106
+
107
+ ### Cancel-visit chain
108
+
109
+ 1. The test opens the cancel dialog.
110
+ 2. A prior visit mutation's invalidation resolves.
111
+ 3. The visit list resorts or reconciles.
112
+ 4. The owning visit card and its dialog remount.
113
+ 5. The title disappears before it reaches a stable asserted state.
114
+
115
+ ## 3. Root cause and confidence
116
+
117
+ Home Health action dialogs bind their lifecycle to `CaseCard` and `VisitCard` instances produced by refetching lists. Those lists are invalidated by the dialogs' mutations and refreshed independently. A refetch during an open dialog can reconcile or remove the owning item, destroying the dialog during user interaction.
118
+
119
+ This is a user-facing product race, not test-only behavior. A real administrator can lose an open dialog during background refresh; the long CI workflow merely expands the race window.
120
+
121
+ Confidence is **4/5**. The render ownership, invalidation path, refresh behavior, and failure signatures establish a terminated static chain. Fault injection is still required for 5/5.
122
+
123
+ ## 4. Designed fix
124
+
125
+ ### Root fix
126
+
127
+ Hoist Home Health case and visit action dialogs out of `CaseCard` and `VisitCard` into one stable dialog host mounted above the query-backed lists.
128
+
129
+ - Add a `HomeHealthDialogHost` at the page or dashboard shell.
130
+ - Make list items request `openDialog({ kind, entityId })` instead of owning modal state and rendering dialogs inline.
131
+ - Keep one dialog instance keyed by entity ID in a subtree unaffected by list reconciliation.
132
+ - Close only on explicit user action or mutation success.
133
+
134
+ ### Defense in depth
135
+
136
+ While a Home Health dialog is open, pause optional focus/interval refresh churn or defer broad invalidation until the dialog closes. This reduces churn but does not replace the lifecycle fix.
137
+
138
+ ### Cleanup after the root fix
139
+
140
+ Remove the `toPass` blocks, immediate-failure trackers, and open-dialog retry apparatus added by prior attempts once fault injection proves the stable host survives refreshes.
141
+
142
+ ### Test decomposition
143
+
144
+ Split the ten-step mega-spec into smaller product-surface workflows with isolated seeded data, retaining one thin lifecycle smoke path if required. This reduces blast radius and gives each real mechanism a distinct fingerprint.
145
+
146
+ ## 5. Fault-injection reproduction
147
+
148
+ 1. Run the affected flow against a seeded case.
149
+ 2. Delay the close mutation while forcing the open-cases query to refetch and resolve during the click.
150
+ 3. Expect the current implementation to reproduce the detached-element signature.
151
+ 4. For cancel visit, force the case-visits refetch to resolve while the dialog is open and expect the title to disappear.
152
+ 5. Apply a prototype of the stable dialog host.
153
+ 6. Repeat both injections and prove the dialogs survive.
154
+ 7. Run the affected specs repeatedly with Playwright retries disabled.
155
+
156
+ The original dry run could not execute these steps because the environment lacked staging VPN access. Under the new skill, that precondition failure requires a provisioning ticket rather than continued investigation.
157
+
158
+ ## 6. Observability fallback
159
+
160
+ If fault injection cannot yet run after access is provisioned, add a CI-visible dialog generation counter and emit an event whenever a Home Health dialog unmounts without explicit close or mutation success. Include dialog kind, entity ID, query keys invalidated in the preceding window, and a correlation ID exposed in the Playwright reporter.
161
+
162
+ The next occurrence should answer directly whether a list refetch caused the unexpected unmount.
163
+
164
+ ## 7. Evidence appendix
165
+
166
+ - Chronic family: `33deef731a10` plus variants.
167
+ - Consolidation: [PR #6616](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6616), commit `afb53308a38`.
168
+ - Latest signatures: STAFF-1682 close-case detachment and STAFF-1659 cancel-dialog invisibility.
169
+ - Historical code checkout: `cbh-admin-frontend@fe4e2cd8424`.
170
+ - Current-main checkout: `cbh-admin-frontend@1bafb36f8f909274b96eef840655ffdc7e15f9d0`.
171
+ - Ownership query: `jq '.services[] | select(.repo == "cbh-admin-frontend")' groundtruth/registry/services.json`.
172
+ - Repository query: `jq --arg repo cbh-admin-frontend '.repos[] | select(.id == $repo)' groundtruth/registry/repos.json`.
173
+ - Current-main verification: `git log origin/main -- HomeHealthActiveDialog.tsx HomeHealthDialogHost.tsx`.
174
+ - Key historical product areas: `CaseDashboard`, `CaseActionsMenu`, `CaseVisits`, `VisitActions`, `useUpdateCase`, `useUpdateVisit`, `useAgencyCases`, and `useCaseVisits`.
175
+ - Prior PRs: [#6902](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6902), [#6903](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6903), [#6915](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6915), [#6930](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6930), [#6987](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6987), [#6990](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6990), [#7050](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7050), [#7251](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7251), [#7316](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7316), [#7321](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7321), [#7374](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7374), [#7390](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7390), [#7559](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7559), [#7574](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7574), and [#7581](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7581).
176
+ - Historical limitation: the surviving document did not retain the exact Actions run URL, trace ID, or request ID. New `flaky-deep-dive` outputs must retain them when applicable.
177
+ - Historical output: Rocky's `hh-full-lifecycle.md`, created 2026-07-07.
@@ -0,0 +1,177 @@
1
+ #!/usr/bin/env bash
2
+
3
+ set -uo pipefail
4
+
5
+ AWS_PROFILE_NAME="${1:-${AWS_PROFILE:-sdlc}}"
6
+ STAGING_HEALTH_URL="${2:-https://apigateway.staging.clipboardhealth.org/api/healthCheck}"
7
+
8
+ passed=0
9
+ failed=0
10
+
11
+ print_pass() {
12
+ printf 'PASS: %s\n' "$1"
13
+ passed=$((passed + 1))
14
+ }
15
+
16
+ print_fail() {
17
+ printf 'FAIL: %s\n' "$1"
18
+ failed=$((failed + 1))
19
+ }
20
+
21
+ print_failure_details() {
22
+ local command_text="$1"
23
+ local exit_code="$2"
24
+ local output="$3"
25
+
26
+ printf ' Command: %s\n' "$command_text"
27
+ printf ' Exit: %s\n' "$exit_code"
28
+ if [[ -n "$output" ]]; then
29
+ printf ' Output:\n'
30
+ printf '%s\n' "$output" | sed -n '1,6p' | sed 's/^/ /'
31
+ fi
32
+ }
33
+
34
+ printf 'Checking flaky deep-dive prerequisites...\n'
35
+
36
+ timeout_command=""
37
+ if command -v gtimeout >/dev/null 2>&1; then
38
+ timeout_command="gtimeout"
39
+ elif command -v timeout >/dev/null 2>&1; then
40
+ timeout_command="timeout"
41
+ else
42
+ print_fail "No timeout utility is installed."
43
+ printf ' Remediation: install GNU coreutils so remote credential checks fail within a bounded time.\n'
44
+ fi
45
+
46
+ if ! command -v pup >/dev/null 2>&1; then
47
+ print_fail "pup is not installed."
48
+ printf ' Remediation: install pup, then authenticate with pup auth login or provide DD_API_KEY and DD_APP_KEY.\n'
49
+ elif [[ -n "$timeout_command" ]]; then
50
+ trace_output="$(
51
+ "$timeout_command" 30 pup traces search \
52
+ --query='service:cbh-backend-main env:staging' \
53
+ --from=15m \
54
+ --limit=1 \
55
+ --read-only \
56
+ --no-agent \
57
+ --jq='.data | length' 2>&1
58
+ )"
59
+ trace_exit_code=$?
60
+ if [[ "$trace_exit_code" -eq 0 ]]; then
61
+ print_pass "Datadog APM read access works through pup."
62
+ else
63
+ print_fail "pup cannot read staging APM spans."
64
+ print_failure_details \
65
+ "pup traces search --query='service:cbh-backend-main env:staging' --from=15m --limit=1" \
66
+ "$trace_exit_code" \
67
+ "$trace_output"
68
+ printf ' Remediation: run pup auth login, provide DD_API_KEY and DD_APP_KEY, or request APM read access for 401/403 failures.\n'
69
+ fi
70
+
71
+ logs_output="$(
72
+ "$timeout_command" 30 pup logs search \
73
+ --query='service:cbh-backend-main env:staging' \
74
+ --from=15m \
75
+ --limit=1 \
76
+ --read-only \
77
+ --no-agent \
78
+ --jq='.data | length' 2>&1
79
+ )"
80
+ logs_exit_code=$?
81
+ if [[ "$logs_exit_code" -eq 0 ]]; then
82
+ print_pass "Datadog log read access works through pup."
83
+ else
84
+ print_fail "pup cannot read staging logs."
85
+ print_failure_details \
86
+ "pup logs search --query='service:cbh-backend-main env:staging' --from=15m --limit=1" \
87
+ "$logs_exit_code" \
88
+ "$logs_output"
89
+ printf ' Remediation: run pup auth login, provide DD_API_KEY and DD_APP_KEY, or request log read access for 401/403 failures.\n'
90
+ fi
91
+ fi
92
+
93
+ if ! command -v curl >/dev/null 2>&1; then
94
+ print_fail "curl is not installed."
95
+ printf ' Remediation: install curl and retry while connected to the non-production VPN.\n'
96
+ else
97
+ staging_output="$(
98
+ curl \
99
+ --silent \
100
+ --show-error \
101
+ --write-out '\n%{http_code}' \
102
+ --connect-timeout 10 \
103
+ --max-time 20 \
104
+ "$STAGING_HEALTH_URL" 2>&1
105
+ )"
106
+ curl_exit_code=$?
107
+ staging_status="$(printf '%s\n' "$staging_output" | tail -n 1)"
108
+ staging_body="$(printf '%s\n' "$staging_output" | sed '$d')"
109
+
110
+ if [[ "$curl_exit_code" -eq 0 && "$staging_status" == "200" ]] &&
111
+ printf '%s\n' "$staging_body" | grep -Eq '"message"[[:space:]]*:[[:space:]]*"OK"'; then
112
+ print_pass "Staging health is reachable through the VPN (HTTP 200, message OK)."
113
+ else
114
+ print_fail "Staging health is not reachable through the VPN."
115
+ print_failure_details \
116
+ "curl --connect-timeout 10 --max-time 20 $STAGING_HEALTH_URL" \
117
+ "$curl_exit_code" \
118
+ "$staging_output"
119
+ printf ' Remediation: connect the Clipboard Health non-production VPN. If this environment cannot use the VPN, request a VPN-capable investigation environment.\n'
120
+ fi
121
+ fi
122
+
123
+ if ! command -v aws >/dev/null 2>&1; then
124
+ print_fail "AWS CLI is not installed."
125
+ printf ' Remediation: install AWS CLI v2 and configure the sdlc SSO profile.\n'
126
+ else
127
+ aws_identity_output="$(
128
+ aws sts get-caller-identity \
129
+ --profile "$AWS_PROFILE_NAME" \
130
+ --query Account \
131
+ --output text \
132
+ --cli-connect-timeout 10 \
133
+ --cli-read-timeout 20 2>&1
134
+ )"
135
+ aws_identity_exit_code=$?
136
+
137
+ if [[ "$aws_identity_exit_code" -eq 0 && -n "$aws_identity_output" && "$aws_identity_output" != "None" ]]; then
138
+ print_pass "AWS credentials are active for profile $AWS_PROFILE_NAME (account $aws_identity_output)."
139
+
140
+ aws_read_output="$(
141
+ aws cloudwatch list-metrics \
142
+ --profile "$AWS_PROFILE_NAME" \
143
+ --region us-west-2 \
144
+ --max-items 1 \
145
+ --output json \
146
+ --cli-connect-timeout 10 \
147
+ --cli-read-timeout 20 2>&1
148
+ )"
149
+ aws_read_exit_code=$?
150
+ if [[ "$aws_read_exit_code" -eq 0 ]]; then
151
+ print_pass "AWS read access works for profile $AWS_PROFILE_NAME."
152
+ else
153
+ print_fail "AWS credentials lack the staging read access required by the deep-dive track."
154
+ print_failure_details \
155
+ "aws cloudwatch list-metrics --profile $AWS_PROFILE_NAME --region us-west-2 --max-items 1" \
156
+ "$aws_read_exit_code" \
157
+ "$aws_read_output"
158
+ printf ' Remediation: request staging read permissions, then rerun the preflight.\n'
159
+ fi
160
+ else
161
+ print_fail "AWS credentials are unavailable for profile $AWS_PROFILE_NAME."
162
+ print_failure_details \
163
+ "aws sts get-caller-identity --profile $AWS_PROFILE_NAME" \
164
+ "$aws_identity_exit_code" \
165
+ "$aws_identity_output"
166
+ printf ' Remediation: run aws sso login --profile %s. If the profile is missing, request staging read access and configure it first.\n' "$AWS_PROFILE_NAME"
167
+ fi
168
+ fi
169
+
170
+ printf '\nSummary: %s passed, %s failed.\n' "$passed" "$failed"
171
+
172
+ if [[ "$failed" -gt 0 ]]; then
173
+ printf '\nDo not proceed with the deep dive. File a provisioning ticket linked to the chronic flake ticket and include the failed checks above.\n'
174
+ exit 1
175
+ fi
176
+
177
+ printf 'All credential preconditions are satisfied.\n'
@@ -68,11 +68,13 @@ Do not merge clusters merely because failures share a run. Same-run failures can
68
68
  4. **Multi-ticket cluster** โ€” pick the canonical (richest evidence: most failures, clearest artifacts). Comment the cluster manifest on it: member IDs, shared mechanism, evidence tier from Phase 3, and the marker `<!-- flaky-triage: cluster=<short-slug> -->`. Close the other members as Duplicate of the canonical (D2 format). Release the canonical.
69
69
  5. **Singleton or uncertain** โ€” release as-is with a brief `<!-- flaky-triage: singleton -->` comment.
70
70
 
71
+ Before any action above releases a canonical ticket, search Linear for the fingerprint family with the `flaky-implementation` label across all states and without a date-window restriction, then count unique prior implementation ticket IDs. At **โ‰ฅ3 prior implementation tickets**, do not release the canonical for normal dispatch and leave it in `Triage` unassigned. Create the Linear `chronic` label first if it does not exist. Apply the `chronic` label and route the family to the credential-checked, dossier-first `flaky-deep-dive` skill. Add a ticket comment with the marker `<!-- flaky-triage: chronic -->` and dossier pointers listing every prior implementation ticket ID โ€” but first search the ticket's existing comments for that exact marker and skip posting when it is already present; that absence check is what makes the comment idempotent across retries and re-runs. This โ‰ฅ3-ticket count is a cheap screen staged before any diagnosis exists; the critic separately applies a sharper **โ‰ฅ2 failed merged fixes** test (see the flaky-critic rubric's Prior-attempts amendment) with the same chronic routing. The thresholds differ by design โ€” by the evidence available at each stage โ€” not by accident.
72
+
71
73
  **Release** = set state to `Todo` and assign to the Linear API user (the viewer). Before release, ensure the ticket satisfies the Groundcrew eligibility contract from `../create-groundcrew-ticket/SKILL.md`: repository text in the description, exactly one `agent-*` label, leaf issue, and no non-terminal blockers unless intentionally blocked. Assignment + Todo is what makes groundcrew dispatch it; it also serves as the lease โ€” released tickets never re-enter the queue.
72
74
 
73
75
  ## Phase 5: Digest
74
76
 
75
- End with a compact report: queue size, cluster count with members and evidence tiers, released / duplicated / canceled counts, in-flight matches found, and risks (possible false merges, missing evidence). When run as a recurring task, follow any re-arm instructions in the task description after the digest.
77
+ End with a compact report: queue size, cluster count with members and evidence tiers, released / duplicated / canceled / chronic-held counts, in-flight matches found, chronic detections with each canonical ticket and its prior implementation ticket IDs, and risks (possible false merges, missing evidence). When run as a recurring task, follow any re-arm instructions in the task description after the digest.
76
78
 
77
79
  ## Dry-run and backtest
78
80