@clipboard-health/ai-rules 2.38.1 → 2.38.3

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@clipboard-health/ai-rules",
-  "version": "2.38.1",
+  "version": "2.38.3",
   "description": "Pre-built AI agent rules for consistent coding standards.",
   "keywords": [
     "ai",

package/skills/cb-babysit/SKILL.md

@@ -19,9 +19,9 @@ Resolve bundled "./scripts" paths relative to SKILL.md.
 
 The skill uses two sentinels with visible footer lines.
 
-**Addressed sentinel**: `<sub>🤖 <code>cb-babysit:addressed v1 core@3.17.0</code></sub>`. Appended on its own line at the end of every reply the skill posts so re-runs know which threads and review-body comments are already handled. Dedupe also recognizes the legacy `babysit-pr:addressed v1` prefix from before this skill was renamed.
+**Addressed sentinel**: `<sub>🤖 <code>cb-babysit:addressed v1 core@3.17.2</code></sub>`. Appended on its own line at the end of every reply the skill posts so re-runs know which threads and review-body comments are already handled. Dedupe also recognizes the legacy `babysit-pr:addressed v1` prefix from before this skill was renamed.
 
-**Follow-up sentinel**: `<sub>🤖 <code>cb-babysit:followup v1 core@3.17.0</code></sub>`. Attached to replies that defer an out-of-scope comment as a tracked follow-up. The sentinel is additive: the post-reply scripts still append the `addressed` sentinel at the end.
+**Follow-up sentinel**: `<sub>🤖 <code>cb-babysit:followup v1 core@3.17.2</code></sub>`. Attached to replies that defer an out-of-scope comment as a tracked follow-up. The sentinel is additive: the post-reply scripts still append the `addressed` sentinel at the end.
 
 **Sentinel recency rules.** The script emits a per-thread `activityState` with three values. Step 6a owns the handling rules for each state.
 
@@ -247,7 +247,7 @@ Body templates (the script appends the `addressed` sentinel if missing):
 - **Agree**: `Addressed in <commit-url>. <one-line what-changed>.`
 - **Disagree**: `Leaving current behavior. <reasoning>.`
 - **Already fixed**: `Already handled by <commit-url-or-file:line>. <brief pointer>.`
-- **Defer**: `Out of scope for this PR; this looks like follow-up work rather than something introduced or required by this change. <one-line rationale or pointer if useful>.\n\n<sub>🤖 <code>cb-babysit:followup v1 core@3.17.0</code></sub>`
+- **Defer**: `Out of scope for this PR; this looks like follow-up work rather than something introduced or required by this change. <one-line rationale or pointer if useful>.\n\n<sub>🤖 <code>cb-babysit:followup v1 core@3.17.2</code></sub>`
 
 For Defer replies, include the follow-up sentinel on its own line as shown. The script will append the `addressed` sentinel after it on its own line, so the final body ends with the follow-up sentinel followed by a blank line followed by the `addressed` sentinel. `grep cb-babysit:followup` finds the deferral and `grep cb-babysit:addressed` still marks the thread handled for dedupe.
 
@@ -263,7 +263,7 @@ The PR-level summary should:
 
 - Group by source. Use `## Review-body findings` for step-7 work and `## Conversation-tab comments` for step-6b work. Omit a section if its list is empty.
 - Inside each section, group verdicts under **Agree / Disagree / Already fixed / Deferred (out of scope)** subheadings. Omit a subheading if its list is empty.
-- Under **Deferred (out of scope)**, list each deferred item as a bullet, followed on its own line by `<sub>🤖 <code>cb-babysit:followup v1 core@3.17.0</code></sub>` so grep catches them individually.
+- Under **Deferred (out of scope)**, list each deferred item as a bullet, followed on its own line by `<sub>🤖 <code>cb-babysit:followup v1 core@3.17.2</code></sub>` so grep catches them individually.
 - Include the commit URL for fixes.
 - End with a fenced fingerprint block listing every current fingerprint (addressed and deferred) one per line. Include both `reviewBodyComments[].fingerprint` (whole-body, one per automated review) and `activeIssueComments[].fingerprint` (per Conversation-tab comment). Future runs dedupe by matching these against `priorBabysitSentinels`.
 

package/skills/cb-babysit/scripts/_sentinel.sh

@@ -10,7 +10,7 @@
 
 SENTINEL_PREFIX='cb-babysit:addressed v1 '
 LEGACY_SENTINEL_PREFIX='babysit-pr:addressed v1 '
-SENTINEL='<sub>🤖 <code>cb-babysit:addressed v1 core@3.17.0</code></sub>'
+SENTINEL='<sub>🤖 <code>cb-babysit:addressed v1 core@3.17.2</code></sub>'
 
 # Bot author allowlist (JSON array literal). Used by unresolvedPrComments.sh
 # as a fallback when GraphQL's `author.__typename == "Bot"` misses a GitHub

package/skills/cb-ship/references/pr-template.md

@@ -17,5 +17,5 @@ Concisely explain the user intent from session history and the meaningful behavi
 
 Optional, don't fabricate: ticket links, rollout plan, residual risk, or specific areas for reviewers to focus.
 
-<sub>🤖 <code>cb-ship:created v1 core@3.17.0</code></sub>
+<sub>🤖 <code>cb-ship:created v1 core@3.17.2</code></sub>
 ```

package/skills/flaky-critic/references/rubric.md

@@ -8,7 +8,7 @@ This rubric turns Rocky's observed review standards into checkable rules for fla
 
 Findings from automated reviewers (Mendral, CodeRabbit, etc.) are **advisory inputs, not binding rules**: the critic evaluates each on rubric merits and may decline one with stated reasoning, exactly as cb-babysit's Disagree replies do.
 
-**Statement-missing vs substance-missing (backtest amendment, 2026-06-11):** B5/B6/B7/B8 each require an explicit statement in the plan (dedup-check results, confidence score plus observability section, current-main status, causal-chain section). When the _statement_ is absent but the violation is not demonstrated — no actual duplicate found, no actual low-confidence code fix, no actual staleness, no demonstrably symptom-terminated diagnosis — the verdict is reject with disposition **amend-and-resubmit**: one bounce asking for exactly the missing sections, never escalation. A demonstrated violation (a real duplicate, a real ≤2/5 code fix, a diagnosis that demonstrably stops at a symptom) is a substantive reject. The 2026-06-11 backtest showed 6 of 7 false rejections were statement-missing on plans predating these conventions; live plans carry the sections by template, but the distinction stays load-bearing for the agreement metric.
+**Statement-missing vs substance-missing (backtest amendment, 2026-06-11):** B5/B6/B7/B8 each require an explicit statement in the plan (dedup-check results, Prior attempts section when applicable, confidence score plus observability section, current-main status, causal-chain section). When the _statement_ is absent but the violation is not demonstrated — no actual duplicate found, no actual prior implementation tickets for the fingerprint family, no actual low-confidence code fix, no actual staleness, no demonstrably symptom-terminated diagnosis — the verdict is reject with disposition **amend-and-resubmit**: one bounce asking for exactly the missing sections, never escalation. A demonstrated violation (a real duplicate, a mechanism already tried by a prior merged fix, a real ≤2/5 code fix, a diagnosis that demonstrably stops at a symptom) is a substantive reject. The 2026-06-11 backtest showed 6 of 7 false rejections were statement-missing on plans predating these conventions; live plans carry the sections by template, but the distinction stays load-bearing for the agreement metric.
 
 ## 2. Banned patterns
 
@@ -54,6 +54,8 @@ Opening one plan/PR per failing test when the failures share one mechanism (same
 
 **Prior-sibling signal (backtest amendment, 2026-06-11):** sibling PRs for the same mechanism that were closed _unmerged_ are a strong reject signal, not mere context. A plan citing closed-unmerged siblings must explain what is materially different this time — narrower scope alone is not an answer (the backtest's one genuine false approval cited prior wontfix siblings and explained them away).
 
+**Prior attempts (amendment, 2026-07-16):** when a fingerprint family has prior implementation tickets, the plan MUST contain a **Prior attempts** section listing each prior ticket/PR, what it blamed, what it changed, and the recurrence evidence showing its diagnosis was wrong or incomplete. A missing section is statement-missing and gets amend-and-resubmit per §1. A plan that re-proposes a mechanism a prior merged fix already tried is a substantive reject. A family with **≥2 failed merged fixes** — merged fixes followed by recurrence — is disqualified from the normal fix path: verdict = reject and escalate as needs-human. Apply the `chronic` label and hold the ticket for the deep-dive track (a dedicated dossier-first investigation stage; until it exists, chronic tickets stay held with the label for human routing). The ≥2 threshold is intentionally different from flaky-triage's ≥3-prior-implementation-tickets screen: triage counts tickets cheaply before any diagnosis exists, while the critic can see merge and recurrence outcomes and applies the sharper failed-merged-fix test. Either stage flagging chronic routes the same way; the counts are staged by the evidence available at each stage, not a contradiction.
+
 ### B6 — Code fix on low-confidence diagnosis
 
 Proposing a code change when root-cause confidence is ≤2/5, or any plan below 5/5 that omits observability follow-ups.

package/skills/flaky-debug/SKILL.md

@@ -3,7 +3,32 @@ name: flaky-debug
 description: Debug and fix flaky tests including Playwright E2E, NestJS service/integration, React component, and unit tests.
 ---
 
-Phases run in order. Skip a phase if you already have the information it produces. Phase 3 runs only in fix mode.
+Phases run in order. Phase 0 is mandatory. Skip a later phase if you already have the information it produces. Phase 3 runs only in fix mode.
+
+## Phase 0: Verify investigation access
+
+Run these checks before classifying or diagnosing the failure. Do not silently continue with reduced evidence if a check fails.
+
+1. Verify that `pup` can read APM spans. Do not use `pup auth status`; it can fail in sandboxes where environment-variable authentication works.
+
+   ```bash
+   command -v pup
+   pup traces search \
+     --query="service:cbh-backend-main env:staging" \
+     --from=15m \
+     --limit=1 \
+     --jq='.data | length'
+   ```
+
+   A successful query is sufficient even when the count is zero. If `pup` is missing, install it and rerun the check. For `401`/authentication failures, run `pup auth login` on a developer machine or provide `DD_API_KEY` and `DD_APP_KEY` in a sandbox/CI session. For `403`, obtain an application key or OAuth session with APM read access.
+
+2. For a CI-sourced E2E failure, verify artifact access by fetching the exact run before doing any diagnosis:
+
+   ```bash
+   bash "<flaky-debug-skill-dir>/scripts/fetch-llm-report.sh" "<github-actions-url>"
+   ```
+
+   If GitHub authentication fails, run `gh auth login --hostname github.com` and ensure the token can read Actions artifacts in the repository. If the artifact is missing or expired, rerun the source workflow or obtain the `playwright-llm-report`/Playwright HTML report from the user. A missing run URL or inaccessible artifact blocks a CI-sourced E2E investigation; report the failed command and remediation instead of degrading the confidence score. Artifact access is not applicable to service, component, or unit failures that were reproduced locally with complete output.
 
 ## Mode: plan vs fix
 
@@ -33,15 +58,28 @@ If the type is ambiguous, check the test file extension and imports to confirm.
 
 If the type is E2E, also classify where the failure surfaced in the lifecycle -- see [Classify the E2E Failure Surface](./references/plan-e2e.md#classify-the-e2e-failure-surface) in `plan-e2e.md`. The failure surface dictates how broadly to investigate before reading or editing the test.
 
-## Phase 1b: Check for Existing Fixes
+## Phase 1b: Build the Dossier and Check for Existing Fixes
+
+Before diagnosing, build the fingerprint family's dossier, then check whether someone (or another agent) has already fixed this flake. Do not limit the dossier to the failing file or recent activity.
+
+1. **Search Linear for the fingerprint family and exact test title.** Derive the fingerprint family from the ticket title or Flake details and the full test title from `testName`. Run these four `list_issues` queries, then union and deduplicate the results by ticket ID:
+
+   ```text
+   project: "Groundcrew", label: "flaky-investigation", query: "<fingerprint-family>"
+   project: "Groundcrew", label: "flaky-investigation", query: "<exact full test title>"
+   project: "Groundcrew", label: "flaky-implementation", query: "<fingerprint-family>"
+   project: "Groundcrew", label: "flaky-implementation", query: "<exact full test title>"
+   ```
 
-Before investigating, check whether someone (or another agent) has already fixed this flake.
+   For all four queries, omit the state filter so every state, including terminal states, is searched. Omit created/updated date filters; there is no 14-day restriction. The `query` field must search issue titles and descriptions, not the file name alone.
+   - Fetch every matching issue with `get_issue`, then read its relations, comments, and linked PRs so closed and merged attempts are not lost to truncated list results
 
-1. **Search open PRs with the `flaky-test-fix` label** that touch the failing test file or its surrounding code. Use GitHub search scoped to the repo:
+2. **Build the `Prior attempts` table before diagnosis.** Add one row for each prior implementation ticket and linked PR found through either query. The table columns are `Prior ticket/PR`, `What it blamed`, `What it changed`, and `Recurrence evidence`. The table must list each prior ticket/PR, what it blamed, what it changed, and the recurrence evidence showing its diagnosis was wrong or incomplete. Use later investigation sightings and post-merge failures as recurrence evidence; do not infer failure merely because an attempt is old. If there are no prior implementation tickets, record `None found` plus the fingerprint and test-title searches run.
+3. **Search open PRs with the `flaky-test-fix` label** that touch the failing test file or its surrounding code. Use GitHub search scoped to the repo:
    - Search PRs labeled `flaky-test-fix` for the test file name or test directory
    - Review the PR's changes to assess whether they address the same flake pattern with reasonable confidence — if so, stop and report it to the user rather than opening a duplicate fix
    - If the PR only partially addresses the flake or targets a different root cause, note it and proceed with investigation
-2. **Check recent commits on `main`** that touch the failing test file or its surrounding code:
+4. **Check recent commits on `main`** that touch the failing test file or its surrounding code:
    - `git log --oneline -20 origin/main -- <test-file-path>` and also check the parent directory or related source files
    - Read the commit messages — if one clearly fixes the same flake pattern, stop and report it to the user
 
@@ -50,6 +88,7 @@ If an existing fix is found, report:
 - The PR number/URL or commit hash
 - A brief summary of what it addresses
 - Whether it fully covers the current flake or only partially
+- The `Prior attempts` table, including recurrence evidence for any failed prior diagnosis
 
 If no existing fix is found, proceed to Phase 2.
 

package/skills/flaky-debug/references/datadog-apm-traces.md

@@ -1,84 +1,165 @@
 # Datadog APM Traces
 
-Fetch and display the full APM trace for a given trace ID, or look up a specific span by span ID.
+Correlate a Playwright network instance with backend spans in Datadog's **APM Spans index**. Use the per-request `attempts[].network.instances[].traceId`; do not use the test-level fixture annotation.
 
 ## Prerequisites
 
-The `pup` CLI must be installed and authenticated. Two auth paths are supported:
+Phase 0 of the flaky-debug skill must already have proved that `pup` can read APM spans and that the exact CI artifact is accessible. Do not run `pup auth status`; call `pup traces search` directly because environment-variable authentication can work in sandboxes where `pup auth status` does not.
 
-- **macOS Keychain** (via `pup auth login`) — the default on developer machines.
-- **Environment variables** (`DD_API_KEY` + `DD_APP_KEY`) — the path used in sandboxes and CI.
+If a query fails with `401`, run `pup auth login` or provide `DD_API_KEY` and `DD_APP_KEY`. For `403`, obtain APM read access. Do not continue an investigation without this access.
 
-Don't run `pup auth status` to verify auth. It fails in sandboxes even when env-var auth is working. Call `pup traces search …` directly. If the query fails with an auth error, check `DD_API_KEY` / `DD_APP_KEY` or run `pup auth login`.
+## Which trace ID is authoritative?
 
-## Key pup conventions
+Use these sources in order:
 
-- **Durations are in NANOSECONDS**: 1 second = 1,000,000,000 ns; 5ms = 5,000,000 ns. Convert to ms for display by dividing by 1,000,000.
-- **Default time range is 1h.** Always pass `--from` explicitly — use `--from=7d` or `--from=30d` for older traces.
-- **Default output is JSON.** Pipe JSON through `jq` for extraction.
-- **`--limit` defaults to 50.** Max is 1000. For large traces, you may need multiple paginated calls (but pup handles most pagination internally).
-- **Query syntax for traces:** `service:<name> resource_name:<path> @duration:>5s env:production status:error operation_name:<op>`
+1. The response `traceparent`, exposed as `network.instances[].traceId`. It identifies the context the backend actually selected.
+2. For an aborted or unreadable response, the request's `x-datadog-trace-id`. Reporter versions containing the Datadog fallback normalize its unsigned decimal value to a 32-character, zero-padded hexadecimal trace ID.
+3. The request `traceparent` only when no Datadog request context exists.
 
-## Steps
+Do **not** query the test-level `traceparent` annotation. The Playwright fixture sets a W3C header, but Datadog RUM also adds a different `x-datadog-*` context to first-party requests. The observed staging backend selects the Datadog context when the two conflict.
 
-### 1. If a span ID was provided, fetch that span first
+For reports produced by an older reporter, inspect the full Playwright trace's request headers. Convert `x-datadog-trace-id` as described below. If neither a response trace nor the Datadog request header is available, use the request-ID correlation path instead of claiming that the fixture ID represents the backend trace.
+
+## Known-good query recipe
+
+### 1. Set an exact time window
+
+Derive the request time from the attempt's `startTime` plus the network instance's `offsetMs`. Use a narrow absolute RFC 3339 window around the request; this avoids false matches and is safer than relying on `pup`'s one-hour default.
+
+```bash
+pup traces search \
+  --query='env:staging trace_id:<TRACE_ID>' \
+  --from='2026-07-06T14:10:00Z' \
+  --to='2026-07-06T14:20:00Z' \
+  --limit=1000
+```
+
+Start without a service filter when ownership is uncertain. Once the service is known, add it to reduce noise.
+
+### 2. Use the staging service name
+
+Staging APM service names are `cbh-*` prefixed. For example:
 
 ```bash
-pup traces search --query="span_id:<SPAN_ID>" --from=30d --limit=1
+pup traces search \
+  --query='env:staging service:cbh-backend-main trace_id:<TRACE_ID>' \
+  --from='<RFC3339_START>' \
+  --to='<RFC3339_END>' \
+  --limit=1000
 ```
 
-Display the span's details (service, operation, resource, duration, status, error if any) before proceeding to fetch the full trace.
+`service:cbh-backend-main` works; `service:backend-main` returns nothing. Other examples include `cbh-user`, `cbh-bg-jobs`, and `cbh-documents-service-backend`.
 
-If the query returns no results, tell the user the span was not found in the APM Spans index. Continue to step 2 using the trace ID from the arguments.
+### 3. Use a supported ID representation
 
-### 2. Fetch the full trace
+`pup traces search` accepts either:
 
-Use the `trace_id` to retrieve all spans in the trace:
+- the full 32-character hexadecimal trace ID returned by the spans API, or
+- the unsigned decimal value of the lower 64 bits.
+
+Do not convert the entire 128-bit value to decimal; the APM Spans search did not match that representation in the live verification.
+
+Convert a 32-character W3C/Datadog hexadecimal ID to its lower-64 decimal form:
 
 ```bash
-pup traces search --query="trace_id:<TRACE_ID>" --from=30d --limit=1000
+node -e '
+const id = process.argv[1].replace(/^0x/, "");
+if (!/^[0-9a-f]{32}$/i.test(id)) process.exit(1);
+console.log(BigInt(`0x${id.slice(-16)}`).toString(10));
+' '<TRACE_ID>'
 ```
 
-If the trace has more than 1000 spans, the response will be truncated. In that case, narrow the query by adding filters like `service:<name>` or `status:error` to focus on relevant spans.
-
-### 3. Parse and summarize the results
-
-The response JSON has this structure per span:
-
-```text
-.data[].attributes:
-  .span_id          — unique span identifier
-  .trace_id         — shared across all spans in the trace
-  .parent_id        — parent span (for building the call tree)
-  .service          — service name (e.g., "cbh-backend-main")
-  .operation_name   — operation (e.g., "express.request", "express.middleware", "http.request")
-  .resource_name    — resource (e.g., "GET /api/v1/users", "<anonymous>")
-  .status           — "ok" or "error"
-  .start_timestamp  — ISO 8601 start time
-  .end_timestamp    — ISO 8601 end time
-  .custom.duration  — duration in NANOSECONDS (divide by 1,000,000 for ms)
-  .custom.env       — environment (e.g., "staging", "production")
-  .custom.error     — error object with .message, .file, .fingerprint (null if no error)
-  .custom.type      — span type (e.g., "web", "http", "mongodb", "redis", "worker")
-  .custom.service   — service name (also at top level)
-  .tags[]           — array of tag strings
+Normalize an `x-datadog-trace-id` decimal value to the reporter's 32-character hexadecimal form:
+
+```bash
+node -e '
+const value = BigInt(process.argv[1]);
+if (value <= 0n || value > 0xffffffffffffffffn) process.exit(1);
+console.log(value.toString(16).padStart(32, "0"));
+' '<X_DATADOG_TRACE_ID>'
 ```
 
-Use `jq` to extract a useful summary. Example:
+### 4. Query a span ID only as a secondary lookup
+
+```bash
+pup traces search \
+  --query='env:staging span_id:<SPAN_ID>' \
+  --from='<RFC3339_START>' \
+  --to='<RFC3339_END>' \
+  --limit=1
+```
+
+A response `traceparent` contains the backend span ID. A request fallback contains the upstream parent span ID, which may not itself be an indexed backend span; the trace ID remains authoritative.
+
+### 5. Summarize relevant spans
+
+Durations are nanoseconds. Divide by 1,000,000 for milliseconds.
 
 ```bash
-# Quick error summary
-pup traces search --query="trace_id:<TRACE_ID>" --from=30d --limit=1000 \
-  | jq '[.data[] | select(.attributes.custom.error) | {
-      span_id: .attributes.span_id,
-      service: .attributes.service,
-      operation: .attributes.operation_name,
-      resource: .attributes.resource_name,
-      duration_ms: ((.attributes.custom.duration // 0) / 1000000 | . * 100 | round / 100),
-      error: .attributes.custom.error.message
+pup traces search \
+  --query='env:staging trace_id:<TRACE_ID>' \
+  --from='<RFC3339_START>' \
+  --to='<RFC3339_END>' \
+  --limit=1000 \
+  | jq '[.data[].attributes | {
+      trace_id,
+      span_id,
+      parent_id,
+      service,
+      operation: .operation_name,
+      resource: .resource_name,
+      status,
+      retained_by,
+      duration_ms: ((.custom.duration // 0) / 1000000 | . * 100 | round / 100),
+      error: .custom.error.message
     }]'
 ```
 
-### 4. Query additional data
+## Retention limitation and fallback correlation
+
+`pup traces search` searches indexed spans, not every ingested span. An ingested trace is available in Live Search for only 15 minutes unless a retention mechanism indexes it. A zero-result indexed query after that window does **not** prove that propagation or ingestion failed.
+
+If no indexed span exists:
+
+1. Record the exact query, environment, service variants, ID representations, and absolute time window tried.
+2. Search logs in the same window using the API Gateway/request ID, exact method/path, and service. The reporter may expose `requestId`; setup diagnostics may expose `apiGatewayRequestId`.
+3. Correlate the response status/body and the owning service from the report. Do not lower confidence merely because a success trace was not selected for retention; lower it only when a necessary causal link remains unproved.
+4. For a new occurrence, run the APM lookup within 15 minutes or add an explicit retention strategy for test traffic before relying on historical APM lookup.
+
+Example log search shape:
+
+```bash
+pup logs search \
+  --query='env:staging "<API_GATEWAY_REQUEST_ID>"' \
+  --from='<RFC3339_START>' \
+  --to='<RFC3339_END>' \
+  --limit=100
+```
+
+## Audit proof: why the fixture ID cannot be the bridge
+
+The STAFF-1792 audit used the failed attempt from `cbh-admin-frontend` CI run `29337068333` (`2026-07-14T13:35:30Z`) and its raw Playwright trace:
+
+- The request carried fixture W3C context `00-0df9d80ac12e173dfcfc55aa29379ba6-5819a315ecd12173-01`.
+- Datadog RUM added `x-datadog-trace-id: 13639781014258445719`, `x-datadog-parent-id: 7268017665183408395`, origin `rum`, and sampling priority `1`.
+- The backend response carried `00-0000000000000000bd4a3c90d8fded97-277d492d139029e1-01`. `bd4a3c90d8fded97` is exactly the hexadecimal form of the request's decimal `x-datadog-trace-id`, not the fixture trace ID.
+
+This proves header propagation reached the backend, but the backend selected the Datadog RUM context. Minting a differently shaped W3C fixture ID cannot fix that conflict by itself. The reporter therefore uses the response trace and, when the response is unreadable, the Datadog request context.
+
+The same audit checked backend and Datadog configuration:
+
+- `clipboard-health/src/tracer.ts` calls `tracer.init({ logLevel: "error" })` without a propagation-style override; the locked `dd-trace` version is `5.27.1`.
+- No customer APM head-sampling rules were returned for `cbh-backend-main`/staging, and adaptive sampling was not onboarded.
+- The retention-filter API returned `401` for the available OAuth session, so the audit could not enumerate organization retention filters and does not infer that none exist. Indexed sample spans reported `retained_by: diversity_sampling`.
+
+Historical indexed proof comes from STAFF-1678. Its backend response trace ID `00000000000000001732fff1044f982e` converts to decimal `1671679822332401710`; this exact query returned retained `cbh-user` spans for `POST /api/user/create`:
+
+```bash
+pup traces search \
+  --query='trace_id:1671679822332401710 env:staging service:cbh-user resource_name:"POST /api/user/create"' \
+  --from='2026-07-06T14:10:00Z' \
+  --to='2026-07-06T14:20:00Z' \
+  --limit=5
+```
 
-If additional data would help diagnose the issue (e.g. logs, rum, cicd), use the pup CLI.
+Those spans reported `retained_by: diversity_sampling`. This is the working alternative correlation path: use the backend-selected per-request trace ID when it was retained; otherwise correlate by API Gateway/request ID plus exact time, method, path, and service.

package/skills/flaky-debug/references/plan-e2e.md

@@ -136,6 +136,8 @@ Scan `attempts[].network.instances[]` for 4xx/5xx responses near the failure's `
 
 **`traceId`** — when present on a failing instance (`attempts[].network.instances[].traceId`), you must follow [`./datadog-apm-traces.md`](./datadog-apm-traces.md) to correlate with backend behavior. This is the bridge between frontend test failure and potential backend root cause.
 
+Use the per-request `network.instances[].traceId`, not the test-level `traceparent` annotation. Datadog RUM adds `x-datadog-*` headers to first-party requests, and the staging backend selects that context when it conflicts with the fixture's W3C `traceparent`. The reporter prefers the response `traceparent`; for an unreadable/aborted response it normalizes the request's `x-datadog-trace-id` to the same zero-padded hexadecimal form. Older reporter versions may fall back to the fixture ID instead; inspect the full Playwright trace for `x-datadog-trace-id` or use the request-ID/time-window correlation path in `datadog-apm-traces.md`.
+
 If no expected request was emitted, say that explicitly and do not diagnose backend latency for that action. Instead, use the trace, console/page errors, session/permission state, and frontend code path to explain why the request never started.
 
 Check `attempts[].network.summary` for saturation. Non-zero `instancesDroppedByGroupCap`, `instancesDroppedByInstanceCap`, or `instancesEvictedAfterAdmission` means retained content is a sample and the request you care about may have been dropped — note this as a confidence-reducing factor. `instancesDroppedByFilter` alone is expected (static assets are filtered by design). v3 caps: instances 500, groups 200, bodies 100 per attempt.

package/skills/flaky-debug/references/plan.md

@@ -94,6 +94,7 @@ Produce the plan with these fields:
 - **Confidence:** score (1-5) with brief justification
 - **Failure surface:** CI/job setup, test setup/auth/data, app bootstrap, user action no-op, backend request, post-success render, assertion/locator, or mixed
 - **Current main status:** whether the failing commit's code path still exists on current `main`, has already been fixed, or has changed enough that the plan must be adjusted
+- **Prior attempts:** list each prior ticket/PR, what it blamed, what it changed, and the recurrence evidence showing its diagnosis was wrong or incomplete. Use a table with `Prior ticket/PR`, `What it blamed`, `What it changed`, and `Recurrence evidence` columns. If the dossier search found no prior implementation tickets, write `None found` and include the fingerprint-family and exact-test-title searches run.
 - **Symptom:** what failed and where
 - **Root cause:** concise technical explanation
 - **Causal chain:** each link from failing assertion to terminal cause with its evidence, or the explicit break point and the observability that would extend it (see [Causal Chain](#causal-chain))

package/skills/flaky-debug/references/root-cause-kb/README.md

@@ -0,0 +1,44 @@
+# Flaky Root-Cause Knowledge Base
+
+Use this index to match a failure signature to a previously diagnosed mechanism before opening a new implementation path. Entries are indexed by mechanism, never by test. A test name or file is evidence for a mechanism, not the identity of an entry.
+
+When adding an entry, keep the same sections used by the seeded entries:
+
+1. Symptom signatures
+2. Mechanism
+3. Affected repositories and surfaces
+4. What fixed it
+5. What failed and why
+6. Current status
+7. Evidence
+
+## Symptom index
+
+| Symptom signature or fingerprint                                                                                                             | Mechanism cue                                                                                     | Entry                                                                                     |
+| -------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
+| `element was detached from the DOM`, `element is not stable`, open dialog or sheet disappears during a mutation/refetch                      | Overlay is rendered below a query-driven mapped row that is reconciled or removed                 | [Query-driven list dialog teardown](./query-driven-list-dialog-teardown.md)               |
+| Dialog title or sheet content never becomes stably visible after it opened                                                                   | The owning list item remounted while the overlay was open                                         | [Query-driven list dialog teardown](./query-driven-list-dialog-teardown.md)               |
+| Fingerprints `33deef731a10`, `2d93cd67b48a`                                                                                                  | Home Health instance of query-driven overlay ownership                                            | [Query-driven list dialog teardown](./query-driven-list-dialog-teardown.md)               |
+| `Cognito user '<email>' does not exist`, `UserNotFoundException` immediately after a successful user-create response                         | Consumer is rediscovering an opaque Cognito username through an eventually consistent email alias | [Cognito email-alias eventual consistency](./cognito-email-alias-eventual-consistency.md) |
+| Email-based Cognito Admin mutation fails while the same user exists under `from_admin_<uuid>`                                                | Strong username identity was discarded at the producer boundary                                   | [Cognito email-alias eventual consistency](./cognito-email-alias-eventual-consistency.md) |
+| `TooManyRequestsException`, `MAKE_TEST_TOKEN: Too many requests`, clustered Cognito setup `401`/`403` across shards                          | Parallel workers exceed the shared Cognito Admin API budget                                       | [Cognito API throttling](./cognito-api-throttling.md)                                     |
+| `AdminUpdateUserAttributes` throttle storm, especially from token setup                                                                      | Stateful Cognito write is on a per-test or per-token hot path                                     | [Cognito API throttling](./cognito-api-throttling.md)                                     |
+| Fresh entity creation succeeds, then consuming UI/API reports `Invalid workplace`, `Worker <id> not found`, opaque `500`, or readiness `422` | CDC/read-model ingestion has not reached the consuming path                                       | [CDC and read-model readiness](./cdc-read-model-readiness.md)                             |
+| Fixed CDC sleep passes usually but fails under load                                                                                          | Time delay is standing in for a missing readiness signal                                          | [CDC and read-model readiness](./cdc-read-model-readiness.md)                             |
+| `ChunkLoadError`, `Failed to fetch dynamically imported module`, lazy route never renders, downstream locator timeout after JS asset `5xx`   | A lazy import became permanently rejected after a transient asset failure                         | [Lazy-chunk load recovery](./lazy-chunk-load-recovery.md)                                 |
+| Many test-specific lazy-route fix PRs for different pages in the same incident                                                               | Shared product loader lacks centralized recovery                                                  | [Lazy-chunk load recovery](./lazy-chunk-load-recovery.md)                                 |
+| Hashed JS/CSS asset returns CloudFront `503 LimitExceeded`; retries replay the same error for about 10 seconds                               | CloudFront error caching amplifies a short S3-origin burst                                        | [CDN 503 amplification](./cdn-503-amplification.md)                                       |
+| Cold-cache deploy stampede followed by widespread static-asset failures                                                                      | Hashed assets lack immutable caching and edge errors are pinned                                   | [CDN 503 amplification](./cdn-503-amplification.md)                                       |
+| `LaunchDarkly initialization failed after N attempts`, app/service fails before tests start                                                  | Feature-flag bootstrap is a hard startup dependency                                               | [Feature-flag and third-party bootstrap](./feature-flag-and-third-party-bootstrap.md)     |
+| Popup/new tab bypasses LaunchDarkly, analytics, monitoring, Maps, or other mocks                                                             | Mocks were installed on one page instead of the browser context                                   | [Feature-flag and third-party bootstrap](./feature-flag-and-third-party-bootstrap.md)     |
+| Stripe/Maps/vendor SDK request fails, hangs, or escapes mocks before readiness UI renders                                                    | Third-party bootstrap is nondeterministic or isolated at the wrong browser scope                  | [Feature-flag and third-party bootstrap](./feature-flag-and-third-party-bootstrap.md)     |
+| Component test hangs after `userEvent.click/type`, especially with fake timers                                                               | Async `user-event` work is not awaited or timers cannot advance                                   | [user-event and fake-timer timing](./user-event-and-fake-timer-timing.md)                 |
+| Test passes but later throws `window is not defined`, act warning, or post-teardown timer error                                              | Pending debounced/async work escaped teardown                                                     | [user-event and fake-timer timing](./user-event-and-fake-timer-timing.md)                 |
+| Failures appeared during `@testing-library/user-event` v13 to v14 or Vitest migration                                                        | Previously synchronous assumptions became explicit promises and timer integration requirements    | [user-event and fake-timer timing](./user-event-and-fake-timer-timing.md)                 |
+
+## Lookup discipline
+
+- Match the mechanism using the failure surface and causal evidence, not only a shared error string.
+- Read the candidate entry's failed-attempt section before proposing a fix.
+- If the evidence contradicts the entry, continue the investigation and add a new mechanism only when the causal terminus is genuinely different.
+- Update an existing entry when a new repository or test exposes the same mechanism.

package/skills/flaky-debug/references/root-cause-kb/cdc-read-model-readiness.md

@@ -0,0 +1,51 @@
+# CDC and Read-Model Readiness
+
+Last reviewed: 2026-07-16.
+
+## Symptom signatures
+
+- A create API returns success, then the UI's consuming API reports `Invalid workplace`, `Worker <id> not found`, missing offer/estimate data, or an opaque `500`.
+- A fresh shift, offer, worker, or workplace is visible in the source service but not the downstream read model.
+- A fixed sleep usually works but fails during load or an environment incident.
+- An outer retry re-creates fresh entities on every attempt and never lets one entity finish propagating.
+
+## Mechanism
+
+The test creates data in a source-of-truth service and immediately exercises a UI path backed by another service or read model. CDC, event ingestion, or asynchronous projection has not made the new entity visible to the exact consuming path yet.
+
+Readiness must be defined by the API/state the UI will consume. A successful producer response, an arbitrary sleep, or readiness in a different store does not prove the user path is ready.
+
+## Affected repositories and surfaces
+
+- `cbh-admin-frontend`: fresh shift offers, rate negotiation, Daily View, and other seeded E2E flows.
+- `clipboard-health` and downstream services such as curated shifts: producer-to-read-model ingestion boundaries.
+- Any test helper that creates cross-service data and then navigates immediately to a consumer.
+
+## What fixed it
+
+- Poll the same consuming API or deterministic state used by the UI.
+- Bound the wait and fail with the last status, body, entity IDs, trace/request metadata, and attempt count.
+- Retry only classified readiness failures; keep genuine validation errors fast-fail.
+- Reuse or extract one shared per-repository readiness helper for the mechanism.
+- Keep the same entity across readiness attempts. Do not re-seed and reset the CDC clock.
+
+[STAFF-1010](https://linear.app/clipboardhealth/issue/STAFF-1010) established the accepted pattern: poll the same offer-estimate path used by the rate-negotiation dialog instead of extending a fixed CDC sleep. [cbh-admin-frontend#6780](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6780) applied the shared bounded helper to shift-offer setup.
+
+## What failed and why
+
+- Increasing the fixed CDC sleep encoded a latency guess and provided no useful exhaustion evidence.
+- The outer HCF retry created a new workplace/worker/shift each time; every retry restarted ingestion instead of waiting for the first entity.
+- Matching only a detailed readiness message failed when the downstream service logged the specific cause but returned an opaque `500` to the client.
+- Retrying all `422` responses in [cbh-admin-frontend#6974](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6974) would have made real validation errors wait 90 seconds. Review narrowed the implementation so opaque `500`s retry while unrelated `422`s still fail fast.
+- Adding a new per-spec readiness loop duplicates policy and lets timeout/classification behavior drift. Extend the shared helper instead.
+
+## Current status
+
+Known recurring mechanism with a documented A1 shared-helper rule in the flaky-fix rubric. The mechanism is not globally “fixed” because new producer/consumer boundaries continue to appear; the reusable resolution is a consuming-path readiness gate with classified failures and shared ownership.
+
+## Evidence
+
+- [STAFF-1010](https://linear.app/clipboardhealth/issue/STAFF-1010): canonical accepted plan for fresh shift-offer readiness.
+- [cbh-admin-frontend#6780](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6780): bounded shared shift-offer readiness helper.
+- [cbh-admin-frontend#6792](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6792): JSON:API readiness-message parsing used by the helper lineage.
+- [cbh-admin-frontend#6974](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6974): opaque `500` classification and reviewer-enforced `422` fast-fail behavior.

package/skills/flaky-debug/references/root-cause-kb/cdn-503-amplification.md

@@ -0,0 +1,50 @@
+# CDN 503 Amplification for Hashed Assets
+
+Last reviewed: 2026-07-16.
+
+## Symptom signatures
+
+- Hashed JavaScript or CSS assets return CloudFront `503`, often `LimitExceeded`.
+- A short S3 website-origin error burst becomes about ten seconds of identical edge failures.
+- Client import retries and one reload all receive the same cached error.
+- Failures cluster immediately after deploy when E2E begins against a cold CloudFront cache.
+- Many routes fail together because their chunks share the same distribution/origin.
+
+## Mechanism
+
+Deploy-time cold-cache demand creates a request stampede against the S3 website endpoint. A brief origin `5xx` is cached by CloudFront's default error TTL, so every retry replays the edge-cached error after the origin may already have recovered. Hashed assets without immutable cache headers also miss an opportunity to stay resident in browsers and at the edge.
+
+The CDN turns a short origin blip into a wider, longer outage. Client recovery is necessary but cannot succeed while the error itself is pinned.
+
+## Affected repositories and surfaces
+
+- `cbh-admin-frontend`: frontend deploy action and CloudFront/S3 Terraform.
+- `cbh-mobile-app`: HCP webapp CloudFront/S3 Terraform.
+- Staging E2E and real browser route loads that fetch content-hashed assets through those distributions.
+
+## What fixed it
+
+- Set `error_caching_min_ttl = 0` for transient `500`, `502`, `503`, and `504` responses.
+- Upload content-hashed JavaScript/CSS with `Cache-Control: public, max-age=31536000, immutable`.
+- Preserve short/default caching for stable-named runtime/configuration files.
+- Add per-status CloudFront metrics and access logs so the next burst can distinguish origin latency, cache behavior, and specific error codes.
+
+The infrastructure fixes landed in [cbh-admin-frontend#7010](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7010) and [cbh-mobile-app#12288](https://github.com/ClipboardHealth/cbh-mobile-app/pull/12288).
+
+## What failed and why
+
+- The central `loadLazily` retry ladder ran three imports and one reload within the period where CloudFront served the same cached error. Correct client recovery was defeated by edge policy.
+- E2E preflight and deployed-asset graph checks from [cbh-admin-frontend#5942](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/5942), [#5957](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/5957), and [#5958](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/5958) improved detection and reduced canary pressure but did not remove error pinning.
+- Selective invalidation in [cbh-admin-frontend#5840](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/5840) reduced deploy churn but did not change the transient-error TTL.
+- Adding more test retries would increase the cold-cache stampede and continue replaying the cached `503`.
+
+## Current status
+
+Fixed in Terraform for both frontend distributions. Origin failover remains deliberately deferred; add it only if measured bursts still fail after error caching is disabled. The log-forwarding/pipeline follow-up is operational observability, not required to understand the mechanism.
+
+## Evidence
+
+- [STAFF-1470](https://linear.app/clipboardhealth/issue/STAFF-1470): cross-repository CDN amplification work.
+- [cbh-admin-frontend#7010](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7010): error TTL, immutable hashed assets, metrics, and access logs.
+- [cbh-mobile-app#12288](https://github.com/ClipboardHealth/cbh-mobile-app/pull/12288): mobile distribution port of the infrastructure fix.
+- [cbh-mobile-app#11898](https://github.com/ClipboardHealth/cbh-mobile-app/pull/11898): deployed-asset preflight and the linked root-cause analysis used by both Terraform fixes.

package/skills/flaky-debug/references/root-cause-kb/cognito-api-throttling.md

@@ -0,0 +1,51 @@
+# Cognito API Throttling Under Parallel Setup Load
+
+Last reviewed: 2026-07-16.
+
+## Symptom signatures
+
+- `TooManyRequestsException` from `AdminUpdateUserAttributes`, `MakeTestToken`, login, or token-mint setup.
+- `MAKE_TEST_TOKEN: Too many requests`.
+- HCF or auth setup produces clustered `401`/`403` failures across shards while unrelated UI assertions never run.
+- Failures correlate with worker/shard fan-out and pass on retry after the burst subsides.
+
+## Mechanism
+
+Parallel Playwright workers and overlapping CI runs share one staging Cognito pool and API quota. Stateful writes or token creation placed on per-test/per-token hot paths multiply across workers. In-process caches do not coordinate separate Node processes, so each worker independently repeats the same setup work.
+
+Retries that do not reduce concurrency or call volume amplify the burst. The terminal cause is shared-capacity pressure, not the individual test that happened to request a token when the quota was exhausted.
+
+## Affected repositories and surfaces
+
+- `cbh-admin-frontend`: Playwright global setup, token minting, shared admin user types, dynamic worker/HCF setup.
+- `cbh-mobile-app`: shared credential acquisition and admin-auth token caches.
+- Any parallel test harness using the same Cognito pool or backend token endpoint without cross-process coordination.
+
+## What fixed it
+
+- Remove stateful Cognito writes from per-token paths and provision invariants at creation/global-setup boundaries.
+- Deduplicate remaining writes by identity and requested attributes.
+- Serialize or cap Cognito Admin/token operations across processes, not only within one worker.
+- Cache shared tokens with a conservative TTL.
+- Cap Playwright worker fan-out when the shared environment cannot support the theoretical concurrency.
+- Preserve structured status/body/request IDs so a throttle is not misdiagnosed as an auth or UI failure.
+
+[cbh-admin-frontend#6697](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6697) removed per-token user-type writes; [cbh-admin-frontend#6731](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6731) added cross-process slots, token caching, diagnostics, and a worker cap.
+
+## What failed and why
+
+- Frontend waits and Playwright retries did not change Cognito call volume. Repeating setup could create more token and attribute operations and worsen the throttle.
+- Per-process throttles/caches were insufficient because each Playwright worker is a separate process.
+- Retrying every auth/setup failure without classification hid real `401`/`403` contract errors and still amplified rate-limited operations.
+- Treating the Upcoming Charges test as the owner was a failure-surface mistake: the observed error occurred in shared auth setup before the page loaded.
+
+## Current status
+
+Mitigated for the known admin staging setup paths. Cross-process coordination and removal of hot-path writes are the reusable fix class. Re-open the capacity question only after measuring call volume with these controls; quota increases or a dedicated E2E pool are secondary options, not the first fix.
+
+## Evidence
+
+- [STAFF-1122](https://linear.app/clipboardhealth/issue/STAFF-1122): canonical investigation; CloudTrail evidence included 980 throttled `AdminUpdateUserAttributes` calls.
+- [cbh-admin-frontend#6697](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6697): removed per-token Cognito writes and deduplicated remaining setup mutations.
+- [STAFF-1169](https://linear.app/clipboardhealth/issue/STAFF-1169): follow-up for broader staging auth fan-out.
+- [cbh-admin-frontend#6731](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6731): shared cross-process throttling, caching, worker cap, and diagnostics.

package/skills/flaky-debug/references/root-cause-kb/cognito-email-alias-eventual-consistency.md

@@ -0,0 +1,50 @@
+# Cognito Email-Alias Eventual Consistency
+
+Last reviewed: 2026-07-16.
+
+## Symptom signatures
+
+- `Cognito user '<email>' does not exist in pool '<pool>'` immediately after the user-create API returned success.
+- `AdminSetUserPassword` or `AdminUpdateUserAttributes` raises `UserNotFoundException` when called with an email.
+- A bounded retry sometimes passes under light load and fails as a storm under overlapping CI runs.
+- Direct lookup by the opaque username succeeds while lookup or mutation by email still fails.
+
+## Mechanism
+
+Backend user creation stores Cognito users under opaque usernames such as `from_admin_<uuid>` and stores the email as an attribute/alias. When the producer discards that username, the test harness must rediscover the identity through Cognito's email-alias index. That lookup path is eventually consistent; the strongly consistent username path is already available but was not returned across the API contract.
+
+This mechanism is distinct from Cognito throttling. Throttling can lengthen the propagation tail, but a `UserNotFoundException` from email-based resolution after a proven successful create is an identity-contract mismatch.
+
+## Affected repositories and surfaces
+
+- `clipboard-health`: workplace/facility user creation and the backend-main contract.
+- `cbh-admin-frontend`: Playwright password and user-type provisioning for freshly created synthetic users.
+- Other consumers that create a Cognito identity through backend-main and then address Cognito Admin APIs by email.
+
+## What fixed it
+
+- Return the opaque Cognito username from the user-creation response and contract.
+- Use that username for subsequent Cognito Admin mutations instead of resolving by email.
+- Fail loudly or emit explicit telemetry when Cognito provisioning fails, rather than returning an apparently usable user with no identity.
+- Keep a readiness probe only as a migration guard for producer paths that cannot yet return the username.
+
+The producer-side contract fix landed in [clipboard-health#26862](https://github.com/ClipboardHealth/clipboard-health/pull/26862) for [STAFF-1793](https://linear.app/clipboardhealth/issue/STAFF-1793).
+
+## What failed and why
+
+- Five `UserNotFoundException` retries spaced two seconds apart guessed that email propagation would finish within ten seconds. The guess usually held, then failed under load.
+- The 75–90 second readiness implementation in [cbh-admin-frontend#7380](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7380) was useful diagnostic armor but remained consumer-side: it still paid the eventually consistent email-resolution cost and added Cognito Admin calls.
+- Increasing the wait cannot fix a swallowed producer failure where no Cognito identity was created. The producer must distinguish absence from propagation.
+- Treating the whole storm as throttling alone would preserve the incorrect email-based identity contract.
+
+## Current status
+
+The workplace-user producer contract now exposes the strongly consistent username. Consumer paths should delete email-resolution probes as they adopt the contract. Keep this entry active until all fresh-user setup paths, including worker/HCP variants, use a returned username or another deterministic identity.
+
+## Evidence
+
+- [STAFF-1667](https://linear.app/clipboardhealth/issue/STAFF-1667): storm family with six tests and the shared `Cognito user does not exist` signature.
+- [STAFF-1672](https://linear.app/clipboardhealth/issue/STAFF-1672): consumer-side readiness design.
+- [cbh-admin-frontend#7380](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7380): closed readiness implementation and its bounded-wait diagnostics.
+- [STAFF-1793](https://linear.app/clipboardhealth/issue/STAFF-1793): producer-side root fix.
+- [clipboard-health#26862](https://github.com/ClipboardHealth/clipboard-health/pull/26862): returned username, fail-loud behavior, cleanup, and metrics.

package/skills/flaky-debug/references/root-cause-kb/feature-flag-and-third-party-bootstrap.md

@@ -0,0 +1,55 @@
+# Feature-Flag and Third-Party Bootstrap Nondeterminism
+
+Last reviewed: 2026-07-16.
+
+## Symptom signatures
+
+- `LaunchDarkly initialization failed after N attempts` aborts application or service startup.
+- A vendor SDK/API request for feature flags, Stripe, Maps, analytics, or monitoring fails or hangs before expected UI renders.
+- A popup/new tab makes real third-party requests even though the original page was mocked.
+- CORS errors or default flag values are followed by unauthorized or wrong-route API calls.
+- The final test failure is a missing button/locator, but the first divergence is an external bootstrap request.
+
+## Mechanism
+
+Application readiness depends on third-party initialization whose availability, page scope, or completion timing is nondeterministic.
+
+There are two important subtypes:
+
+1. A true runtime dependency such as LaunchDarkly may intentionally be hard. Failing open is safe only when every flag has a verified safe default and the client will recover correctly.
+2. E2E-only external dependencies should be isolated at the right scope. Page-scoped mocks do not apply to a popup/new tab, and route navigation is not ready merely because the DOM content loaded event fired.
+
+## Affected repositories and surfaces
+
+- `clipboard-health`: NestJS feature-flag provider startup.
+- `cbh-admin-frontend`: facility onboarding popups, LaunchDarkly/analytics/monitoring/Places dependencies.
+- `cbh-mobile-app`: vendor-backed onboarding and post-auth route bootstraps when the external integration is the earliest failing dependency.
+- Any browser test that opens another page or relies on external scripts before rendering route readiness UI.
+
+## What fixed it
+
+- Install E2E mocks on `browserContext.route()` when every page/tab in the context must inherit them.
+- Register popup/new-page waits before the click, then wait for a product readiness signal on the new page.
+- For vendor bootstraps, wait on an explicit SDK or product readiness signal and report the failed external request.
+- Mock third-party behavior when it is not the system under test, while preserving the real contract shape.
+- Decide LaunchDarkly degraded behavior explicitly from product safety, not from test convenience.
+
+[cbh-admin-frontend#5989](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/5989) moved feature-flag and third-party mocks to the browser context for popup coverage.
+
+## What failed and why
+
+- The blanket LaunchDarkly fail-open proposal in [clipboard-health#26748](https://github.com/ClipboardHealth/clipboard-health/pull/26748) was closed. Review established that defaults are not reliably safe and an initialized client might not recover in the intended way, so making LaunchDarkly optional could produce a broader incorrect-state outage.
+- Page-scoped `page.route()` mocks passed on the original page but leaked real LaunchDarkly and third-party requests from a newly opened tab.
+- Waiting only for navigation or the DOM content loaded event treated the HTML document as application readiness; a vendor SDK or flag bootstrap could still prevent React from rendering.
+- Longer locator timeouts hid the external request that failed and provided no bootstrap diagnostics.
+- Mocking away the third party is wrong when that integration is the behavior under test; use contract-faithful mocks only for dependencies outside the scenario.
+- Do not classify every Stripe- or Maps-named test as this mechanism. [STAFF-1486](https://linear.app/clipboardhealth/issue/STAFF-1486) and [cbh-mobile-app#12309](https://github.com/ClipboardHealth/cbh-mobile-app/pull/12309) were static-asset `503` failures on a Stripe route, so they belong to the lazy-chunk/CDN entries.
+
+## Current status
+
+Mixed by dependency. Browser-context mock scope and explicit vendor readiness signals are the established test patterns. LaunchDarkly remains an intentional hard dependency in backend-main until safe defaults and recovery semantics are proven; do not cite this KB entry as permission to swallow its initialization errors.
+
+## Evidence
+
+- [cbh-admin-frontend#5989](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/5989): popup failure caused by page-scoped LaunchDarkly/third-party mocks.
+- [clipboard-health#26748](https://github.com/ClipboardHealth/clipboard-health/pull/26748): rejected LaunchDarkly fail-open attempt and reviewer rationale.

package/skills/flaky-debug/references/root-cause-kb/lazy-chunk-load-recovery.md

@@ -0,0 +1,51 @@
+# Lazy-Chunk Load Failures Need Product-Side Recovery
+
+Last reviewed: 2026-07-16.
+
+## Symptom signatures
+
+- `ChunkLoadError`.
+- `Failed to fetch dynamically imported module`.
+- A JavaScript asset request returns `5xx`, then a lazy route never renders.
+- Playwright times out on a page locator even though the earliest failure is a static chunk request.
+- Multiple routes/tests fail with different selectors during the same asset incident.
+
+## Mechanism
+
+A transient static-asset failure rejects a dynamic import. The lazy loader or route remains stuck behind the rejected import/error boundary even after the origin recovers. Downstream locator failures are secondary because the route's code never loaded.
+
+The recovery belongs in the shared product loader: classify chunk-load failures and perform one guarded reload for the affected app version, commit, page, and session. The test may add diagnostics, but it should not be the only recovery layer.
+
+## Affected repositories and surfaces
+
+- `cbh-admin-frontend`: React lazy routes through the central `loadLazily` wrapper.
+- `cbh-mobile-app`: Playwright route-bootstrap and static-asset recovery helpers; these protect tests, not real users.
+- Any deployed SPA using content-hashed chunks that may briefly fail during deploy/origin/CDN incidents.
+
+## What fixed it
+
+- Centralize recognition of browser chunk-load/dynamic-import failures.
+- Retry the import as appropriate, then reload at most once using a session guard scoped to app version, commit, and route.
+- Fall through to the normal error boundary after the bounded recovery is exhausted.
+- Log whether recovery was attempted, skipped by the guard, or exhausted.
+
+[cbh-admin-frontend#5650](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/5650) is the canonical product fix. Mobile later added Playwright-only static-asset route-bootstrap armor in [cbh-mobile-app#11958](https://github.com/ClipboardHealth/cbh-mobile-app/pull/11958) and reused it on the Stripe-named asset failure in [cbh-mobile-app#12309](https://github.com/ClipboardHealth/cbh-mobile-app/pull/12309).
+
+## What failed and why
+
+- Five sibling PRs ([#5645](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/5645), [#5646](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/5646), [#5647](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/5647), [#5648](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/5648), and [#5649](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/5649)) treated individual routes/tests as separate fixes. They were consolidated because the mechanism was shared.
+- Per-test route-ready retries and chunk monitors can improve diagnostics but leave real users stuck when the lazy loader itself has no recovery.
+- Unbounded reloads can create a loop during a persistent deploy or asset mismatch. Recovery must be one-time and version/route scoped.
+- Loader recovery alone could not defeat a CDN that cached the same `503` for every retry; that is the separate [CDN amplification mechanism](./cdn-503-amplification.md).
+
+## Current status
+
+Product-side recovery is merged for the known admin loader. Mobile's cited recovery is test-harness-only, so a real-user mobile web failure still needs a product-side loader diagnosis. Future per-route sightings should first verify whether the surface has product recovery, harness armor, or neither, and whether the asset failure is transient. Do not create one implementation per test.
+
+## Evidence
+
+- [cbh-admin-frontend#5645](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/5645): representative closed per-surface attempt that mixed product and E2E recovery.
+- [cbh-admin-frontend#5650](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/5650): merged centralized lazy-chunk recovery.
+- [cbh-admin-frontend#5492](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/5492): earlier chunk-load logging and user-facing refresh behavior.
+- [cbh-mobile-app#11958](https://github.com/ClipboardHealth/cbh-mobile-app/pull/11958): Playwright app-bootstrap static-asset retry helper.
+- [cbh-mobile-app#12309](https://github.com/ClipboardHealth/cbh-mobile-app/pull/12309): Stripe-route sighting whose actual cause was CloudFront/static assets, not Stripe bootstrap.

package/skills/flaky-debug/references/root-cause-kb/query-driven-list-dialog-teardown.md

@@ -0,0 +1,52 @@
+# Query-Driven List Dialog Teardown
+
+Last reviewed: 2026-07-16.
+
+## Symptom signatures
+
+- Playwright reports `element is not stable` followed by `element was detached from the DOM`.
+- A dialog or sheet opens, then its title/content disappears or never becomes stably visible.
+- The failure moves between actions on different rows of the same query-backed list.
+- A mutation succeeds, but the confirmation interaction or post-success assertion loses the overlay.
+- Known families: `33deef731a10` and `2d93cd67b48a`.
+
+## Mechanism
+
+The overlay is owned by a component rendered beneath a mapped query result, such as `cases.map(...<CaseCard>)` or `visits.map(...<VisitCard>)`. Background, focus, interval, or mutation-triggered query refreshes reconcile, reorder, or remove the row while the overlay is open. Because the dialog or sheet lifecycle is tied to that row, the overlay remounts or unmounts during interaction.
+
+The decisive test is structural: if deleting or replacing the launching row also destroys the open overlay, the product has this mechanism even when the visible failure is a locator timeout.
+
+## Affected repositories and surfaces
+
+- `cbh-admin-frontend`: Home Health case and visit action dialogs; other query-driven list dialogs found by the repository sweep.
+- `cbh-mobile-app`: placement sign-on-bonus and badge-history sheets; workplace-review comment and reply action sheets.
+- Any React surface that renders modal state and overlay JSX inside a query-backed `.map()` descendant.
+
+## What fixed it
+
+- Hoist overlay ownership above the mapped list into one stable, entity-keyed host. Rows become trigger-only and pass entity identity to the host.
+- Add regression coverage that opens the overlay, removes or refreshes the launching row, and proves the overlay survives.
+- Add an architecture guard for direct dialog/modal/sheet JSX inside `.map()` callbacks, then perform a semantic sweep for indirect descendants the syntax guard cannot see.
+- Split oversized sequential E2E flows after the product fix so one mechanism does not surface as many indistinguishable steps.
+
+The durable Home Health fix is [cbh-admin-frontend#7574](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7574). Repository-wide follow-ups landed in [cbh-admin-frontend#7569](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7569), [cbh-mobile-app#12857](https://github.com/ClipboardHealth/cbh-mobile-app/pull/12857), and [cbh-mobile-app#12860](https://github.com/ClipboardHealth/cbh-mobile-app/pull/12860).
+
+## What failed and why
+
+- Repeated `toPass` blocks, response waits, locator retries, failure trackers, and expanded timeouts hardened whichever Home Health step failed most recently but left overlay ownership unchanged. The family recurred across [STAFF-1329](https://linear.app/clipboardhealth/issue/STAFF-1329), [STAFF-1369](https://linear.app/clipboardhealth/issue/STAFF-1369), [STAFF-1495](https://linear.app/clipboardhealth/issue/STAFF-1495), [STAFF-1554](https://linear.app/clipboardhealth/issue/STAFF-1554), [STAFF-1582](https://linear.app/clipboardhealth/issue/STAFF-1582), [STAFF-1673](https://linear.app/clipboardhealth/issue/STAFF-1673), and [STAFF-1685](https://linear.app/clipboardhealth/issue/STAFF-1685).
+- Preserving modal state across a remount did not preserve the DOM node Playwright or a user was interacting with. The Team Members recurrence from [STAFF-1422](https://linear.app/clipboardhealth/issue/STAFF-1422) to [STAFF-1459](https://linear.app/clipboardhealth/issue/STAFF-1459) is the precedent.
+- Pausing backing-list observers and deferring invalidations in [cbh-admin-frontend#7559](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7559) reduced the race window but was explicitly an interim mitigation: another ancestor update could still remove the row. The hoisted host later deleted that workaround.
+- Splitting the E2E test alone would reduce blast radius and improve fingerprints, but it would not fix the user-facing overlay teardown.
+
+## Current status
+
+Fixed for the known admin and mobile surfaces covered by [STAFF-1789](https://linear.app/clipboardhealth/issue/STAFF-1789), [STAFF-1790](https://linear.app/clipboardhealth/issue/STAFF-1790), [STAFF-1796](https://linear.app/clipboardhealth/issue/STAFF-1796), [STAFF-1797](https://linear.app/clipboardhealth/issue/STAFF-1797), and [STAFF-1806](https://linear.app/clipboardhealth/issue/STAFF-1806). Treat future instances as the same mechanism when overlay ownership is still below a replaceable query row; extend the stable-host pattern and guard rather than adding per-test waits.
+
+## Evidence
+
+- [STAFF-1789](https://linear.app/clipboardhealth/issue/STAFF-1789): mitigation and fault-injection proof for Home Health list refreshes.
+- [STAFF-1790](https://linear.app/clipboardhealth/issue/STAFF-1790): durable host design and E2E decomposition.
+- [cbh-admin-frontend#7559](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7559): refetch guard, with residual ownership risk documented.
+- [cbh-admin-frontend#7574](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7574): hoisted host and deletion of the temporary guard.
+- [cbh-mobile-app#12857](https://github.com/ClipboardHealth/cbh-mobile-app/pull/12857): mobile sheet hoists and architecture guard.
+- [cbh-mobile-app#12860](https://github.com/ClipboardHealth/cbh-mobile-app/pull/12860): indirect workplace-review descendants hoisted after the syntactic guard could not detect them.

package/skills/flaky-debug/references/root-cause-kb/user-event-and-fake-timer-timing.md

@@ -0,0 +1,52 @@
+# user-event and Fake-Timer Component-Test Timing
+
+Last reviewed: 2026-07-16.
+
+## Symptom signatures
+
+- Component test hangs or times out after `userEvent.click`, `type`, `clear`, or keyboard input.
+- Assertions run before event-driven state updates complete.
+- Fake-timer suites stop progressing because `user-event` scheduled work cannot advance.
+- A test passes, then emits an act warning, unhandled rejection, or `window is not defined` after jsdom teardown.
+- Failures appear after a `user-event`, Vitest, React, or Testing Library upgrade exposes previously implicit timing.
+
+## Mechanism
+
+`@testing-library/user-event` v14 APIs are asynchronous and model multiple browser events. Tests written with v13-style synchronous calls can assert or tear down before those events finish. When fake timers are active, the configured `user-event` instance must be able to advance the runner's timers; otherwise its internal delays and the component's scheduled work deadlock.
+
+A related teardown class occurs when debounced component-library timers remain pending after the DOM environment is destroyed. Timer ownership and cleanup, not a larger timeout, determine correctness.
+
+## Affected repositories and surfaces
+
+- `cbh-mobile-app`: Jest and Vitest component tests migrated from `user-event` v13.5 to v14.
+- `cbh-admin-frontend`: component-library debounces and fake-timer teardown patterns.
+- Any component test mixing `user-event`, fake timers, MSW, debounced UI libraries, animations, or scheduled state updates.
+
+## What fixed it
+
+- Upgrade to `user-event` v14 and await every interaction.
+- Prefer one `const user = userEvent.setup(...)` instance per test.
+- In fake-timer suites, configure `advanceTimers` with the active runner, such as `vi.advanceTimersByTime` or `jest.advanceTimersByTime`.
+- Flush only the work the scenario owns, restore real timers in `afterEach`, and keep cleanup before jsdom teardown.
+- Enforce `testing-library/await-async-events` so new interactions that are not awaited fail lint.
+
+The repository-wide migration landed in [cbh-mobile-app#10310](https://github.com/ClipboardHealth/cbh-mobile-app/pull/10310).
+
+## What failed and why
+
+- [cbh-mobile-app#5565](https://github.com/ClipboardHealth/cbh-mobile-app/pull/5565) attempted a broad `fireEvent` to `userEvent` lint conversion before the full async/timer migration. It was closed with unresolved lint and test failures; changing the API name alone did not establish correct timing.
+- Static `userEvent.*` calls mixed with `userEvent.setup()` left inconsistent isolation and made missing awaits easy to reintroduce.
+- Adding per-test or global timeout increases, including the Vitest 4 migration's timeout accommodation in [cbh-mobile-app#11947](https://github.com/ClipboardHealth/cbh-mobile-app/pull/11947), can make slow suites pass but does not fix a deadlocked fake timer or interaction that was not awaited.
+- Fake timers alone are not sufficient. Without `advanceTimers`, `user-event` may wait forever; without teardown flushing/restoration, callbacks may fire after jsdom is gone.
+- The closed [cbh-admin-frontend#6699](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6699) correctly identified a pending MUI debounce but never landed, illustrating that a plausible timer diagnosis still needs a maintained, merged cleanup pattern.
+
+## Current status
+
+The mobile v14 migration and lint enforcement are merged. Treat new component-test timing failures as one of three explicit subtypes before editing: interaction not awaited, fake-timer integration, or pending teardown work. Do not default to timeout inflation.
+
+## Evidence
+
+- [cbh-mobile-app#10310](https://github.com/ClipboardHealth/cbh-mobile-app/pull/10310): v13.5 to v14 migration, approximately 1,300 awaited calls, fake-timer `advanceTimers` setup, and lint enforcement.
+- [cbh-mobile-app#5565](https://github.com/ClipboardHealth/cbh-mobile-app/pull/5565): incomplete earlier mechanical conversion.
+- [cbh-mobile-app#11947](https://github.com/ClipboardHealth/cbh-mobile-app/pull/11947): Vitest 4 migration showing remaining user-event/fake-timer/MSW timing pressure.
+- [cbh-admin-frontend#6699](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6699): post-teardown MUI debounce signature and proposed timer cleanup.

package/skills/flaky-triage/SKILL.md

@@ -68,11 +68,13 @@ Do not merge clusters merely because failures share a run. Same-run failures can
 4. **Multi-ticket cluster** — pick the canonical (richest evidence: most failures, clearest artifacts). Comment the cluster manifest on it: member IDs, shared mechanism, evidence tier from Phase 3, and the marker `<!-- flaky-triage: cluster=<short-slug> -->`. Close the other members as Duplicate of the canonical (D2 format). Release the canonical.
 5. **Singleton or uncertain** — release as-is with a brief `<!-- flaky-triage: singleton -->` comment.
 
+Before any action above releases a canonical ticket, search Linear for the fingerprint family with the `flaky-implementation` label across all states and without a date-window restriction, then count unique prior implementation ticket IDs. At **≥3 prior implementation tickets**, do not release the canonical for normal dispatch and leave it in `Triage` unassigned. Create the Linear `chronic` label first if it does not exist. Apply the `chronic` label and hold the ticket for the deep-dive track (a dedicated dossier-first investigation stage; until it exists, chronic tickets stay held with the label for human routing). Add a ticket comment with the marker `<!-- flaky-triage: chronic -->` and dossier pointers listing every prior implementation ticket ID — but first search the ticket's existing comments for that exact marker and skip posting when it is already present; that absence check is what makes the comment idempotent across retries and re-runs. This ≥3-ticket count is a cheap screen staged before any diagnosis exists; the critic separately applies a sharper **≥2 failed merged fixes** test (see the flaky-critic rubric's Prior-attempts amendment) with the same chronic routing. The thresholds differ by design — by the evidence available at each stage — not by accident.
+
 **Release** = set state to `Todo` and assign to the Linear API user (the viewer). Before release, ensure the ticket satisfies the Groundcrew eligibility contract from `../create-groundcrew-ticket/SKILL.md`: repository text in the description, exactly one `agent-*` label, leaf issue, and no non-terminal blockers unless intentionally blocked. Assignment + Todo is what makes groundcrew dispatch it; it also serves as the lease — released tickets never re-enter the queue.
 
 ## Phase 5: Digest
 
-End with a compact report: queue size, cluster count with members and evidence tiers, released / duplicated / canceled counts, in-flight matches found, and risks (possible false merges, missing evidence). When run as a recurring task, follow any re-arm instructions in the task description after the digest.
+End with a compact report: queue size, cluster count with members and evidence tiers, released / duplicated / canceled / chronic-held counts, in-flight matches found, chronic detections with each canonical ticket and its prior implementation ticket IDs, and risks (possible false merges, missing evidence). When run as a recurring task, follow any re-arm instructions in the task description after the digest.
 
 ## Dry-run and backtest