@clipboard-health/ai-rules 2.38.0 → 2.38.2

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@clipboard-health/ai-rules",
-  "version": "2.38.0",
+  "version": "2.38.2",
   "description": "Pre-built AI agent rules for consistent coding standards.",
   "keywords": [
     "ai",

package/scripts/toErrorMessage.js

@@ -1,7 +1,7 @@
 "use strict";
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.toErrorMessage = toErrorMessage;
-// Prefer stack over message for Error instances to preserve debugging context.
+// Prefer stack over message for Error instances to preserve debugging context!
 function toErrorMessage(error) {
     return error instanceof Error ? (error.stack ?? error.message) : String(error);
 }

package/skills/cb-babysit/SKILL.md

@@ -19,9 +19,9 @@ Resolve bundled "./scripts" paths relative to SKILL.md.
 
 The skill uses two sentinels with visible footer lines.
 
-**Addressed sentinel**: `<sub>🤖 <code>cb-babysit:addressed v1 core@3.17.0</code></sub>`. Appended on its own line at the end of every reply the skill posts so re-runs know which threads and review-body comments are already handled. Dedupe also recognizes the legacy `babysit-pr:addressed v1` prefix from before this skill was renamed.
+**Addressed sentinel**: `<sub>🤖 <code>cb-babysit:addressed v1 core@3.17.1</code></sub>`. Appended on its own line at the end of every reply the skill posts so re-runs know which threads and review-body comments are already handled. Dedupe also recognizes the legacy `babysit-pr:addressed v1` prefix from before this skill was renamed.
 
-**Follow-up sentinel**: `<sub>🤖 <code>cb-babysit:followup v1 core@3.17.0</code></sub>`. Attached to replies that defer an out-of-scope comment as a tracked follow-up. The sentinel is additive: the post-reply scripts still append the `addressed` sentinel at the end.
+**Follow-up sentinel**: `<sub>🤖 <code>cb-babysit:followup v1 core@3.17.1</code></sub>`. Attached to replies that defer an out-of-scope comment as a tracked follow-up. The sentinel is additive: the post-reply scripts still append the `addressed` sentinel at the end.
 
 **Sentinel recency rules.** The script emits a per-thread `activityState` with three values. Step 6a owns the handling rules for each state.
 
@@ -247,7 +247,7 @@ Body templates (the script appends the `addressed` sentinel if missing):
 - **Agree**: `Addressed in <commit-url>. <one-line what-changed>.`
 - **Disagree**: `Leaving current behavior. <reasoning>.`
 - **Already fixed**: `Already handled by <commit-url-or-file:line>. <brief pointer>.`
-- **Defer**: `Out of scope for this PR; this looks like follow-up work rather than something introduced or required by this change. <one-line rationale or pointer if useful>.\n\n<sub>🤖 <code>cb-babysit:followup v1 core@3.17.0</code></sub>`
+- **Defer**: `Out of scope for this PR; this looks like follow-up work rather than something introduced or required by this change. <one-line rationale or pointer if useful>.\n\n<sub>🤖 <code>cb-babysit:followup v1 core@3.17.1</code></sub>`
 
 For Defer replies, include the follow-up sentinel on its own line as shown. The script will append the `addressed` sentinel after it on its own line, so the final body ends with the follow-up sentinel followed by a blank line followed by the `addressed` sentinel. `grep cb-babysit:followup` finds the deferral and `grep cb-babysit:addressed` still marks the thread handled for dedupe.
 
@@ -263,7 +263,7 @@ The PR-level summary should:
 
 - Group by source. Use `## Review-body findings` for step-7 work and `## Conversation-tab comments` for step-6b work. Omit a section if its list is empty.
 - Inside each section, group verdicts under **Agree / Disagree / Already fixed / Deferred (out of scope)** subheadings. Omit a subheading if its list is empty.
-- Under **Deferred (out of scope)**, list each deferred item as a bullet, followed on its own line by `<sub>🤖 <code>cb-babysit:followup v1 core@3.17.0</code></sub>` so grep catches them individually.
+- Under **Deferred (out of scope)**, list each deferred item as a bullet, followed on its own line by `<sub>🤖 <code>cb-babysit:followup v1 core@3.17.1</code></sub>` so grep catches them individually.
 - Include the commit URL for fixes.
 - End with a fenced fingerprint block listing every current fingerprint (addressed and deferred) one per line. Include both `reviewBodyComments[].fingerprint` (whole-body, one per automated review) and `activeIssueComments[].fingerprint` (per Conversation-tab comment). Future runs dedupe by matching these against `priorBabysitSentinels`.
 

package/skills/cb-babysit/scripts/_sentinel.sh

@@ -10,7 +10,7 @@
 
 SENTINEL_PREFIX='cb-babysit:addressed v1 '
 LEGACY_SENTINEL_PREFIX='babysit-pr:addressed v1 '
-SENTINEL='<sub>🤖 <code>cb-babysit:addressed v1 core@3.17.0</code></sub>'
+SENTINEL='<sub>🤖 <code>cb-babysit:addressed v1 core@3.17.1</code></sub>'
 
 # Bot author allowlist (JSON array literal). Used by unresolvedPrComments.sh
 # as a fallback when GraphQL's `author.__typename == "Bot"` misses a GitHub

package/skills/cb-ship/references/pr-template.md

@@ -17,5 +17,5 @@ Concisely explain the user intent from session history and the meaningful behavi
 
 Optional, don't fabricate: ticket links, rollout plan, residual risk, or specific areas for reviewers to focus.
 
-<sub>🤖 <code>cb-ship:created v1 core@3.17.0</code></sub>
+<sub>🤖 <code>cb-ship:created v1 core@3.17.1</code></sub>
 ```

package/skills/flaky-critic/references/rubric.md

@@ -8,7 +8,7 @@ This rubric turns Rocky's observed review standards into checkable rules for fla
 
 Findings from automated reviewers (Mendral, CodeRabbit, etc.) are **advisory inputs, not binding rules**: the critic evaluates each on rubric merits and may decline one with stated reasoning, exactly as cb-babysit's Disagree replies do.
 
-**Statement-missing vs substance-missing (backtest amendment, 2026-06-11):** B5/B6/B7 each require an explicit statement in the plan (dedup-check results, confidence score plus observability section, current-main status). When the _statement_ is absent but the violation is not demonstrated — no actual duplicate found, no actual low-confidence code fix, no actual staleness — the verdict is reject with disposition **amend-and-resubmit**: one bounce asking for exactly the missing sections, never escalation. A demonstrated violation (a real duplicate, a real ≤2/5 code fix) is a substantive reject. The 2026-06-11 backtest showed 6 of 7 false rejections were statement-missing on plans predating these conventions; live plans carry the sections by template, but the distinction stays load-bearing for the agreement metric.
+**Statement-missing vs substance-missing (backtest amendment, 2026-06-11):** B5/B6/B7/B8 each require an explicit statement in the plan (dedup-check results, confidence score plus observability section, current-main status, causal-chain section). When the _statement_ is absent but the violation is not demonstrated — no actual duplicate found, no actual low-confidence code fix, no actual staleness, no demonstrably symptom-terminated diagnosis — the verdict is reject with disposition **amend-and-resubmit**: one bounce asking for exactly the missing sections, never escalation. A demonstrated violation (a real duplicate, a real ≤2/5 code fix, a diagnosis that demonstrably stops at a symptom) is a substantive reject. The 2026-06-11 backtest showed 6 of 7 false rejections were statement-missing on plans predating these conventions; live plans carry the sections by template, but the distinction stays load-bearing for the agreement metric.
 
 ## 2. Banned patterns
 
@@ -72,6 +72,21 @@ Implementing a plan against code that has changed on `main`, or for a flake alre
 
 **Staleness clause (backtest amendment, 2026-06-11):** a plan more than ~14 days old, or one whose fingerprint has new sightings since the plan was written, requires re-verification against current main before approval — treat unverified staleness as B7. (Two backtest "rejections" were plans that sat unworked until a sweep canceled them; freshness is part of a plan being gateable.)
 
+### B8 — Unterminated causal chain (symptom-level diagnosis)
+
+Proposing a code fix from a diagnosis that stops at the failure symptom — "the request 500ed/timed out," "setup was throttled," "the element never appeared" — without establishing _why_ at a specific cause in the owning code, config, or infrastructure.
+
+**Why (amendment, 2026-07-07):** post-Done recurrence is the pipeline's dominant failure mode: in the Jun 2–Jul 6 window, 21 fingerprint families re-flaked after a Done closure, and the fullLifecycle family (33deef731a10) was closed Done six times. Each recurrence traces to a diagnosis that dead-ended at the repo boundary or at a status code and patched the symptom locally. Local-symptom plans are the cheapest to produce, so as long as the gate accepts them, agents keep producing them; this rule makes boundary-crossing the only way to pass.
+
+**Required structure:** the plan must contain a **Causal chain** section: failing assertion → app/UI state → network/trace/log evidence (trace ID, or run-window log queries when the causal event is outside the test's request path: seeding, deploys, CDC/async jobs) → owning service and repo, resolved via the groundtruth ownership registry (`ClipboardHealth/groundtruth`, `registry/services.json` → `registry/repos.json`) → terminal cause cited at file/line, config key, or specific log line. The link types adapt to the failure surface: CI/setup, fixture, component, and unit failures substitute build logs, fixture state, or runner artifacts for network evidence and the owning package/repo for a deployed service — the critic checks the terminus, not a fixed link shape. Two valid terminal states:
+
+1. **Terminated:** the chain ends at a cause. A status code, timeout, or throttle is a link, never a terminus.
+2. **Explicitly broken:** the chain names the link where evidence ran out and the observability that would extend it; confidence is capped at 2/5 and B6 applies — the deliverable is the instrumentation, not a code fix.
+
+**Confidence ceiling:** 5/5 additionally requires reproduction by inducing the blamed cause — fault injection in the harness (delay or fail the specific response, disable the seed step) or a focused lower-level test that deterministically reproduces the race. Without reproduction, the ceiling is 4/5. One inferred _intermediate_ link with an evidenced terminus is still a terminated chain (≤4/5); an unevidenced terminus is a broken chain (cap 2/5).
+
+**Detection heuristic:** plan lacks a Causal chain section (statement-missing → amend-and-resubmit per §1); the terminal link is a symptom rather than a cause; the chain implicates another service or repo but cites no code/log evidence from it and doesn't route per C5/D5; or the plan claims 5/5 without reproduction evidence.
+
 ## 3. Allowed/safe patterns
 
 These have strong precedent (counts from the 74 merged `flaky-test-fix` PRs; classification approximate) and are auto-merge candidates **only** when the listed evidence is present in the PR body.

package/skills/flaky-debug/SKILL.md

@@ -3,7 +3,32 @@ name: flaky-debug
 description: Debug and fix flaky tests including Playwright E2E, NestJS service/integration, React component, and unit tests.
 ---
 
-Phases run in order. Skip a phase if you already have the information it produces. Phase 3 runs only in fix mode.
+Phases run in order. Phase 0 is mandatory. Skip a later phase if you already have the information it produces. Phase 3 runs only in fix mode.
+
+## Phase 0: Verify investigation access
+
+Run these checks before classifying or diagnosing the failure. Do not silently continue with reduced evidence if a check fails.
+
+1. Verify that `pup` can read APM spans. Do not use `pup auth status`; it can fail in sandboxes where environment-variable authentication works.
+
+   ```bash
+   command -v pup
+   pup traces search \
+     --query="service:cbh-backend-main env:staging" \
+     --from=15m \
+     --limit=1 \
+     --jq='.data | length'
+   ```
+
+   A successful query is sufficient even when the count is zero. If `pup` is missing, install it and rerun the check. For `401`/authentication failures, run `pup auth login` on a developer machine or provide `DD_API_KEY` and `DD_APP_KEY` in a sandbox/CI session. For `403`, obtain an application key or OAuth session with APM read access.
+
+2. For a CI-sourced E2E failure, verify artifact access by fetching the exact run before doing any diagnosis:
+
+   ```bash
+   bash "<flaky-debug-skill-dir>/scripts/fetch-llm-report.sh" "<github-actions-url>"
+   ```
+
+   If GitHub authentication fails, run `gh auth login --hostname github.com` and ensure the token can read Actions artifacts in the repository. If the artifact is missing or expired, rerun the source workflow or obtain the `playwright-llm-report`/Playwright HTML report from the user. A missing run URL or inaccessible artifact blocks a CI-sourced E2E investigation; report the failed command and remediation instead of degrading the confidence score. Artifact access is not applicable to service, component, or unit failures that were reproduced locally with complete output.
 
 ## Mode: plan vs fix
 

package/skills/flaky-debug/references/datadog-apm-traces.md

@@ -1,84 +1,165 @@
 # Datadog APM Traces
 
-Fetch and display the full APM trace for a given trace ID, or look up a specific span by span ID.
+Correlate a Playwright network instance with backend spans in Datadog's **APM Spans index**. Use the per-request `attempts[].network.instances[].traceId`; do not use the test-level fixture annotation.
 
 ## Prerequisites
 
-The `pup` CLI must be installed and authenticated. Two auth paths are supported:
+Phase 0 of the flaky-debug skill must already have proved that `pup` can read APM spans and that the exact CI artifact is accessible. Do not run `pup auth status`; call `pup traces search` directly because environment-variable authentication can work in sandboxes where `pup auth status` does not.
 
-- **macOS Keychain** (via `pup auth login`) — the default on developer machines.
-- **Environment variables** (`DD_API_KEY` + `DD_APP_KEY`) — the path used in sandboxes and CI.
+If a query fails with `401`, run `pup auth login` or provide `DD_API_KEY` and `DD_APP_KEY`. For `403`, obtain APM read access. Do not continue an investigation without this access.
 
-Don't run `pup auth status` to verify auth. It fails in sandboxes even when env-var auth is working. Call `pup traces search …` directly. If the query fails with an auth error, check `DD_API_KEY` / `DD_APP_KEY` or run `pup auth login`.
+## Which trace ID is authoritative?
 
-## Key pup conventions
+Use these sources in order:
 
-- **Durations are in NANOSECONDS**: 1 second = 1,000,000,000 ns; 5ms = 5,000,000 ns. Convert to ms for display by dividing by 1,000,000.
-- **Default time range is 1h.** Always pass `--from` explicitly — use `--from=7d` or `--from=30d` for older traces.
-- **Default output is JSON.** Pipe JSON through `jq` for extraction.
-- **`--limit` defaults to 50.** Max is 1000. For large traces, you may need multiple paginated calls (but pup handles most pagination internally).
-- **Query syntax for traces:** `service:<name> resource_name:<path> @duration:>5s env:production status:error operation_name:<op>`
+1. The response `traceparent`, exposed as `network.instances[].traceId`. It identifies the context the backend actually selected.
+2. For an aborted or unreadable response, the request's `x-datadog-trace-id`. Reporter versions containing the Datadog fallback normalize its unsigned decimal value to a 32-character, zero-padded hexadecimal trace ID.
+3. The request `traceparent` only when no Datadog request context exists.
 
-## Steps
+Do **not** query the test-level `traceparent` annotation. The Playwright fixture sets a W3C header, but Datadog RUM also adds a different `x-datadog-*` context to first-party requests. The observed staging backend selects the Datadog context when the two conflict.
 
-### 1. If a span ID was provided, fetch that span first
+For reports produced by an older reporter, inspect the full Playwright trace's request headers. Convert `x-datadog-trace-id` as described below. If neither a response trace nor the Datadog request header is available, use the request-ID correlation path instead of claiming that the fixture ID represents the backend trace.
+
+## Known-good query recipe
+
+### 1. Set an exact time window
+
+Derive the request time from the attempt's `startTime` plus the network instance's `offsetMs`. Use a narrow absolute RFC 3339 window around the request; this avoids false matches and is safer than relying on `pup`'s one-hour default.
+
+```bash
+pup traces search \
+  --query='env:staging trace_id:<TRACE_ID>' \
+  --from='2026-07-06T14:10:00Z' \
+  --to='2026-07-06T14:20:00Z' \
+  --limit=1000
+```
+
+Start without a service filter when ownership is uncertain. Once the service is known, add it to reduce noise.
+
+### 2. Use the staging service name
+
+Staging APM service names are `cbh-*` prefixed. For example:
 
 ```bash
-pup traces search --query="span_id:<SPAN_ID>" --from=30d --limit=1
+pup traces search \
+  --query='env:staging service:cbh-backend-main trace_id:<TRACE_ID>' \
+  --from='<RFC3339_START>' \
+  --to='<RFC3339_END>' \
+  --limit=1000
 ```
 
-Display the span's details (service, operation, resource, duration, status, error if any) before proceeding to fetch the full trace.
+`service:cbh-backend-main` works; `service:backend-main` returns nothing. Other examples include `cbh-user`, `cbh-bg-jobs`, and `cbh-documents-service-backend`.
 
-If the query returns no results, tell the user the span was not found in the APM Spans index. Continue to step 2 using the trace ID from the arguments.
+### 3. Use a supported ID representation
 
-### 2. Fetch the full trace
+`pup traces search` accepts either:
 
-Use the `trace_id` to retrieve all spans in the trace:
+- the full 32-character hexadecimal trace ID returned by the spans API, or
+- the unsigned decimal value of the lower 64 bits.
+
+Do not convert the entire 128-bit value to decimal; the APM Spans search did not match that representation in the live verification.
+
+Convert a 32-character W3C/Datadog hexadecimal ID to its lower-64 decimal form:
 
 ```bash
-pup traces search --query="trace_id:<TRACE_ID>" --from=30d --limit=1000
+node -e '
+const id = process.argv[1].replace(/^0x/, "");
+if (!/^[0-9a-f]{32}$/i.test(id)) process.exit(1);
+console.log(BigInt(`0x${id.slice(-16)}`).toString(10));
+' '<TRACE_ID>'
 ```
 
-If the trace has more than 1000 spans, the response will be truncated. In that case, narrow the query by adding filters like `service:<name>` or `status:error` to focus on relevant spans.
-
-### 3. Parse and summarize the results
-
-The response JSON has this structure per span:
-
-```text
-.data[].attributes:
-  .span_id          — unique span identifier
-  .trace_id         — shared across all spans in the trace
-  .parent_id        — parent span (for building the call tree)
-  .service          — service name (e.g., "cbh-backend-main")
-  .operation_name   — operation (e.g., "express.request", "express.middleware", "http.request")
-  .resource_name    — resource (e.g., "GET /api/v1/users", "<anonymous>")
-  .status           — "ok" or "error"
-  .start_timestamp  — ISO 8601 start time
-  .end_timestamp    — ISO 8601 end time
-  .custom.duration  — duration in NANOSECONDS (divide by 1,000,000 for ms)
-  .custom.env       — environment (e.g., "staging", "production")
-  .custom.error     — error object with .message, .file, .fingerprint (null if no error)
-  .custom.type      — span type (e.g., "web", "http", "mongodb", "redis", "worker")
-  .custom.service   — service name (also at top level)
-  .tags[]           — array of tag strings
+Normalize an `x-datadog-trace-id` decimal value to the reporter's 32-character hexadecimal form:
+
+```bash
+node -e '
+const value = BigInt(process.argv[1]);
+if (value <= 0n || value > 0xffffffffffffffffn) process.exit(1);
+console.log(value.toString(16).padStart(32, "0"));
+' '<X_DATADOG_TRACE_ID>'
 ```
 
-Use `jq` to extract a useful summary. Example:
+### 4. Query a span ID only as a secondary lookup
+
+```bash
+pup traces search \
+  --query='env:staging span_id:<SPAN_ID>' \
+  --from='<RFC3339_START>' \
+  --to='<RFC3339_END>' \
+  --limit=1
+```
+
+A response `traceparent` contains the backend span ID. A request fallback contains the upstream parent span ID, which may not itself be an indexed backend span; the trace ID remains authoritative.
+
+### 5. Summarize relevant spans
+
+Durations are nanoseconds. Divide by 1,000,000 for milliseconds.
 
 ```bash
-# Quick error summary
-pup traces search --query="trace_id:<TRACE_ID>" --from=30d --limit=1000 \
-  | jq '[.data[] | select(.attributes.custom.error) | {
-      span_id: .attributes.span_id,
-      service: .attributes.service,
-      operation: .attributes.operation_name,
-      resource: .attributes.resource_name,
-      duration_ms: ((.attributes.custom.duration // 0) / 1000000 | . * 100 | round / 100),
-      error: .attributes.custom.error.message
+pup traces search \
+  --query='env:staging trace_id:<TRACE_ID>' \
+  --from='<RFC3339_START>' \
+  --to='<RFC3339_END>' \
+  --limit=1000 \
+  | jq '[.data[].attributes | {
+      trace_id,
+      span_id,
+      parent_id,
+      service,
+      operation: .operation_name,
+      resource: .resource_name,
+      status,
+      retained_by,
+      duration_ms: ((.custom.duration // 0) / 1000000 | . * 100 | round / 100),
+      error: .custom.error.message
     }]'
 ```
 
-### 4. Query additional data
+## Retention limitation and fallback correlation
+
+`pup traces search` searches indexed spans, not every ingested span. An ingested trace is available in Live Search for only 15 minutes unless a retention mechanism indexes it. A zero-result indexed query after that window does **not** prove that propagation or ingestion failed.
+
+If no indexed span exists:
+
+1. Record the exact query, environment, service variants, ID representations, and absolute time window tried.
+2. Search logs in the same window using the API Gateway/request ID, exact method/path, and service. The reporter may expose `requestId`; setup diagnostics may expose `apiGatewayRequestId`.
+3. Correlate the response status/body and the owning service from the report. Do not lower confidence merely because a success trace was not selected for retention; lower it only when a necessary causal link remains unproved.
+4. For a new occurrence, run the APM lookup within 15 minutes or add an explicit retention strategy for test traffic before relying on historical APM lookup.
+
+Example log search shape:
+
+```bash
+pup logs search \
+  --query='env:staging "<API_GATEWAY_REQUEST_ID>"' \
+  --from='<RFC3339_START>' \
+  --to='<RFC3339_END>' \
+  --limit=100
+```
+
+## Audit proof: why the fixture ID cannot be the bridge
+
+The STAFF-1792 audit used the failed attempt from `cbh-admin-frontend` CI run `29337068333` (`2026-07-14T13:35:30Z`) and its raw Playwright trace:
+
+- The request carried fixture W3C context `00-0df9d80ac12e173dfcfc55aa29379ba6-5819a315ecd12173-01`.
+- Datadog RUM added `x-datadog-trace-id: 13639781014258445719`, `x-datadog-parent-id: 7268017665183408395`, origin `rum`, and sampling priority `1`.
+- The backend response carried `00-0000000000000000bd4a3c90d8fded97-277d492d139029e1-01`. `bd4a3c90d8fded97` is exactly the hexadecimal form of the request's decimal `x-datadog-trace-id`, not the fixture trace ID.
+
+This proves header propagation reached the backend, but the backend selected the Datadog RUM context. Minting a differently shaped W3C fixture ID cannot fix that conflict by itself. The reporter therefore uses the response trace and, when the response is unreadable, the Datadog request context.
+
+The same audit checked backend and Datadog configuration:
+
+- `clipboard-health/src/tracer.ts` calls `tracer.init({ logLevel: "error" })` without a propagation-style override; the locked `dd-trace` version is `5.27.1`.
+- No customer APM head-sampling rules were returned for `cbh-backend-main`/staging, and adaptive sampling was not onboarded.
+- The retention-filter API returned `401` for the available OAuth session, so the audit could not enumerate organization retention filters and does not infer that none exist. Indexed sample spans reported `retained_by: diversity_sampling`.
+
+Historical indexed proof comes from STAFF-1678. Its backend response trace ID `00000000000000001732fff1044f982e` converts to decimal `1671679822332401710`; this exact query returned retained `cbh-user` spans for `POST /api/user/create`:
+
+```bash
+pup traces search \
+  --query='trace_id:1671679822332401710 env:staging service:cbh-user resource_name:"POST /api/user/create"' \
+  --from='2026-07-06T14:10:00Z' \
+  --to='2026-07-06T14:20:00Z' \
+  --limit=5
+```
 
-If additional data would help diagnose the issue (e.g. logs, rum, cicd), use the pup CLI.
+Those spans reported `retained_by: diversity_sampling`. This is the working alternative correlation path: use the backend-selected per-request trace ID when it was retained; otherwise correlate by API Gateway/request ID plus exact time, method, path, and service.

package/skills/flaky-debug/references/plan-e2e.md

@@ -136,6 +136,8 @@ Scan `attempts[].network.instances[]` for 4xx/5xx responses near the failure's `
 
 **`traceId`** — when present on a failing instance (`attempts[].network.instances[].traceId`), you must follow [`./datadog-apm-traces.md`](./datadog-apm-traces.md) to correlate with backend behavior. This is the bridge between frontend test failure and potential backend root cause.
 
+Use the per-request `network.instances[].traceId`, not the test-level `traceparent` annotation. Datadog RUM adds `x-datadog-*` headers to first-party requests, and the staging backend selects that context when it conflicts with the fixture's W3C `traceparent`. The reporter prefers the response `traceparent`; for an unreadable/aborted response it normalizes the request's `x-datadog-trace-id` to the same zero-padded hexadecimal form. Older reporter versions may fall back to the fixture ID instead; inspect the full Playwright trace for `x-datadog-trace-id` or use the request-ID/time-window correlation path in `datadog-apm-traces.md`.
+
 If no expected request was emitted, say that explicitly and do not diagnose backend latency for that action. Instead, use the trace, console/page errors, session/permission state, and frontend code path to explain why the request never started.
 
 Check `attempts[].network.summary` for saturation. Non-zero `instancesDroppedByGroupCap`, `instancesDroppedByInstanceCap`, or `instancesEvictedAfterAdmission` means retained content is a sample and the request you care about may have been dropped — note this as a confidence-reducing factor. `instancesDroppedByFilter` alone is expected (static assets are filtered by design). v3 caps: instances 500, groups 200, bodies 100 per attempt.

package/skills/flaky-debug/references/plan.md

@@ -6,20 +6,32 @@ Shared tail of the flaky-debug planning phase, used by both [`plan-e2e.md`](./pl
 
 Rate your confidence in the root cause on a 1-5 scale. Report this score alongside your evidence.
 
-| Score | Meaning             | Criteria                                                                                                                                                                                       |
-| ----- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| **5** | Certain             | Root cause is directly visible in artifacts (e.g., assertion diff shows stale data, network response confirms 5xx, screenshot shows error banner)                                              |
-| **4** | High confidence     | Evidence strongly supports the diagnosis but one link in the chain is inferred rather than observed (e.g., timeline shows the right sequence but no Datadog trace to confirm backend behavior) |
-| **3** | Moderate confidence | Evidence is consistent with the diagnosis but alternative explanations remain plausible. Flag the alternatives explicitly                                                                      |
-| **2** | Low confidence      | Limited evidence, mostly reasoning from code patterns rather than observed artifacts. Recommend gathering more data before committing to a fix                                                 |
-| **1** | Speculative         | No direct evidence for the root cause. The fix is a best guess. Recommend reproducing the failure locally or adding instrumentation before proceeding                                          |
+| Score | Meaning             | Criteria                                                                                                                                                                                                                                                           |
+| ----- | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| **5** | Certain             | Root cause is directly visible in artifacts AND reproduced by inducing it — fault injection in the harness (delay/fail the blamed response or step) or a focused lower-level test that deterministically triggers the race. Without reproduction, the ceiling is 4 |
+| **4** | High confidence     | The chain terminates at an evidenced cause (file/line, config key, or log line) but one intermediate link is inferred rather than observed. An unevidenced terminal cause is a broken chain, capped at 2 (see [Causal Chain](#causal-chain))                       |
+| **3** | Moderate confidence | Evidence is consistent with the diagnosis but alternative explanations remain plausible. Flag the alternatives explicitly                                                                                                                                          |
+| **2** | Low confidence      | Limited evidence, mostly reasoning from code patterns rather than observed artifacts. Recommend gathering more data before committing to a fix                                                                                                                     |
+| **1** | Speculative         | No direct evidence for the root cause. The fix is a best guess. Recommend reproducing the failure locally or adding instrumentation before proceeding                                                                                                              |
 
 Apply the score:
 
 - **If >2:** continue to [Decide Fix Approach](#decide-fix-approach).
-- **If less than 5/5:** the plan must include the frontend and/or backend observability changes that would make the next occurrence diagnosable at 5/5 confidence. Scope recommendations to repositories and services implicated by the evidence; include multiple repositories only when the artifacts link them to the failure.
+- **If less than 5/5:** the plan must include the frontend and/or backend observability changes that would make the next occurrence's root cause directly visible in artifacts (the artifact half of 5/5; reaching 5/5 additionally requires reproduction per [Causal Chain](#causal-chain)). Scope recommendations to the repositories and services on the causal chain; resolve owners via the groundtruth registry rather than assuming the failure belongs to the repo the test lives in.
 - **If confidence is 2 or below:** do not propose a code fix. Instead, recommend specific instrumentation or reproduction steps to raise confidence.
 
+## Causal Chain
+
+Every plan must trace the failure to a terminal _cause_, not a symptom:
+
+failing assertion → app/UI state → network/trace/log evidence → owning service and repo → cause at a file/line, config key, or specific log line.
+
+- A status code, timeout, or throttle is a link in the chain, never the terminus. "The request 500ed" or "setup was throttled" is where the investigation continues, not where it stops.
+- The link types adapt to the failure surface. CI/setup, fixture, component, and unit failures substitute build logs, fixture state, or runner artifacts for network evidence, and the owning package/repo for a deployed service. The invariant is the terminus — a cause, not a symptom — not the specific link types.
+- When a network request or deployed service is implicated, resolve its owning service and repo via the groundtruth ownership registry (`ClipboardHealth/groundtruth`: `registry/services.json` → `registry/repos.json`; per-repo context in `context/devin-wiki/<repo>/`). Follow the chain into that repo's code — do not stop at the repo the test lives in.
+- When the causal event may be outside the test's own request path (seeding, deploys, CDC lag, async jobs), trace-ID lookup cannot reach it. Use time-window log queries scoped to the run instead.
+- If evidence runs out, state exactly which link breaks and what observability would extend it. That caps confidence at 2, and the instrumentation becomes the deliverable.
+
 ## Decide Fix Approach
 
 Applies to all test types.
@@ -84,9 +96,10 @@ Produce the plan with these fields:
 - **Current main status:** whether the failing commit's code path still exists on current `main`, has already been fixed, or has changed enough that the plan must be adjusted
 - **Symptom:** what failed and where
 - **Root cause:** concise technical explanation
+- **Causal chain:** each link from failing assertion to terminal cause with its evidence, or the explicit break point and the observability that would extend it (see [Causal Chain](#causal-chain))
 - **Evidence:** artifacts supporting the diagnosis (traces, network, error messages, screenshots as applicable)
 - **Proposed fix:** test harness, product, or both — with the specific file(s) and the change you would make
-- **Observability to reach 5/5:** required when confidence is less than 5/5. List the frontend and/or backend telemetry, logging, tracing, reporter, or metric changes that would make this flake diagnosable with 5/5 confidence next time. Include another repository only when the evidence implicates it. Use "N/A -- confidence is 5/5" only for a 5/5 plan.
+- **Observability to reach 5/5:** required when confidence is less than 5/5. List the frontend and/or backend telemetry, logging, tracing, reporter, or metric changes that would make this flake's root cause directly visible in artifacts next time (reproduction then completes 5/5). Include another repository only when the evidence implicates it. Use "N/A -- confidence is 5/5" only for a 5/5 plan.
 - **Sibling candidates:** files that appear to share the same anti-pattern, for the reviewer (or fix.md) to confirm. Or "N/A -- fix is test-specific" if the issue is one-off (see [`fix.md`](./fix.md) for what counts as a structural anti-pattern worth searching for).
 - **Validation plan:** lint/typecheck commands and test commands to run after applying the fix
 - **Open questions:** anything that needs human input before fixing