@clipboard-health/ai-rules 2.38.1 → 2.38.2

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@clipboard-health/ai-rules",
-  "version": "2.38.1",
+  "version": "2.38.2",
   "description": "Pre-built AI agent rules for consistent coding standards.",
   "keywords": [
     "ai",

package/skills/cb-babysit/SKILL.md

@@ -19,9 +19,9 @@ Resolve bundled "./scripts" paths relative to SKILL.md.
 
 The skill uses two sentinels with visible footer lines.
 
-**Addressed sentinel**: `<sub>🤖 <code>cb-babysit:addressed v1 core@3.17.0</code></sub>`. Appended on its own line at the end of every reply the skill posts so re-runs know which threads and review-body comments are already handled. Dedupe also recognizes the legacy `babysit-pr:addressed v1` prefix from before this skill was renamed.
+**Addressed sentinel**: `<sub>🤖 <code>cb-babysit:addressed v1 core@3.17.1</code></sub>`. Appended on its own line at the end of every reply the skill posts so re-runs know which threads and review-body comments are already handled. Dedupe also recognizes the legacy `babysit-pr:addressed v1` prefix from before this skill was renamed.
 
-**Follow-up sentinel**: `<sub>🤖 <code>cb-babysit:followup v1 core@3.17.0</code></sub>`. Attached to replies that defer an out-of-scope comment as a tracked follow-up. The sentinel is additive: the post-reply scripts still append the `addressed` sentinel at the end.
+**Follow-up sentinel**: `<sub>🤖 <code>cb-babysit:followup v1 core@3.17.1</code></sub>`. Attached to replies that defer an out-of-scope comment as a tracked follow-up. The sentinel is additive: the post-reply scripts still append the `addressed` sentinel at the end.
 
 **Sentinel recency rules.** The script emits a per-thread `activityState` with three values. Step 6a owns the handling rules for each state.
 
@@ -247,7 +247,7 @@ Body templates (the script appends the `addressed` sentinel if missing):
 - **Agree**: `Addressed in <commit-url>. <one-line what-changed>.`
 - **Disagree**: `Leaving current behavior. <reasoning>.`
 - **Already fixed**: `Already handled by <commit-url-or-file:line>. <brief pointer>.`
-- **Defer**: `Out of scope for this PR; this looks like follow-up work rather than something introduced or required by this change. <one-line rationale or pointer if useful>.\n\n<sub>🤖 <code>cb-babysit:followup v1 core@3.17.0</code></sub>`
+- **Defer**: `Out of scope for this PR; this looks like follow-up work rather than something introduced or required by this change. <one-line rationale or pointer if useful>.\n\n<sub>🤖 <code>cb-babysit:followup v1 core@3.17.1</code></sub>`
 
 For Defer replies, include the follow-up sentinel on its own line as shown. The script will append the `addressed` sentinel after it on its own line, so the final body ends with the follow-up sentinel followed by a blank line followed by the `addressed` sentinel. `grep cb-babysit:followup` finds the deferral and `grep cb-babysit:addressed` still marks the thread handled for dedupe.
 
@@ -263,7 +263,7 @@ The PR-level summary should:
 
 - Group by source. Use `## Review-body findings` for step-7 work and `## Conversation-tab comments` for step-6b work. Omit a section if its list is empty.
 - Inside each section, group verdicts under **Agree / Disagree / Already fixed / Deferred (out of scope)** subheadings. Omit a subheading if its list is empty.
-- Under **Deferred (out of scope)**, list each deferred item as a bullet, followed on its own line by `<sub>🤖 <code>cb-babysit:followup v1 core@3.17.0</code></sub>` so grep catches them individually.
+- Under **Deferred (out of scope)**, list each deferred item as a bullet, followed on its own line by `<sub>🤖 <code>cb-babysit:followup v1 core@3.17.1</code></sub>` so grep catches them individually.
 - Include the commit URL for fixes.
 - End with a fenced fingerprint block listing every current fingerprint (addressed and deferred) one per line. Include both `reviewBodyComments[].fingerprint` (whole-body, one per automated review) and `activeIssueComments[].fingerprint` (per Conversation-tab comment). Future runs dedupe by matching these against `priorBabysitSentinels`.
 

package/skills/cb-babysit/scripts/_sentinel.sh

@@ -10,7 +10,7 @@
 
 SENTINEL_PREFIX='cb-babysit:addressed v1 '
 LEGACY_SENTINEL_PREFIX='babysit-pr:addressed v1 '
-SENTINEL='<sub>🤖 <code>cb-babysit:addressed v1 core@3.17.0</code></sub>'
+SENTINEL='<sub>🤖 <code>cb-babysit:addressed v1 core@3.17.1</code></sub>'
 
 # Bot author allowlist (JSON array literal). Used by unresolvedPrComments.sh
 # as a fallback when GraphQL's `author.__typename == "Bot"` misses a GitHub

package/skills/cb-ship/references/pr-template.md

@@ -17,5 +17,5 @@ Concisely explain the user intent from session history and the meaningful behavi
 
 Optional, don't fabricate: ticket links, rollout plan, residual risk, or specific areas for reviewers to focus.
 
-<sub>🤖 <code>cb-ship:created v1 core@3.17.0</code></sub>
+<sub>🤖 <code>cb-ship:created v1 core@3.17.1</code></sub>
 ```

package/skills/flaky-debug/SKILL.md

@@ -3,7 +3,32 @@ name: flaky-debug
 description: Debug and fix flaky tests including Playwright E2E, NestJS service/integration, React component, and unit tests.
 ---
 
-Phases run in order. Skip a phase if you already have the information it produces. Phase 3 runs only in fix mode.
+Phases run in order. Phase 0 is mandatory. Skip a later phase if you already have the information it produces. Phase 3 runs only in fix mode.
+
+## Phase 0: Verify investigation access
+
+Run these checks before classifying or diagnosing the failure. Do not silently continue with reduced evidence if a check fails.
+
+1. Verify that `pup` can read APM spans. Do not use `pup auth status`; it can fail in sandboxes where environment-variable authentication works.
+
+   ```bash
+   command -v pup
+   pup traces search \
+     --query="service:cbh-backend-main env:staging" \
+     --from=15m \
+     --limit=1 \
+     --jq='.data | length'
+   ```
+
+   A successful query is sufficient even when the count is zero. If `pup` is missing, install it and rerun the check. For `401`/authentication failures, run `pup auth login` on a developer machine or provide `DD_API_KEY` and `DD_APP_KEY` in a sandbox/CI session. For `403`, obtain an application key or OAuth session with APM read access.
+
+2. For a CI-sourced E2E failure, verify artifact access by fetching the exact run before doing any diagnosis:
+
+   ```bash
+   bash "<flaky-debug-skill-dir>/scripts/fetch-llm-report.sh" "<github-actions-url>"
+   ```
+
+   If GitHub authentication fails, run `gh auth login --hostname github.com` and ensure the token can read Actions artifacts in the repository. If the artifact is missing or expired, rerun the source workflow or obtain the `playwright-llm-report`/Playwright HTML report from the user. A missing run URL or inaccessible artifact blocks a CI-sourced E2E investigation; report the failed command and remediation instead of degrading the confidence score. Artifact access is not applicable to service, component, or unit failures that were reproduced locally with complete output.
 
 ## Mode: plan vs fix
 

package/skills/flaky-debug/references/datadog-apm-traces.md

@@ -1,84 +1,165 @@
 # Datadog APM Traces
 
-Fetch and display the full APM trace for a given trace ID, or look up a specific span by span ID.
+Correlate a Playwright network instance with backend spans in Datadog's **APM Spans index**. Use the per-request `attempts[].network.instances[].traceId`; do not use the test-level fixture annotation.
 
 ## Prerequisites
 
-The `pup` CLI must be installed and authenticated. Two auth paths are supported:
+Phase 0 of the flaky-debug skill must already have proved that `pup` can read APM spans and that the exact CI artifact is accessible. Do not run `pup auth status`; call `pup traces search` directly because environment-variable authentication can work in sandboxes where `pup auth status` does not.
 
-- **macOS Keychain** (via `pup auth login`) — the default on developer machines.
-- **Environment variables** (`DD_API_KEY` + `DD_APP_KEY`) — the path used in sandboxes and CI.
+If a query fails with `401`, run `pup auth login` or provide `DD_API_KEY` and `DD_APP_KEY`. For `403`, obtain APM read access. Do not continue an investigation without this access.
 
-Don't run `pup auth status` to verify auth. It fails in sandboxes even when env-var auth is working. Call `pup traces search …` directly. If the query fails with an auth error, check `DD_API_KEY` / `DD_APP_KEY` or run `pup auth login`.
+## Which trace ID is authoritative?
 
-## Key pup conventions
+Use these sources in order:
 
-- **Durations are in NANOSECONDS**: 1 second = 1,000,000,000 ns; 5ms = 5,000,000 ns. Convert to ms for display by dividing by 1,000,000.
-- **Default time range is 1h.** Always pass `--from` explicitly — use `--from=7d` or `--from=30d` for older traces.
-- **Default output is JSON.** Pipe JSON through `jq` for extraction.
-- **`--limit` defaults to 50.** Max is 1000. For large traces, you may need multiple paginated calls (but pup handles most pagination internally).
-- **Query syntax for traces:** `service:<name> resource_name:<path> @duration:>5s env:production status:error operation_name:<op>`
+1. The response `traceparent`, exposed as `network.instances[].traceId`. It identifies the context the backend actually selected.
+2. For an aborted or unreadable response, the request's `x-datadog-trace-id`. Reporter versions containing the Datadog fallback normalize its unsigned decimal value to a 32-character, zero-padded hexadecimal trace ID.
+3. The request `traceparent` only when no Datadog request context exists.
 
-## Steps
+Do **not** query the test-level `traceparent` annotation. The Playwright fixture sets a W3C header, but Datadog RUM also adds a different `x-datadog-*` context to first-party requests. The observed staging backend selects the Datadog context when the two conflict.
 
-### 1. If a span ID was provided, fetch that span first
+For reports produced by an older reporter, inspect the full Playwright trace's request headers. Convert `x-datadog-trace-id` as described below. If neither a response trace nor the Datadog request header is available, use the request-ID correlation path instead of claiming that the fixture ID represents the backend trace.
+
+## Known-good query recipe
+
+### 1. Set an exact time window
+
+Derive the request time from the attempt's `startTime` plus the network instance's `offsetMs`. Use a narrow absolute RFC 3339 window around the request; this avoids false matches and is safer than relying on `pup`'s one-hour default.
+
+```bash
+pup traces search \
+  --query='env:staging trace_id:<TRACE_ID>' \
+  --from='2026-07-06T14:10:00Z' \
+  --to='2026-07-06T14:20:00Z' \
+  --limit=1000
+```
+
+Start without a service filter when ownership is uncertain. Once the service is known, add it to reduce noise.
+
+### 2. Use the staging service name
+
+Staging APM service names are `cbh-*` prefixed. For example:
 
 ```bash
-pup traces search --query="span_id:<SPAN_ID>" --from=30d --limit=1
+pup traces search \
+  --query='env:staging service:cbh-backend-main trace_id:<TRACE_ID>' \
+  --from='<RFC3339_START>' \
+  --to='<RFC3339_END>' \
+  --limit=1000
 ```
 
-Display the span's details (service, operation, resource, duration, status, error if any) before proceeding to fetch the full trace.
+`service:cbh-backend-main` works; `service:backend-main` returns nothing. Other examples include `cbh-user`, `cbh-bg-jobs`, and `cbh-documents-service-backend`.
 
-If the query returns no results, tell the user the span was not found in the APM Spans index. Continue to step 2 using the trace ID from the arguments.
+### 3. Use a supported ID representation
 
-### 2. Fetch the full trace
+`pup traces search` accepts either:
 
-Use the `trace_id` to retrieve all spans in the trace:
+- the full 32-character hexadecimal trace ID returned by the spans API, or
+- the unsigned decimal value of the lower 64 bits.
+
+Do not convert the entire 128-bit value to decimal; the APM Spans search did not match that representation in the live verification.
+
+Convert a 32-character W3C/Datadog hexadecimal ID to its lower-64 decimal form:
 
 ```bash
-pup traces search --query="trace_id:<TRACE_ID>" --from=30d --limit=1000
+node -e '
+const id = process.argv[1].replace(/^0x/, "");
+if (!/^[0-9a-f]{32}$/i.test(id)) process.exit(1);
+console.log(BigInt(`0x${id.slice(-16)}`).toString(10));
+' '<TRACE_ID>'
 ```
 
-If the trace has more than 1000 spans, the response will be truncated. In that case, narrow the query by adding filters like `service:<name>` or `status:error` to focus on relevant spans.
-
-### 3. Parse and summarize the results
-
-The response JSON has this structure per span:
-
-```text
-.data[].attributes:
-  .span_id          — unique span identifier
-  .trace_id         — shared across all spans in the trace
-  .parent_id        — parent span (for building the call tree)
-  .service          — service name (e.g., "cbh-backend-main")
-  .operation_name   — operation (e.g., "express.request", "express.middleware", "http.request")
-  .resource_name    — resource (e.g., "GET /api/v1/users", "<anonymous>")
-  .status           — "ok" or "error"
-  .start_timestamp  — ISO 8601 start time
-  .end_timestamp    — ISO 8601 end time
-  .custom.duration  — duration in NANOSECONDS (divide by 1,000,000 for ms)
-  .custom.env       — environment (e.g., "staging", "production")
-  .custom.error     — error object with .message, .file, .fingerprint (null if no error)
-  .custom.type      — span type (e.g., "web", "http", "mongodb", "redis", "worker")
-  .custom.service   — service name (also at top level)
-  .tags[]           — array of tag strings
+Normalize an `x-datadog-trace-id` decimal value to the reporter's 32-character hexadecimal form:
+
+```bash
+node -e '
+const value = BigInt(process.argv[1]);
+if (value <= 0n || value > 0xffffffffffffffffn) process.exit(1);
+console.log(value.toString(16).padStart(32, "0"));
+' '<X_DATADOG_TRACE_ID>'
 ```
 
-Use `jq` to extract a useful summary. Example:
+### 4. Query a span ID only as a secondary lookup
+
+```bash
+pup traces search \
+  --query='env:staging span_id:<SPAN_ID>' \
+  --from='<RFC3339_START>' \
+  --to='<RFC3339_END>' \
+  --limit=1
+```
+
+A response `traceparent` contains the backend span ID. A request fallback contains the upstream parent span ID, which may not itself be an indexed backend span; the trace ID remains authoritative.
+
+### 5. Summarize relevant spans
+
+Durations are nanoseconds. Divide by 1,000,000 for milliseconds.
 
 ```bash
-# Quick error summary
-pup traces search --query="trace_id:<TRACE_ID>" --from=30d --limit=1000 \
-  | jq '[.data[] | select(.attributes.custom.error) | {
-      span_id: .attributes.span_id,
-      service: .attributes.service,
-      operation: .attributes.operation_name,
-      resource: .attributes.resource_name,
-      duration_ms: ((.attributes.custom.duration // 0) / 1000000 | . * 100 | round / 100),
-      error: .attributes.custom.error.message
+pup traces search \
+  --query='env:staging trace_id:<TRACE_ID>' \
+  --from='<RFC3339_START>' \
+  --to='<RFC3339_END>' \
+  --limit=1000 \
+  | jq '[.data[].attributes | {
+      trace_id,
+      span_id,
+      parent_id,
+      service,
+      operation: .operation_name,
+      resource: .resource_name,
+      status,
+      retained_by,
+      duration_ms: ((.custom.duration // 0) / 1000000 | . * 100 | round / 100),
+      error: .custom.error.message
     }]'
 ```
 
-### 4. Query additional data
+## Retention limitation and fallback correlation
+
+`pup traces search` searches indexed spans, not every ingested span. An ingested trace is available in Live Search for only 15 minutes unless a retention mechanism indexes it. A zero-result indexed query after that window does **not** prove that propagation or ingestion failed.
+
+If no indexed span exists:
+
+1. Record the exact query, environment, service variants, ID representations, and absolute time window tried.
+2. Search logs in the same window using the API Gateway/request ID, exact method/path, and service. The reporter may expose `requestId`; setup diagnostics may expose `apiGatewayRequestId`.
+3. Correlate the response status/body and the owning service from the report. Do not lower confidence merely because a success trace was not selected for retention; lower it only when a necessary causal link remains unproved.
+4. For a new occurrence, run the APM lookup within 15 minutes or add an explicit retention strategy for test traffic before relying on historical APM lookup.
+
+Example log search shape:
+
+```bash
+pup logs search \
+  --query='env:staging "<API_GATEWAY_REQUEST_ID>"' \
+  --from='<RFC3339_START>' \
+  --to='<RFC3339_END>' \
+  --limit=100
+```
+
+## Audit proof: why the fixture ID cannot be the bridge
+
+The STAFF-1792 audit used the failed attempt from `cbh-admin-frontend` CI run `29337068333` (`2026-07-14T13:35:30Z`) and its raw Playwright trace:
+
+- The request carried fixture W3C context `00-0df9d80ac12e173dfcfc55aa29379ba6-5819a315ecd12173-01`.
+- Datadog RUM added `x-datadog-trace-id: 13639781014258445719`, `x-datadog-parent-id: 7268017665183408395`, origin `rum`, and sampling priority `1`.
+- The backend response carried `00-0000000000000000bd4a3c90d8fded97-277d492d139029e1-01`. `bd4a3c90d8fded97` is exactly the hexadecimal form of the request's decimal `x-datadog-trace-id`, not the fixture trace ID.
+
+This proves header propagation reached the backend, but the backend selected the Datadog RUM context. Minting a differently shaped W3C fixture ID cannot fix that conflict by itself. The reporter therefore uses the response trace and, when the response is unreadable, the Datadog request context.
+
+The same audit checked backend and Datadog configuration:
+
+- `clipboard-health/src/tracer.ts` calls `tracer.init({ logLevel: "error" })` without a propagation-style override; the locked `dd-trace` version is `5.27.1`.
+- No customer APM head-sampling rules were returned for `cbh-backend-main`/staging, and adaptive sampling was not onboarded.
+- The retention-filter API returned `401` for the available OAuth session, so the audit could not enumerate organization retention filters and does not infer that none exist. Indexed sample spans reported `retained_by: diversity_sampling`.
+
+Historical indexed proof comes from STAFF-1678. Its backend response trace ID `00000000000000001732fff1044f982e` converts to decimal `1671679822332401710`; this exact query returned retained `cbh-user` spans for `POST /api/user/create`:
+
+```bash
+pup traces search \
+  --query='trace_id:1671679822332401710 env:staging service:cbh-user resource_name:"POST /api/user/create"' \
+  --from='2026-07-06T14:10:00Z' \
+  --to='2026-07-06T14:20:00Z' \
+  --limit=5
+```
 
-If additional data would help diagnose the issue (e.g. logs, rum, cicd), use the pup CLI.
+Those spans reported `retained_by: diversity_sampling`. This is the working alternative correlation path: use the backend-selected per-request trace ID when it was retained; otherwise correlate by API Gateway/request ID plus exact time, method, path, and service.

package/skills/flaky-debug/references/plan-e2e.md

@@ -136,6 +136,8 @@ Scan `attempts[].network.instances[]` for 4xx/5xx responses near the failure's `
 
 **`traceId`** — when present on a failing instance (`attempts[].network.instances[].traceId`), you must follow [`./datadog-apm-traces.md`](./datadog-apm-traces.md) to correlate with backend behavior. This is the bridge between frontend test failure and potential backend root cause.
 
+Use the per-request `network.instances[].traceId`, not the test-level `traceparent` annotation. Datadog RUM adds `x-datadog-*` headers to first-party requests, and the staging backend selects that context when it conflicts with the fixture's W3C `traceparent`. The reporter prefers the response `traceparent`; for an unreadable/aborted response it normalizes the request's `x-datadog-trace-id` to the same zero-padded hexadecimal form. Older reporter versions may fall back to the fixture ID instead; inspect the full Playwright trace for `x-datadog-trace-id` or use the request-ID/time-window correlation path in `datadog-apm-traces.md`.
+
 If no expected request was emitted, say that explicitly and do not diagnose backend latency for that action. Instead, use the trace, console/page errors, session/permission state, and frontend code path to explain why the request never started.
 
 Check `attempts[].network.summary` for saturation. Non-zero `instancesDroppedByGroupCap`, `instancesDroppedByInstanceCap`, or `instancesEvictedAfterAdmission` means retained content is a sample and the request you care about may have been dropped — note this as a confidence-reducing factor. `instancesDroppedByFilter` alone is expected (static assets are filtered by design). v3 caps: instances 500, groups 200, bodies 100 per attempt.