@clipboard-health/ai-rules 2.38.3 โ 2.39.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/package.json +1 -1
- package/skills/cb-babysit/SKILL.md +4 -4
- package/skills/cb-babysit/scripts/_sentinel.sh +1 -1
- package/skills/cb-ship/references/pr-template.md +1 -1
- package/skills/flaky-critic/references/rubric.md +8 -2
- package/skills/flaky-debug/SKILL.md +1 -1
- package/skills/flaky-debug/references/fix.md +37 -1
- package/skills/flaky-debug/references/plan.md +1 -0
- package/skills/flaky-deep-dive/SKILL.md +138 -0
- package/skills/flaky-deep-dive/references/dry-run-home-health-full-lifecycle.md +177 -0
- package/skills/flaky-deep-dive/scripts/check-prerequisites.sh +177 -0
- package/skills/flaky-triage/SKILL.md +1 -1
package/package.json
CHANGED
|
@@ -19,9 +19,9 @@ Resolve bundled "./scripts" paths relative to SKILL.md.
|
|
|
19
19
|
|
|
20
20
|
The skill uses two sentinels with visible footer lines.
|
|
21
21
|
|
|
22
|
-
**Addressed sentinel**: `<sub>๐ค <code>cb-babysit:addressed v1 core@3.
|
|
22
|
+
**Addressed sentinel**: `<sub>๐ค <code>cb-babysit:addressed v1 core@3.18.1</code></sub>`. Appended on its own line at the end of every reply the skill posts so re-runs know which threads and review-body comments are already handled. Dedupe also recognizes the legacy `babysit-pr:addressed v1` prefix from before this skill was renamed.
|
|
23
23
|
|
|
24
|
-
**Follow-up sentinel**: `<sub>๐ค <code>cb-babysit:followup v1 core@3.
|
|
24
|
+
**Follow-up sentinel**: `<sub>๐ค <code>cb-babysit:followup v1 core@3.18.1</code></sub>`. Attached to replies that defer an out-of-scope comment as a tracked follow-up. The sentinel is additive: the post-reply scripts still append the `addressed` sentinel at the end.
|
|
25
25
|
|
|
26
26
|
**Sentinel recency rules.** The script emits a per-thread `activityState` with three values. Step 6a owns the handling rules for each state.
|
|
27
27
|
|
|
@@ -247,7 +247,7 @@ Body templates (the script appends the `addressed` sentinel if missing):
|
|
|
247
247
|
- **Agree**: `Addressed in <commit-url>. <one-line what-changed>.`
|
|
248
248
|
- **Disagree**: `Leaving current behavior. <reasoning>.`
|
|
249
249
|
- **Already fixed**: `Already handled by <commit-url-or-file:line>. <brief pointer>.`
|
|
250
|
-
- **Defer**: `Out of scope for this PR; this looks like follow-up work rather than something introduced or required by this change. <one-line rationale or pointer if useful>.\n\n<sub>๐ค <code>cb-babysit:followup v1 core@3.
|
|
250
|
+
- **Defer**: `Out of scope for this PR; this looks like follow-up work rather than something introduced or required by this change. <one-line rationale or pointer if useful>.\n\n<sub>๐ค <code>cb-babysit:followup v1 core@3.18.1</code></sub>`
|
|
251
251
|
|
|
252
252
|
For Defer replies, include the follow-up sentinel on its own line as shown. The script will append the `addressed` sentinel after it on its own line, so the final body ends with the follow-up sentinel followed by a blank line followed by the `addressed` sentinel. `grep cb-babysit:followup` finds the deferral and `grep cb-babysit:addressed` still marks the thread handled for dedupe.
|
|
253
253
|
|
|
@@ -263,7 +263,7 @@ The PR-level summary should:
|
|
|
263
263
|
|
|
264
264
|
- Group by source. Use `## Review-body findings` for step-7 work and `## Conversation-tab comments` for step-6b work. Omit a section if its list is empty.
|
|
265
265
|
- Inside each section, group verdicts under **Agree / Disagree / Already fixed / Deferred (out of scope)** subheadings. Omit a subheading if its list is empty.
|
|
266
|
-
- Under **Deferred (out of scope)**, list each deferred item as a bullet, followed on its own line by `<sub>๐ค <code>cb-babysit:followup v1 core@3.
|
|
266
|
+
- Under **Deferred (out of scope)**, list each deferred item as a bullet, followed on its own line by `<sub>๐ค <code>cb-babysit:followup v1 core@3.18.1</code></sub>` so grep catches them individually.
|
|
267
267
|
- Include the commit URL for fixes.
|
|
268
268
|
- End with a fenced fingerprint block listing every current fingerprint (addressed and deferred) one per line. Include both `reviewBodyComments[].fingerprint` (whole-body, one per automated review) and `activeIssueComments[].fingerprint` (per Conversation-tab comment). Future runs dedupe by matching these against `priorBabysitSentinels`.
|
|
269
269
|
|
|
@@ -10,7 +10,7 @@
|
|
|
10
10
|
|
|
11
11
|
SENTINEL_PREFIX='cb-babysit:addressed v1 '
|
|
12
12
|
LEGACY_SENTINEL_PREFIX='babysit-pr:addressed v1 '
|
|
13
|
-
SENTINEL='<sub>๐ค <code>cb-babysit:addressed v1 core@3.
|
|
13
|
+
SENTINEL='<sub>๐ค <code>cb-babysit:addressed v1 core@3.18.1</code></sub>'
|
|
14
14
|
|
|
15
15
|
# Bot author allowlist (JSON array literal). Used by unresolvedPrComments.sh
|
|
16
16
|
# as a fallback when GraphQL's `author.__typename == "Bot"` misses a GitHub
|
|
@@ -17,5 +17,5 @@ Concisely explain the user intent from session history and the meaningful behavi
|
|
|
17
17
|
|
|
18
18
|
Optional, don't fabricate: ticket links, rollout plan, residual risk, or specific areas for reviewers to focus.
|
|
19
19
|
|
|
20
|
-
<sub>๐ค <code>cb-ship:created v1 core@3.
|
|
20
|
+
<sub>๐ค <code>cb-ship:created v1 core@3.18.1</code></sub>
|
|
21
21
|
```
|
|
@@ -8,7 +8,7 @@ This rubric turns Rocky's observed review standards into checkable rules for fla
|
|
|
8
8
|
|
|
9
9
|
Findings from automated reviewers (Mendral, CodeRabbit, etc.) are **advisory inputs, not binding rules**: the critic evaluates each on rubric merits and may decline one with stated reasoning, exactly as cb-babysit's Disagree replies do.
|
|
10
10
|
|
|
11
|
-
**Statement-missing vs substance-missing (backtest amendment, 2026-06-11):** B5/B6/B7/B8 each require an explicit statement in the plan (dedup-check results, Prior attempts section when applicable, confidence score plus observability section, current-main status, causal-chain section). When the _statement_ is absent but the violation is not demonstrated โ no actual duplicate found, no actual prior implementation tickets for the fingerprint family, no actual low-confidence code fix, no actual staleness, no demonstrably symptom-terminated diagnosis โ the verdict is reject with disposition **amend-and-resubmit**: one bounce asking for exactly the missing sections, never escalation. A demonstrated violation (a real duplicate, a mechanism already tried by a prior merged fix, a real โค2/5 code fix, a diagnosis that demonstrably stops at a symptom) is a substantive reject. The 2026-06-11 backtest showed 6 of 7 false rejections were statement-missing on plans predating these conventions; live plans carry the sections by template, but the distinction stays load-bearing for the agreement metric.
|
|
11
|
+
**Statement-missing vs substance-missing (backtest amendment, 2026-06-11):** B5/B6/B7/B8 and C6 each require an explicit statement in the plan (dedup-check results, Prior attempts section when applicable, confidence score plus observability section, current-main status, causal-chain section, sibling-repo check when applicable). When the _statement_ is absent but the violation is not demonstrated โ no actual duplicate found, no actual prior implementation tickets for the fingerprint family, no actual low-confidence code fix, no actual staleness, no demonstrably symptom-terminated diagnosis, no confirmed sibling-repo match โ the verdict is reject with disposition **amend-and-resubmit**: one bounce asking for exactly the missing sections, never escalation. A demonstrated violation (a real duplicate, a mechanism already tried by a prior merged fix, a real โค2/5 code fix, a diagnosis that demonstrably stops at a symptom, a confirmed sibling-repo match with no linked mirror implementation ticket deliverable) is a substantive reject. The 2026-06-11 backtest showed 6 of 7 false rejections were statement-missing on plans predating these conventions; live plans carry the sections by template, but the distinction stays load-bearing for the agreement metric.
|
|
12
12
|
|
|
13
13
|
## 2. Banned patterns
|
|
14
14
|
|
|
@@ -54,7 +54,7 @@ Opening one plan/PR per failing test when the failures share one mechanism (same
|
|
|
54
54
|
|
|
55
55
|
**Prior-sibling signal (backtest amendment, 2026-06-11):** sibling PRs for the same mechanism that were closed _unmerged_ are a strong reject signal, not mere context. A plan citing closed-unmerged siblings must explain what is materially different this time โ narrower scope alone is not an answer (the backtest's one genuine false approval cited prior wontfix siblings and explained them away).
|
|
56
56
|
|
|
57
|
-
**Prior attempts (amendment, 2026-07-16):** when a fingerprint family has prior implementation tickets, the plan MUST contain a **Prior attempts** section listing each prior ticket/PR, what it blamed, what it changed, and the recurrence evidence showing its diagnosis was wrong or incomplete. A missing section is statement-missing and gets amend-and-resubmit per ยง1. A plan that re-proposes a mechanism a prior merged fix already tried is a substantive reject. A family with **โฅ2 failed merged fixes** โ merged fixes followed by recurrence โ is disqualified from the normal fix path: verdict = reject and escalate as needs-human. Apply the `chronic` label and
|
|
57
|
+
**Prior attempts (amendment, 2026-07-16):** when a fingerprint family has prior implementation tickets, the plan MUST contain a **Prior attempts** section listing each prior ticket/PR, what it blamed, what it changed, and the recurrence evidence showing its diagnosis was wrong or incomplete. A missing section is statement-missing and gets amend-and-resubmit per ยง1. A plan that re-proposes a mechanism a prior merged fix already tried is a substantive reject. A family with **โฅ2 failed merged fixes** โ merged fixes followed by recurrence โ is disqualified from the normal fix path: verdict = reject and escalate as needs-human. Apply the `chronic` label and route the family to the credential-checked, dossier-first `flaky-deep-dive` skill. The โฅ2 threshold is intentionally different from flaky-triage's โฅ3-prior-implementation-tickets screen: triage counts tickets cheaply before any diagnosis exists, while the critic can see merge and recurrence outcomes and applies the sharper failed-merged-fix test. Either stage flagging chronic routes the same way; the counts are staged by the evidence available at each stage, not a contradiction.
|
|
58
58
|
|
|
59
59
|
### B6 โ Code fix on low-confidence diagnosis
|
|
60
60
|
|
|
@@ -167,6 +167,12 @@ Allowed when many tests fail before user-flow code and the evidence is a tool/ve
|
|
|
167
167
|
|
|
168
168
|
When the root cause is in another team's service, the plan may ship the local resilience fix and must route a recommendation to the owning team (STAFF-1413: frontend rethrow fix + "backend follow-up recommendation (โฆ 503 instead of 401) is noted in the ticket for the owning team"). It may not silently absorb a backend bug into test code.
|
|
169
169
|
|
|
170
|
+
### C6 โ Sibling-repo propagation
|
|
171
|
+
|
|
172
|
+
**Sibling-repo propagation (amendment, 2026-07-16):** For a mechanism plausibly shared between `cbh-admin-frontend` and `cbh-mobile-app`, state the sibling repository, current `main` commit searched, exact helper names or grep-able code patterns, commands and scope, and matches found (or `None found`). A mechanism is plausibly shared when the proposed fix touches the test harness, authentication/session/bootstrap, cache, API client, or a copied capability, helper, or convention. If the same mechanism is present, state that the fix deliverable will include a linked mirror implementation ticket following the Groundcrew ticket conventions, linked to the source implementation ticket and referencing the landed source fix as the pattern to adopt. If the mechanism is not plausibly shared, state `N/A` and why.
|
|
173
|
+
|
|
174
|
+
A plan that fixes a mechanism plausibly shared between the frontend repositories, including a shared-harness mechanism, without a **Sibling-repo check** statement is statement-missing and gets amend-and-resubmit per ยง1. A confirmed match with no linked mirror implementation ticket deliverable is a substantive reject.
|
|
175
|
+
|
|
170
176
|
## 5. Dispositions that need no code
|
|
171
177
|
|
|
172
178
|
### D1 โ Infra blip / environment incident (close as Canceled)
|
|
@@ -100,4 +100,4 @@ If you are in plan mode, present the plan and stop here.
|
|
|
100
100
|
|
|
101
101
|
## Phase 3: Apply the plan (fix mode only)
|
|
102
102
|
|
|
103
|
-
Follow [`references/fix.md`](./references/fix.md). It takes the plan from Phase 2, applies the proposed fix, searches for sibling anti-patterns, and verifies. PR creation is out of scope -- if the user later opens one (or invokes a PR-shipping skill), label it `flaky-test-fix`.
|
|
103
|
+
Follow [`references/fix.md`](./references/fix.md). It takes the plan from Phase 2, validates the recorded sibling frontend check and reruns it when stale or missing, applies the proposed fix, searches for same-repo sibling anti-patterns, and verifies. PR creation is out of scope -- if the user later opens one (or invokes a PR-shipping skill), label it `flaky-test-fix`.
|
|
@@ -12,6 +12,41 @@ Before editing, verify the plan is still current:
|
|
|
12
12
|
- The proposed fix targets the diagnosed failure surface, not only the final assertion.
|
|
13
13
|
- Any retry/wait change is safe and idempotent; it must not repeat one-time credentials, duplicate writes, destructive actions, or rate-limited setup calls.
|
|
14
14
|
|
|
15
|
+
## Validate the Sibling Frontend Check
|
|
16
|
+
|
|
17
|
+
During Phase 2, after the proposed fix is designed and before the plan is finalized, use this recipe to complete the plan's **Sibling-repo check** when the current repository is `cbh-admin-frontend` or `cbh-mobile-app`. During the fix preflight, consume that result by confirming that the recorded sibling `main` commit is still current; rerun the search only when it changed or the plan omitted the field.
|
|
18
|
+
|
|
19
|
+
For a mechanism plausibly shared between `cbh-admin-frontend` and `cbh-mobile-app`, state the sibling repository, current `main` commit searched, exact helper names or grep-able code patterns, commands and scope, and matches found (or `None found`). A mechanism is plausibly shared when the proposed fix touches the test harness, authentication/session/bootstrap, cache, API client, or a copied capability, helper, or convention. If the same mechanism is present, state that the fix deliverable will include a linked mirror implementation ticket following the Groundcrew ticket conventions, linked to the source implementation ticket and referencing the landed source fix as the pattern to adopt. If the mechanism is not plausibly shared, state `N/A` and why.
|
|
20
|
+
|
|
21
|
+
Use this search recipe:
|
|
22
|
+
|
|
23
|
+
1. Map `cbh-admin-frontend` to `cbh-mobile-app`, and `cbh-mobile-app` to `cbh-admin-frontend`.
|
|
24
|
+
2. Search a temporary shallow clone pinned to the sibling repository's `main`:
|
|
25
|
+
|
|
26
|
+
```bash
|
|
27
|
+
sibling_repo="<cbh-admin-frontend-or-cbh-mobile-app>"
|
|
28
|
+
sibling_root="$(mktemp -d)/$sibling_repo"
|
|
29
|
+
gh repo clone "ClipboardHealth/$sibling_repo" "$sibling_root" -- \
|
|
30
|
+
--depth=1 --branch main --single-branch
|
|
31
|
+
git -C "$sibling_root" rev-parse HEAD
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
3. Search for both the exact helper or capability names and the grep-able structural pattern behind the fix. Limit the first pass to likely harness and test paths, then widen to the repository if needed:
|
|
35
|
+
|
|
36
|
+
```bash
|
|
37
|
+
rg -n --glob '*.{ts,tsx}' \
|
|
38
|
+
'<exact-helper-name>|<grep-able-structural-pattern>' \
|
|
39
|
+
"$sibling_root/<likely-path>"
|
|
40
|
+
# Run only when the scoped search cannot rule the mechanism in or out.
|
|
41
|
+
rg -n --glob '*.{ts,tsx}' \
|
|
42
|
+
'<exact-helper-name>|<grep-able-structural-pattern>' \
|
|
43
|
+
"$sibling_root"
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
4. Inspect each match to decide whether it carries the same mechanism, rather than relying on the text match alone. Record each query run, its scope, and the matching files or `None found` in the plan before applying the fix.
|
|
47
|
+
|
|
48
|
+
Do not edit the sibling repository as part of the current fix. When the same mechanism is present, the deliverable includes a linked mirror implementation ticket created after the source fix lands. Follow the `create-groundcrew-ticket` conventions: include the exact `Repository: <sibling-repo>` line in the description, assign the current Linear user, use Todo with exactly one appropriate `agent-*` label, link the source implementation ticket, and cite the landed PR or commit as the pattern to adopt. If the current workflow stops before merge, carry this as an explicit post-merge deliverable and do not report sibling-repo propagation as complete until the ticket exists.
|
|
49
|
+
|
|
15
50
|
## Apply the Proposed Fix
|
|
16
51
|
|
|
17
52
|
Edit the files listed in the plan's **Proposed fix** field. Keep the change minimal -- the plan already chose between test harness, product, and both.
|
|
@@ -59,7 +94,7 @@ Run the plan's **Validation plan** commands โ including the previously-flaky t
|
|
|
59
94
|
|
|
60
95
|
When opening a PR for a flaky test fix, include `--label flaky-test-fix` in the `gh pr create` command so other agents can find it during Phase 1b deduplication.
|
|
61
96
|
|
|
62
|
-
When documenting the fix in a PR or issue, use this structure. Carry **Confidence**, **Symptom**, **Root cause**, **Evidence**, and **Residual risk** straight over from the plan. Three plan fields rename: **Proposed fix** โ **Fix**, **Sibling candidates** โ **Siblings fixed**, **Validation plan** โ **Validation**. Drop **Open questions** (resolved by fix time):
|
|
97
|
+
When documenting the fix in a PR or issue, use this structure. Carry **Confidence**, **Symptom**, **Root cause**, **Evidence**, **Sibling-repo check**, and **Residual risk** straight over from the plan. Three plan fields rename: **Proposed fix** โ **Fix**, **Sibling candidates** โ **Siblings fixed**, **Validation plan** โ **Validation**. Add the linked mirror ticket URL to **Sibling-repo check** once the source fix lands and the ticket exists. Before then, record `Pending: create linked mirror implementation ticket after <source-PR> lands`; replace that status with the ticket URL when created. Drop **Open questions** (resolved by fix time):
|
|
63
98
|
|
|
64
99
|
- **Test ID:** if provided in prompt
|
|
65
100
|
- **Agent session ID:** your running session ID to resume if needed
|
|
@@ -71,5 +106,6 @@ When documenting the fix in a PR or issue, use this structure. Carry **Confidenc
|
|
|
71
106
|
- **Evidence:** artifacts supporting the diagnosis (traces, network, error messages, screenshots as applicable)
|
|
72
107
|
- **Fix:** test-only, product-only, or both
|
|
73
108
|
- **Siblings fixed:** list of other files where the same anti-pattern was or should be corrected (or "N/A -- fix was test-specific")
|
|
109
|
+
- **Sibling-repo check:** sibling repository and current `main` commit searched; exact helper names or grep-able code patterns; commands, scope, and matches; linked mirror implementation ticket URL when created, or the explicit pending post-merge deliverable before then; or `N/A` with the reason the mechanism was not plausibly shared
|
|
74
110
|
- **Validation:** commands and suites run
|
|
75
111
|
- **Residual risk:** what could still be flaky
|
|
@@ -102,6 +102,7 @@ Produce the plan with these fields:
|
|
|
102
102
|
- **Proposed fix:** test harness, product, or both โ with the specific file(s) and the change you would make
|
|
103
103
|
- **Observability to reach 5/5:** required when confidence is less than 5/5. List the frontend and/or backend telemetry, logging, tracing, reporter, or metric changes that would make this flake's root cause directly visible in artifacts next time (reproduction then completes 5/5). Include another repository only when the evidence implicates it. Use "N/A -- confidence is 5/5" only for a 5/5 plan.
|
|
104
104
|
- **Sibling candidates:** files that appear to share the same anti-pattern, for the reviewer (or fix.md) to confirm. Or "N/A -- fix is test-specific" if the issue is one-off (see [`fix.md`](./fix.md) for what counts as a structural anti-pattern worth searching for).
|
|
105
|
+
- **Sibling-repo check:** for a mechanism plausibly shared between `cbh-admin-frontend` and `cbh-mobile-app`, state the sibling repository, current `main` commit searched, exact helper names or grep-able code patterns, commands and scope, and matches found (or `None found`). A mechanism is plausibly shared when the proposed fix touches the test harness, authentication/session/bootstrap, cache, API client, or a copied capability, helper, or convention. If the same mechanism is present, state that the fix deliverable will include a linked mirror implementation ticket following the Groundcrew ticket conventions, linked to the source implementation ticket and referencing the landed source fix as the pattern to adopt. If the mechanism is not plausibly shared, state `N/A` and why. Use the concrete search recipe in [`fix.md`](./fix.md#validate-the-sibling-frontend-check).
|
|
105
106
|
- **Validation plan:** lint/typecheck commands and test commands to run after applying the fix
|
|
106
107
|
- **Open questions:** anything that needs human input before fixing
|
|
107
108
|
- **Residual risk:** what could still be flaky after the fix
|
|
@@ -0,0 +1,138 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: flaky-deep-dive
|
|
3
|
+
description: Investigate chronic flaky-test fingerprint families with a credential-checked, dossier-first, cross-repository causal-chain method and produce a designed fix or observability design for human review. Use when flaky-triage routes a family with at least three prior implementation tickets, flaky-critic routes a family with at least two failed merged fixes, or a user asks for the chronic flake deep-dive track.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Flaky Deep Dive
|
|
7
|
+
|
|
8
|
+
Run a diagnosis-and-design investigation for a chronic fingerprint family. Produce a document for human review; never edit product code, create a fix branch, commit a fix, or open a fix PR.
|
|
9
|
+
|
|
10
|
+
Read the bundled [Home Health dry run](references/dry-run-home-health-full-lifecycle.md) for the expected dossier and causal narrative depth. Its structure is distilled from Rocky's `hh-full-lifecycle.md` and `staging-seed-reliability.md` reference deep dives; the Phase 6 checklist remains binding where the historical artifact records an evidence limitation.
|
|
11
|
+
|
|
12
|
+
## Phase 0: Enforce credential preconditions
|
|
13
|
+
|
|
14
|
+
Run the preflight before reading source code or diagnosing:
|
|
15
|
+
|
|
16
|
+
```bash
|
|
17
|
+
bash "<flaky-deep-dive-skill-dir>/scripts/check-prerequisites.sh" "${AWS_PROFILE:-sdlc}"
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
The preflight must prove all three:
|
|
21
|
+
|
|
22
|
+
1. `pup` can query Datadog APM spans and logs.
|
|
23
|
+
2. The staging API is reachable through the non-production VPN.
|
|
24
|
+
3. AWS read credentials are active through `aws sts get-caller-identity`.
|
|
25
|
+
|
|
26
|
+
If any check fails, stop the investigation. Do not continue with reduced evidence or silently cap confidence. File a provisioning ticket linked to the chronic ticket with:
|
|
27
|
+
|
|
28
|
+
- the fingerprint family and blocked investigation;
|
|
29
|
+
- the failed command and exact error class;
|
|
30
|
+
- the missing access: Datadog APM/log read, non-production VPN, or AWS `sdlc` read credentials;
|
|
31
|
+
- the causal-chain link or fault-injection step the access blocks;
|
|
32
|
+
- the remediation printed by the preflight.
|
|
33
|
+
|
|
34
|
+
If the environment cannot file Linear tickets, output the exact title and body for a human to file. Do not proceed blind.
|
|
35
|
+
|
|
36
|
+
For a CI-sourced E2E family, also prove access to the exact run and reporter artifact:
|
|
37
|
+
|
|
38
|
+
```bash
|
|
39
|
+
bash "<flaky-deep-dive-skill-dir>/../flaky-debug/scripts/fetch-llm-report.sh" \
|
|
40
|
+
"<github-actions-url>"
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
If `gh` authentication fails, run `gh auth login --hostname github.com`. If the artifact expired, rerun the source workflow or obtain the `playwright-llm-report`/Playwright HTML report. An inaccessible source artifact blocks the investigation.
|
|
44
|
+
|
|
45
|
+
## Phase 1: Build the dossier first
|
|
46
|
+
|
|
47
|
+
Do not diagnose until the dossier is complete.
|
|
48
|
+
|
|
49
|
+
Read and follow `../flaky-debug/SKILL.md` Phase 1b in full. Its four unbounded Linear searches for the fingerprint family and exact test title are the canonical dossier recipe; do not narrow, copy, or alter them.
|
|
50
|
+
|
|
51
|
+
The chronic track adds these requirements for every implementation attempt returned by that recipe:
|
|
52
|
+
|
|
53
|
+
1. Read the ticket and investigation plan.
|
|
54
|
+
2. Read the complete PR diff, review discussion, merge state, and fix commit through `gh`.
|
|
55
|
+
3. Record what the attempt blamed and exactly what it changed.
|
|
56
|
+
4. Find a later post-merge sighting that proves recurrence. Do not infer failure merely from age.
|
|
57
|
+
5. Check current `main` in every repository the attempt touched.
|
|
58
|
+
|
|
59
|
+
Create a `Prior attempts` table with these columns:
|
|
60
|
+
|
|
61
|
+
| Prior ticket/PR/commit | What it blamed | What it changed | Recurrence evidence |
|
|
62
|
+
| ---------------------- | -------------- | --------------- | ------------------- |
|
|
63
|
+
|
|
64
|
+
Every prior ticket, PR, and fix commit must appear. Attempt N+1 must rule out attempts 1..N: after the table, list each prior blamed mechanism and the evidence that falsifies it, limits it to one symptom, or proves it incomplete. A new attempt that merely repeats a prior mechanism is not a deep dive.
|
|
65
|
+
|
|
66
|
+
Also search open and closed-unmerged PRs labeled `flaky-test-fix`, plus recent `main` commits touching the test, helper, product surface, and implicated services. Stop and report if a current fix already covers the family.
|
|
67
|
+
|
|
68
|
+
## Phase 2: Resolve ownership from the workspace
|
|
69
|
+
|
|
70
|
+
Start from evidence, not the repository containing the test.
|
|
71
|
+
|
|
72
|
+
Read and follow the ownership and causal-chain rules in `../flaky-debug/references/plan.md`. The chronic track additionally requires recording the registry result and checkout SHA for every implicated repository.
|
|
73
|
+
|
|
74
|
+
Locate a `ClipboardHealth/groundtruth` checkout. Resolve deployed services through `registry/services.json`, then resolve repositories and owners through `registry/repos.json`:
|
|
75
|
+
|
|
76
|
+
```bash
|
|
77
|
+
jq --arg service "<service-id>" \
|
|
78
|
+
'.services[] | select(.id == $service)' \
|
|
79
|
+
"<groundtruth>/registry/services.json"
|
|
80
|
+
|
|
81
|
+
jq --arg repo "<repo-id>" \
|
|
82
|
+
'.repos[] | select(.id == $repo)' \
|
|
83
|
+
"<groundtruth>/registry/repos.json"
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
Read `context/devin-wiki/<repo>/` when it helps locate the owning code. Follow the causal chain into every implicated repository and inspect current `main`; never stop at the repository where the test lives. Record the resolved service, repository, owner, route or responsibility, and code checkout SHA in the deliverable.
|
|
87
|
+
|
|
88
|
+
If the registry is missing or cannot resolve an implicated service, treat ownership as a broken causal-chain link and design the registry/telemetry change needed to resolve it.
|
|
89
|
+
|
|
90
|
+
## Phase 3: Gather evidence across every link
|
|
91
|
+
|
|
92
|
+
Use all applicable evidence sources:
|
|
93
|
+
|
|
94
|
+
- **LLM reporter and Playwright artifacts:** identify the exact attempt, lifecycle step, network instance, request/response body, trace ID, request ID, screenshot, and stack.
|
|
95
|
+
- **Datadog APM:** follow the per-request backend-selected trace ID. Read `../flaky-debug/references/datadog-apm-traces.md` before querying.
|
|
96
|
+
- **Datadog run-window logs:** query the absolute CI window for seed operations, deploys, CDC lag, queues, background jobs, and other causal events outside the request path. A trace-only search cannot establish these events.
|
|
97
|
+
- **GitHub Actions:** use `gh run view`, job logs, annotations, artifacts, retry history, commit SHA, and deployed-version evidence.
|
|
98
|
+
- **AWS read-only evidence:** use the active staging credentials for service-specific read commands when Datadog and CI artifacts leave an AWS-managed link unresolved.
|
|
99
|
+
- **Current source:** cite file and line, config key, workflow step, or specific log event in the owning repository.
|
|
100
|
+
|
|
101
|
+
Preserve exact commands, absolute time windows, trace/request IDs, run URLs, commit SHAs, and zero-result limitations in the evidence appendix.
|
|
102
|
+
|
|
103
|
+
## Phase 4: Terminate the causal chain
|
|
104
|
+
|
|
105
|
+
Apply the causal-chain and confidence rules in `../flaky-debug/references/plan.md` without changing their score meanings or valid terminal states.
|
|
106
|
+
|
|
107
|
+
Fault-injection reproduction is mandatory for 5/5. Use a focused lower-level test or harness control that delays, fails, reorders, throttles, or disables the blamed response or step. A normal green rerun is not reproduction.
|
|
108
|
+
|
|
109
|
+
When the chain is broken, do not design a speculative code fix. Convert the deliverable into the observability design that would terminate the chain on the next occurrence, including event names, fields, correlation IDs, service/repository, retention, reporter exposure, and the query that would prove or falsify the hypothesis.
|
|
110
|
+
|
|
111
|
+
## Phase 5: Design the fix for review
|
|
112
|
+
|
|
113
|
+
For a terminated chain, design the smallest fix at the terminal cause and name:
|
|
114
|
+
|
|
115
|
+
- owning repository, files, and team;
|
|
116
|
+
- behavior and contract changes;
|
|
117
|
+
- migration, rollout, and compatibility constraints;
|
|
118
|
+
- defensive apparatus or prior mitigations to delete after the fix;
|
|
119
|
+
- fault-injection and no-retry validation;
|
|
120
|
+
- residual risks and falsifying outcomes.
|
|
121
|
+
|
|
122
|
+
Separate the root fix from optional resilience or observability follow-ups. Do not turn another service's defect into a local test retry. Do not implement the design.
|
|
123
|
+
|
|
124
|
+
## Phase 6: Write the deliverable
|
|
125
|
+
|
|
126
|
+
Write Markdown to the user-specified path, or `/tmp/flaky-deep-dive-<fingerprint>.md` by default, with:
|
|
127
|
+
|
|
128
|
+
1. Title and metadata: family, repositories, status, investigator, current-main SHAs, confidence.
|
|
129
|
+
2. Credential preflight evidence.
|
|
130
|
+
3. `Dossier` and `Prior attempts`.
|
|
131
|
+
4. Latest recurrence signatures.
|
|
132
|
+
5. Causal chain with ownership resolution.
|
|
133
|
+
6. Root-cause statement and discriminated alternatives.
|
|
134
|
+
7. Designed fix, or `Observability design` when the chain is broken.
|
|
135
|
+
8. Fault-injection reproduction and validation plan.
|
|
136
|
+
9. Evidence appendix with links, commands, IDs, timestamps, and citations.
|
|
137
|
+
|
|
138
|
+
End with the document path/link and a one-sentence review request. Never open a direct fix PR.
|
|
@@ -0,0 +1,177 @@
|
|
|
1
|
+
# Dry run: Home Health `fullLifecycle.spec.ts` chronic flake
|
|
2
|
+
|
|
3
|
+
- **Repo:** `cbh-admin-frontend`
|
|
4
|
+
- **Spec:** `playwright/e2e/homeHealth/fullLifecycle.spec.ts`
|
|
5
|
+
- **Fingerprint family:** `33deef731a10` with related variants
|
|
6
|
+
- **Status:** Closed as Done at least six times before recurring
|
|
7
|
+
- **Investigator:** Deep-dive track; diagnosis and design only
|
|
8
|
+
- **Confidence:** 4/5; the causal chain terminates, but fault injection remains
|
|
9
|
+
- **Source:** Real chronic-family deep dive completed on 2026-07-07
|
|
10
|
+
- **Historical code SHA:** `fe4e2cd8424`
|
|
11
|
+
- **Current-main SHA checked:** `1bafb36f8f909274b96eef840655ffdc7e15f9d0`
|
|
12
|
+
|
|
13
|
+
This checked-in dry run demonstrates the expected depth and document shape. The original investigation lacked a VPN-capable environment, which correctly held confidence below 5/5. The credential-checked forward rerun below stopped before diagnosis, as the new skill requires.
|
|
14
|
+
|
|
15
|
+
## 0. Credential preflight and current-main status
|
|
16
|
+
|
|
17
|
+
### Credential-checked forward rerun on 2026-07-16
|
|
18
|
+
|
|
19
|
+
- Datadog APM: passed a staging span query through `pup`.
|
|
20
|
+
- Datadog logs: passed a staging log query through `pup`.
|
|
21
|
+
- Staging VPN: failed. `curl --connect-timeout 10 --max-time 20 https://apigateway.staging.clipboardhealth.org/api/healthCheck` exited 56 with `CONNECT tunnel failed, response 403`.
|
|
22
|
+
- AWS: failed. `aws sts get-caller-identity --profile sdlc` exited 255 because the `sdlc` profile was not configured.
|
|
23
|
+
|
|
24
|
+
The live forward test stopped at Phase 0 and produced a provisioning-ticket body. It did not reuse the historical diagnosis as if current credentials had passed.
|
|
25
|
+
|
|
26
|
+
### Current main
|
|
27
|
+
|
|
28
|
+
The historical designed fix has since landed:
|
|
29
|
+
|
|
30
|
+
- [PR #7574](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7574), commit `e6818dabd950d43755c3ec96de0373062a4522ba`, hoisted the Home Health action dialogs and split the 2,096-line mega-spec.
|
|
31
|
+
- [PR #7581](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7581), commit `ed0f1e63e39e24e771b47cbe883f5925be9d45d6`, hardened the stable host and case-dialog loading path.
|
|
32
|
+
- At `origin/main@1bafb36f8f909274b96eef840655ffdc7e15f9d0`, `HomeHealthDialogHost.tsx:67-77` renders the active dialog beside the list children, and `HomeHealthActiveDialog.tsx:63-81` owns every case and visit action dialog.
|
|
33
|
+
|
|
34
|
+
A new investigation must stop as already fixed unless recurrence evidence post-dates those commits. The implementation matching the dry run's design is evidence that the causal document was actionable; it is not permission for this skill to open a PR.
|
|
35
|
+
|
|
36
|
+
## 1. Dossier
|
|
37
|
+
|
|
38
|
+
The spec was created by consolidating seven Home Health E2E files into one ten-step lifecycle test. Every later fix hardened the step where the failure last surfaced. None removed the product lifecycle churn shared by the failures.
|
|
39
|
+
|
|
40
|
+
| Prior ticket/PR/commit | What it blamed | What it changed | Recurrence evidence |
|
|
41
|
+
| ------------------------------------- | --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------- |
|
|
42
|
+
| STAFF-1329 / PR #6902 / `d6633c656e3` | Occurrence tab readiness | Added waits for occurrence tabs | Failure moved to visit-row lookup in STAFF-1369 |
|
|
43
|
+
| STAFF-1327 / PR #6903 / `f1bf003b6c2` | Visit-verification tab state | Changed product tab state and spec waits | Failure moved to visit-row and ROC lookup |
|
|
44
|
+
| STAFF-1369 / PR #6915 / `3a11b8effc8` | Visit-row render race | Added row-lookup retries and a negotiation tweak | Recurred as ROC visit lookup in STAFF-1495 |
|
|
45
|
+
| STAFF-1377 / PR #6930 / `07491124e72` | Remove-booked-worker assertion | Changed the dialog and assertion | Later failures surfaced in other dialogs |
|
|
46
|
+
| STAFF-1449 / PR #6987 / `53330b9c0ea` | Create transport errors | Added fail-fast handling and more waits | The family recurred; machinery grew without removing the cause |
|
|
47
|
+
| STAFF-1441 / PR #6990 / `51577fa1778` | Response-wait timing | Rewrote lifecycle response waits | STAFF-1495 recurred within days |
|
|
48
|
+
| STAFF-1495 / PR #7050 / `5ebd5ee839a` | ROC visit lookup | Hardened ROC-specific lookup | STAFF-1554 recurred at auth-refresh waits |
|
|
49
|
+
| STAFF-1554 / PR #7251 / `2f942662c37` | Auth refresh during waits | Added a 306-line response helper tolerant of refresh | STAFF-1582 recurred at close case |
|
|
50
|
+
| STAFF-1582 / PR #7316 / `28bdf114626` | Close-case flow | Reworked close-case steps | STAFF-1685 recurred with the identical title |
|
|
51
|
+
| STAFF-1601 / PR #7321 / `b09225ddc35` | Visit-verification modal state | Changed occurrence-table dialog behavior | STAFF-1673 recurred at cancel-visit dialog |
|
|
52
|
+
| STAFF-1673 / PR #7374 / `ac7c1b79ae5` | Cancel-visit dialog open | Added `toPass`, immediate-failure tracking, and diagnostics | STAFF-1682 recurred six days later at close case |
|
|
53
|
+
| STAFF-1685 / PR #7390 / `fe4e2cd8424` | Close case again | Added another large close-case hardening block | STAFF-1659 and STAFF-1682 still failed |
|
|
54
|
+
| STAFF-1789 / PR #7559 / `2b7ebc87c2d` | Backing-list refreshes while dialogs are open | Paused/canceled list observers and deferred invalidations behind `HomeHealthListRefetchGuard` with fault-injection tests | The PR documented that dialogs still lived below mapped cards; PR #7574 removed the 217-line guard and implemented the durable host |
|
|
55
|
+
| STAFF-1790 / PR #7574 / `e6818dabd95` | Dialog lifecycle coupled to mapped cards | Hoisted all desktop action dialogs and split the mega-spec | No post-merge recurrence found in the current-main recheck |
|
|
56
|
+
| PR #7581 / `ed0f1e63e39` | Remaining host loading/history edges | Hardened the stable host and case-dialog loading path | No post-merge recurrence found in the current-main recheck |
|
|
57
|
+
|
|
58
|
+
### Ruled-out prior hypotheses
|
|
59
|
+
|
|
60
|
+
Before the durable host in PR #7574, the next attempt had to rule out the twelve symptom fixes and the #7559 mitigation:
|
|
61
|
+
|
|
62
|
+
- Readiness and locator timing are incomplete explanations because the same family migrated across tabs, rows, auth waits, and several independent dialogs after those waits landed.
|
|
63
|
+
- Response-wait rewrites are incomplete because later signatures show DOM detachment after the response machinery was centralized.
|
|
64
|
+
- Preserving one dialog's state is incomplete because sibling dialogs under the same list-item lifecycle continued to unmount.
|
|
65
|
+
- More retries are falsified by recurrence after roughly one thousand net lines of retry and diagnostic apparatus accumulated in the spec and helpers.
|
|
66
|
+
- Pausing known list refreshes was incomplete because #7559's own residual-risk statement left the dialogs below mapped cards and exposed them to ancestor updates outside those queries.
|
|
67
|
+
|
|
68
|
+
### Latest recurrence signatures
|
|
69
|
+
|
|
70
|
+
- STAFF-1682: the close button repeatedly became unstable and detached from the DOM during `locator.click`.
|
|
71
|
+
- STAFF-1659: the cancel-visit dialog title never reached a stable visible state within 30 seconds.
|
|
72
|
+
|
|
73
|
+
Both signatures point to component lifecycle churn, not a slow assertion.
|
|
74
|
+
|
|
75
|
+
## 2. Causal chain
|
|
76
|
+
|
|
77
|
+
**Terminal cause:** Home Health action dialogs are React descendants of query-backed list items. The dialogs' own mutations invalidate and refresh those lists. Background refreshes and status changes reconcile, reorder, or remove the list item while its dialog is open, destabilizing or unmounting the dialog subtree.
|
|
78
|
+
|
|
79
|
+
```text
|
|
80
|
+
Playwright click or dialog-title assertion fails
|
|
81
|
+
โ the open dialog becomes unstable or disappears
|
|
82
|
+
โ query invalidation/background refetch returns a new case or visit list
|
|
83
|
+
โ CaseCard or VisitCard reconciles, reorders, or leaves the open list
|
|
84
|
+
โ the dialog subtree, owned by that list item, remounts or unmounts
|
|
85
|
+
โ terminal cause: dialog lifecycle is coupled to invalidated list-item lifecycle
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
### Ownership and historical code
|
|
89
|
+
|
|
90
|
+
- `useUpdateVisit.ts:66` invalidates the case-visits query after mutations.
|
|
91
|
+
- `useUpdateCase.ts:44-56` invalidates open and closed agency-case queries plus case-visits data.
|
|
92
|
+
- `useCaseVisits.ts` and `useAgencyCases.ts:69` use five-minute stale and ten-minute interval refresh behavior.
|
|
93
|
+
- `CaseVisits.tsx:42` sorts and maps visits on render.
|
|
94
|
+
- `CaseActionsMenu.tsx:92` renders `CaseCloseDialog` inside the case card's actions.
|
|
95
|
+
- `Visit/VisitActions.tsx:70` renders `CancelVisitDialog` inside the visit card's actions.
|
|
96
|
+
|
|
97
|
+
Groundtruth resolves service `admin-web-app` to repository `cbh-admin-frontend`; its listed owners are Synapse, Platform, Growth, Tempo, Alchemy, and Action. The repository registry owner is Platform. No backend service is needed to terminate this particular chain.
|
|
98
|
+
|
|
99
|
+
### Close-case chain
|
|
100
|
+
|
|
101
|
+
1. The test opens the close dialog and clicks Close.
|
|
102
|
+
2. A list invalidation or background refetch resolves while the dialog is open.
|
|
103
|
+
3. The cases array changes and the owning card rerenders, producing the observed unstable element.
|
|
104
|
+
4. The close mutation changes the case to closed.
|
|
105
|
+
5. The open-cases query returns without that case, unmounting the card and dialog, producing the observed detached element.
|
|
106
|
+
|
|
107
|
+
### Cancel-visit chain
|
|
108
|
+
|
|
109
|
+
1. The test opens the cancel dialog.
|
|
110
|
+
2. A prior visit mutation's invalidation resolves.
|
|
111
|
+
3. The visit list resorts or reconciles.
|
|
112
|
+
4. The owning visit card and its dialog remount.
|
|
113
|
+
5. The title disappears before it reaches a stable asserted state.
|
|
114
|
+
|
|
115
|
+
## 3. Root cause and confidence
|
|
116
|
+
|
|
117
|
+
Home Health action dialogs bind their lifecycle to `CaseCard` and `VisitCard` instances produced by refetching lists. Those lists are invalidated by the dialogs' mutations and refreshed independently. A refetch during an open dialog can reconcile or remove the owning item, destroying the dialog during user interaction.
|
|
118
|
+
|
|
119
|
+
This is a user-facing product race, not test-only behavior. A real administrator can lose an open dialog during background refresh; the long CI workflow merely expands the race window.
|
|
120
|
+
|
|
121
|
+
Confidence is **4/5**. The render ownership, invalidation path, refresh behavior, and failure signatures establish a terminated static chain. Fault injection is still required for 5/5.
|
|
122
|
+
|
|
123
|
+
## 4. Designed fix
|
|
124
|
+
|
|
125
|
+
### Root fix
|
|
126
|
+
|
|
127
|
+
Hoist Home Health case and visit action dialogs out of `CaseCard` and `VisitCard` into one stable dialog host mounted above the query-backed lists.
|
|
128
|
+
|
|
129
|
+
- Add a `HomeHealthDialogHost` at the page or dashboard shell.
|
|
130
|
+
- Make list items request `openDialog({ kind, entityId })` instead of owning modal state and rendering dialogs inline.
|
|
131
|
+
- Keep one dialog instance keyed by entity ID in a subtree unaffected by list reconciliation.
|
|
132
|
+
- Close only on explicit user action or mutation success.
|
|
133
|
+
|
|
134
|
+
### Defense in depth
|
|
135
|
+
|
|
136
|
+
While a Home Health dialog is open, pause optional focus/interval refresh churn or defer broad invalidation until the dialog closes. This reduces churn but does not replace the lifecycle fix.
|
|
137
|
+
|
|
138
|
+
### Cleanup after the root fix
|
|
139
|
+
|
|
140
|
+
Remove the `toPass` blocks, immediate-failure trackers, and open-dialog retry apparatus added by prior attempts once fault injection proves the stable host survives refreshes.
|
|
141
|
+
|
|
142
|
+
### Test decomposition
|
|
143
|
+
|
|
144
|
+
Split the ten-step mega-spec into smaller product-surface workflows with isolated seeded data, retaining one thin lifecycle smoke path if required. This reduces blast radius and gives each real mechanism a distinct fingerprint.
|
|
145
|
+
|
|
146
|
+
## 5. Fault-injection reproduction
|
|
147
|
+
|
|
148
|
+
1. Run the affected flow against a seeded case.
|
|
149
|
+
2. Delay the close mutation while forcing the open-cases query to refetch and resolve during the click.
|
|
150
|
+
3. Expect the current implementation to reproduce the detached-element signature.
|
|
151
|
+
4. For cancel visit, force the case-visits refetch to resolve while the dialog is open and expect the title to disappear.
|
|
152
|
+
5. Apply a prototype of the stable dialog host.
|
|
153
|
+
6. Repeat both injections and prove the dialogs survive.
|
|
154
|
+
7. Run the affected specs repeatedly with Playwright retries disabled.
|
|
155
|
+
|
|
156
|
+
The original dry run could not execute these steps because the environment lacked staging VPN access. Under the new skill, that precondition failure requires a provisioning ticket rather than continued investigation.
|
|
157
|
+
|
|
158
|
+
## 6. Observability fallback
|
|
159
|
+
|
|
160
|
+
If fault injection cannot yet run after access is provisioned, add a CI-visible dialog generation counter and emit an event whenever a Home Health dialog unmounts without explicit close or mutation success. Include dialog kind, entity ID, query keys invalidated in the preceding window, and a correlation ID exposed in the Playwright reporter.
|
|
161
|
+
|
|
162
|
+
The next occurrence should answer directly whether a list refetch caused the unexpected unmount.
|
|
163
|
+
|
|
164
|
+
## 7. Evidence appendix
|
|
165
|
+
|
|
166
|
+
- Chronic family: `33deef731a10` plus variants.
|
|
167
|
+
- Consolidation: [PR #6616](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6616), commit `afb53308a38`.
|
|
168
|
+
- Latest signatures: STAFF-1682 close-case detachment and STAFF-1659 cancel-dialog invisibility.
|
|
169
|
+
- Historical code checkout: `cbh-admin-frontend@fe4e2cd8424`.
|
|
170
|
+
- Current-main checkout: `cbh-admin-frontend@1bafb36f8f909274b96eef840655ffdc7e15f9d0`.
|
|
171
|
+
- Ownership query: `jq '.services[] | select(.repo == "cbh-admin-frontend")' groundtruth/registry/services.json`.
|
|
172
|
+
- Repository query: `jq --arg repo cbh-admin-frontend '.repos[] | select(.id == $repo)' groundtruth/registry/repos.json`.
|
|
173
|
+
- Current-main verification: `git log origin/main -- HomeHealthActiveDialog.tsx HomeHealthDialogHost.tsx`.
|
|
174
|
+
- Key historical product areas: `CaseDashboard`, `CaseActionsMenu`, `CaseVisits`, `VisitActions`, `useUpdateCase`, `useUpdateVisit`, `useAgencyCases`, and `useCaseVisits`.
|
|
175
|
+
- Prior PRs: [#6902](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6902), [#6903](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6903), [#6915](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6915), [#6930](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6930), [#6987](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6987), [#6990](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/6990), [#7050](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7050), [#7251](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7251), [#7316](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7316), [#7321](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7321), [#7374](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7374), [#7390](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7390), [#7559](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7559), [#7574](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7574), and [#7581](https://github.com/ClipboardHealth/cbh-admin-frontend/pull/7581).
|
|
176
|
+
- Historical limitation: the surviving document did not retain the exact Actions run URL, trace ID, or request ID. New `flaky-deep-dive` outputs must retain them when applicable.
|
|
177
|
+
- Historical output: Rocky's `hh-full-lifecycle.md`, created 2026-07-07.
|
|
@@ -0,0 +1,177 @@
|
|
|
1
|
+
#!/usr/bin/env bash
|
|
2
|
+
|
|
3
|
+
set -uo pipefail
|
|
4
|
+
|
|
5
|
+
AWS_PROFILE_NAME="${1:-${AWS_PROFILE:-sdlc}}"
|
|
6
|
+
STAGING_HEALTH_URL="${2:-https://apigateway.staging.clipboardhealth.org/api/healthCheck}"
|
|
7
|
+
|
|
8
|
+
passed=0
|
|
9
|
+
failed=0
|
|
10
|
+
|
|
11
|
+
print_pass() {
|
|
12
|
+
printf 'PASS: %s\n' "$1"
|
|
13
|
+
passed=$((passed + 1))
|
|
14
|
+
}
|
|
15
|
+
|
|
16
|
+
print_fail() {
|
|
17
|
+
printf 'FAIL: %s\n' "$1"
|
|
18
|
+
failed=$((failed + 1))
|
|
19
|
+
}
|
|
20
|
+
|
|
21
|
+
print_failure_details() {
|
|
22
|
+
local command_text="$1"
|
|
23
|
+
local exit_code="$2"
|
|
24
|
+
local output="$3"
|
|
25
|
+
|
|
26
|
+
printf ' Command: %s\n' "$command_text"
|
|
27
|
+
printf ' Exit: %s\n' "$exit_code"
|
|
28
|
+
if [[ -n "$output" ]]; then
|
|
29
|
+
printf ' Output:\n'
|
|
30
|
+
printf '%s\n' "$output" | sed -n '1,6p' | sed 's/^/ /'
|
|
31
|
+
fi
|
|
32
|
+
}
|
|
33
|
+
|
|
34
|
+
printf 'Checking flaky deep-dive prerequisites...\n'
|
|
35
|
+
|
|
36
|
+
timeout_command=""
|
|
37
|
+
if command -v gtimeout >/dev/null 2>&1; then
|
|
38
|
+
timeout_command="gtimeout"
|
|
39
|
+
elif command -v timeout >/dev/null 2>&1; then
|
|
40
|
+
timeout_command="timeout"
|
|
41
|
+
else
|
|
42
|
+
print_fail "No timeout utility is installed."
|
|
43
|
+
printf ' Remediation: install GNU coreutils so remote credential checks fail within a bounded time.\n'
|
|
44
|
+
fi
|
|
45
|
+
|
|
46
|
+
if ! command -v pup >/dev/null 2>&1; then
|
|
47
|
+
print_fail "pup is not installed."
|
|
48
|
+
printf ' Remediation: install pup, then authenticate with pup auth login or provide DD_API_KEY and DD_APP_KEY.\n'
|
|
49
|
+
elif [[ -n "$timeout_command" ]]; then
|
|
50
|
+
trace_output="$(
|
|
51
|
+
"$timeout_command" 30 pup traces search \
|
|
52
|
+
--query='service:cbh-backend-main env:staging' \
|
|
53
|
+
--from=15m \
|
|
54
|
+
--limit=1 \
|
|
55
|
+
--read-only \
|
|
56
|
+
--no-agent \
|
|
57
|
+
--jq='.data | length' 2>&1
|
|
58
|
+
)"
|
|
59
|
+
trace_exit_code=$?
|
|
60
|
+
if [[ "$trace_exit_code" -eq 0 ]]; then
|
|
61
|
+
print_pass "Datadog APM read access works through pup."
|
|
62
|
+
else
|
|
63
|
+
print_fail "pup cannot read staging APM spans."
|
|
64
|
+
print_failure_details \
|
|
65
|
+
"pup traces search --query='service:cbh-backend-main env:staging' --from=15m --limit=1" \
|
|
66
|
+
"$trace_exit_code" \
|
|
67
|
+
"$trace_output"
|
|
68
|
+
printf ' Remediation: run pup auth login, provide DD_API_KEY and DD_APP_KEY, or request APM read access for 401/403 failures.\n'
|
|
69
|
+
fi
|
|
70
|
+
|
|
71
|
+
logs_output="$(
|
|
72
|
+
"$timeout_command" 30 pup logs search \
|
|
73
|
+
--query='service:cbh-backend-main env:staging' \
|
|
74
|
+
--from=15m \
|
|
75
|
+
--limit=1 \
|
|
76
|
+
--read-only \
|
|
77
|
+
--no-agent \
|
|
78
|
+
--jq='.data | length' 2>&1
|
|
79
|
+
)"
|
|
80
|
+
logs_exit_code=$?
|
|
81
|
+
if [[ "$logs_exit_code" -eq 0 ]]; then
|
|
82
|
+
print_pass "Datadog log read access works through pup."
|
|
83
|
+
else
|
|
84
|
+
print_fail "pup cannot read staging logs."
|
|
85
|
+
print_failure_details \
|
|
86
|
+
"pup logs search --query='service:cbh-backend-main env:staging' --from=15m --limit=1" \
|
|
87
|
+
"$logs_exit_code" \
|
|
88
|
+
"$logs_output"
|
|
89
|
+
printf ' Remediation: run pup auth login, provide DD_API_KEY and DD_APP_KEY, or request log read access for 401/403 failures.\n'
|
|
90
|
+
fi
|
|
91
|
+
fi
|
|
92
|
+
|
|
93
|
+
if ! command -v curl >/dev/null 2>&1; then
|
|
94
|
+
print_fail "curl is not installed."
|
|
95
|
+
printf ' Remediation: install curl and retry while connected to the non-production VPN.\n'
|
|
96
|
+
else
|
|
97
|
+
staging_output="$(
|
|
98
|
+
curl \
|
|
99
|
+
--silent \
|
|
100
|
+
--show-error \
|
|
101
|
+
--write-out '\n%{http_code}' \
|
|
102
|
+
--connect-timeout 10 \
|
|
103
|
+
--max-time 20 \
|
|
104
|
+
"$STAGING_HEALTH_URL" 2>&1
|
|
105
|
+
)"
|
|
106
|
+
curl_exit_code=$?
|
|
107
|
+
staging_status="$(printf '%s\n' "$staging_output" | tail -n 1)"
|
|
108
|
+
staging_body="$(printf '%s\n' "$staging_output" | sed '$d')"
|
|
109
|
+
|
|
110
|
+
if [[ "$curl_exit_code" -eq 0 && "$staging_status" == "200" ]] &&
|
|
111
|
+
printf '%s\n' "$staging_body" | grep -Eq '"message"[[:space:]]*:[[:space:]]*"OK"'; then
|
|
112
|
+
print_pass "Staging health is reachable through the VPN (HTTP 200, message OK)."
|
|
113
|
+
else
|
|
114
|
+
print_fail "Staging health is not reachable through the VPN."
|
|
115
|
+
print_failure_details \
|
|
116
|
+
"curl --connect-timeout 10 --max-time 20 $STAGING_HEALTH_URL" \
|
|
117
|
+
"$curl_exit_code" \
|
|
118
|
+
"$staging_output"
|
|
119
|
+
printf ' Remediation: connect the Clipboard Health non-production VPN. If this environment cannot use the VPN, request a VPN-capable investigation environment.\n'
|
|
120
|
+
fi
|
|
121
|
+
fi
|
|
122
|
+
|
|
123
|
+
if ! command -v aws >/dev/null 2>&1; then
|
|
124
|
+
print_fail "AWS CLI is not installed."
|
|
125
|
+
printf ' Remediation: install AWS CLI v2 and configure the sdlc SSO profile.\n'
|
|
126
|
+
else
|
|
127
|
+
aws_identity_output="$(
|
|
128
|
+
aws sts get-caller-identity \
|
|
129
|
+
--profile "$AWS_PROFILE_NAME" \
|
|
130
|
+
--query Account \
|
|
131
|
+
--output text \
|
|
132
|
+
--cli-connect-timeout 10 \
|
|
133
|
+
--cli-read-timeout 20 2>&1
|
|
134
|
+
)"
|
|
135
|
+
aws_identity_exit_code=$?
|
|
136
|
+
|
|
137
|
+
if [[ "$aws_identity_exit_code" -eq 0 && -n "$aws_identity_output" && "$aws_identity_output" != "None" ]]; then
|
|
138
|
+
print_pass "AWS credentials are active for profile $AWS_PROFILE_NAME (account $aws_identity_output)."
|
|
139
|
+
|
|
140
|
+
aws_read_output="$(
|
|
141
|
+
aws cloudwatch list-metrics \
|
|
142
|
+
--profile "$AWS_PROFILE_NAME" \
|
|
143
|
+
--region us-west-2 \
|
|
144
|
+
--max-items 1 \
|
|
145
|
+
--output json \
|
|
146
|
+
--cli-connect-timeout 10 \
|
|
147
|
+
--cli-read-timeout 20 2>&1
|
|
148
|
+
)"
|
|
149
|
+
aws_read_exit_code=$?
|
|
150
|
+
if [[ "$aws_read_exit_code" -eq 0 ]]; then
|
|
151
|
+
print_pass "AWS read access works for profile $AWS_PROFILE_NAME."
|
|
152
|
+
else
|
|
153
|
+
print_fail "AWS credentials lack the staging read access required by the deep-dive track."
|
|
154
|
+
print_failure_details \
|
|
155
|
+
"aws cloudwatch list-metrics --profile $AWS_PROFILE_NAME --region us-west-2 --max-items 1" \
|
|
156
|
+
"$aws_read_exit_code" \
|
|
157
|
+
"$aws_read_output"
|
|
158
|
+
printf ' Remediation: request staging read permissions, then rerun the preflight.\n'
|
|
159
|
+
fi
|
|
160
|
+
else
|
|
161
|
+
print_fail "AWS credentials are unavailable for profile $AWS_PROFILE_NAME."
|
|
162
|
+
print_failure_details \
|
|
163
|
+
"aws sts get-caller-identity --profile $AWS_PROFILE_NAME" \
|
|
164
|
+
"$aws_identity_exit_code" \
|
|
165
|
+
"$aws_identity_output"
|
|
166
|
+
printf ' Remediation: run aws sso login --profile %s. If the profile is missing, request staging read access and configure it first.\n' "$AWS_PROFILE_NAME"
|
|
167
|
+
fi
|
|
168
|
+
fi
|
|
169
|
+
|
|
170
|
+
printf '\nSummary: %s passed, %s failed.\n' "$passed" "$failed"
|
|
171
|
+
|
|
172
|
+
if [[ "$failed" -gt 0 ]]; then
|
|
173
|
+
printf '\nDo not proceed with the deep dive. File a provisioning ticket linked to the chronic flake ticket and include the failed checks above.\n'
|
|
174
|
+
exit 1
|
|
175
|
+
fi
|
|
176
|
+
|
|
177
|
+
printf 'All credential preconditions are satisfied.\n'
|
|
@@ -68,7 +68,7 @@ Do not merge clusters merely because failures share a run. Same-run failures can
|
|
|
68
68
|
4. **Multi-ticket cluster** โ pick the canonical (richest evidence: most failures, clearest artifacts). Comment the cluster manifest on it: member IDs, shared mechanism, evidence tier from Phase 3, and the marker `<!-- flaky-triage: cluster=<short-slug> -->`. Close the other members as Duplicate of the canonical (D2 format). Release the canonical.
|
|
69
69
|
5. **Singleton or uncertain** โ release as-is with a brief `<!-- flaky-triage: singleton -->` comment.
|
|
70
70
|
|
|
71
|
-
Before any action above releases a canonical ticket, search Linear for the fingerprint family with the `flaky-implementation` label across all states and without a date-window restriction, then count unique prior implementation ticket IDs. At **โฅ3 prior implementation tickets**, do not release the canonical for normal dispatch and leave it in `Triage` unassigned. Create the Linear `chronic` label first if it does not exist. Apply the `chronic` label and
|
|
71
|
+
Before any action above releases a canonical ticket, search Linear for the fingerprint family with the `flaky-implementation` label across all states and without a date-window restriction, then count unique prior implementation ticket IDs. At **โฅ3 prior implementation tickets**, do not release the canonical for normal dispatch and leave it in `Triage` unassigned. Create the Linear `chronic` label first if it does not exist. Apply the `chronic` label and route the family to the credential-checked, dossier-first `flaky-deep-dive` skill. Add a ticket comment with the marker `<!-- flaky-triage: chronic -->` and dossier pointers listing every prior implementation ticket ID โ but first search the ticket's existing comments for that exact marker and skip posting when it is already present; that absence check is what makes the comment idempotent across retries and re-runs. This โฅ3-ticket count is a cheap screen staged before any diagnosis exists; the critic separately applies a sharper **โฅ2 failed merged fixes** test (see the flaky-critic rubric's Prior-attempts amendment) with the same chronic routing. The thresholds differ by design โ by the evidence available at each stage โ not by accident.
|
|
72
72
|
|
|
73
73
|
**Release** = set state to `Todo` and assign to the Linear API user (the viewer). Before release, ensure the ticket satisfies the Groundcrew eligibility contract from `../create-groundcrew-ticket/SKILL.md`: repository text in the description, exactly one `agent-*` label, leaf issue, and no non-terminal blockers unless intentionally blocked. Assignment + Todo is what makes groundcrew dispatch it; it also serves as the lease โ released tickets never re-enter the queue.
|
|
74
74
|
|