@clipboard-health/ai-rules 2.37.3 β 2.38.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/package.json +1 -1
- package/scripts/toErrorMessage.js +1 -1
- package/skills/cb-babysit/SKILL.md +4 -4
- package/skills/cb-babysit/scripts/_sentinel.sh +1 -1
- package/skills/cb-review/SKILL.md +8 -5
- package/skills/cb-ship/SKILL.md +3 -2
- package/skills/cb-ship/references/pr-template.md +1 -1
- package/skills/cb-work/SKILL.md +12 -3
- package/skills/flaky-critic/references/rubric.md +16 -1
- package/skills/flaky-debug/references/plan.md +22 -9
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
"use strict";
|
|
2
2
|
Object.defineProperty(exports, "__esModule", { value: true });
|
|
3
3
|
exports.toErrorMessage = toErrorMessage;
|
|
4
|
-
// Prefer stack over message for Error instances to preserve debugging context
|
|
4
|
+
// Prefer stack over message for Error instances to preserve debugging context!
|
|
5
5
|
function toErrorMessage(error) {
|
|
6
6
|
return error instanceof Error ? (error.stack ?? error.message) : String(error);
|
|
7
7
|
}
|
|
@@ -19,9 +19,9 @@ Resolve bundled "./scripts" paths relative to SKILL.md.
|
|
|
19
19
|
|
|
20
20
|
The skill uses two sentinels with visible footer lines.
|
|
21
21
|
|
|
22
|
-
**Addressed sentinel**: `<sub>π€ <code>cb-babysit:addressed v1 core@3.
|
|
22
|
+
**Addressed sentinel**: `<sub>π€ <code>cb-babysit:addressed v1 core@3.17.0</code></sub>`. Appended on its own line at the end of every reply the skill posts so re-runs know which threads and review-body comments are already handled. Dedupe also recognizes the legacy `babysit-pr:addressed v1` prefix from before this skill was renamed.
|
|
23
23
|
|
|
24
|
-
**Follow-up sentinel**: `<sub>π€ <code>cb-babysit:followup v1 core@3.
|
|
24
|
+
**Follow-up sentinel**: `<sub>π€ <code>cb-babysit:followup v1 core@3.17.0</code></sub>`. Attached to replies that defer an out-of-scope comment as a tracked follow-up. The sentinel is additive: the post-reply scripts still append the `addressed` sentinel at the end.
|
|
25
25
|
|
|
26
26
|
**Sentinel recency rules.** The script emits a per-thread `activityState` with three values. Step 6a owns the handling rules for each state.
|
|
27
27
|
|
|
@@ -247,7 +247,7 @@ Body templates (the script appends the `addressed` sentinel if missing):
|
|
|
247
247
|
- **Agree**: `Addressed in <commit-url>. <one-line what-changed>.`
|
|
248
248
|
- **Disagree**: `Leaving current behavior. <reasoning>.`
|
|
249
249
|
- **Already fixed**: `Already handled by <commit-url-or-file:line>. <brief pointer>.`
|
|
250
|
-
- **Defer**: `Out of scope for this PR; this looks like follow-up work rather than something introduced or required by this change. <one-line rationale or pointer if useful>.\n\n<sub>π€ <code>cb-babysit:followup v1 core@3.
|
|
250
|
+
- **Defer**: `Out of scope for this PR; this looks like follow-up work rather than something introduced or required by this change. <one-line rationale or pointer if useful>.\n\n<sub>π€ <code>cb-babysit:followup v1 core@3.17.0</code></sub>`
|
|
251
251
|
|
|
252
252
|
For Defer replies, include the follow-up sentinel on its own line as shown. The script will append the `addressed` sentinel after it on its own line, so the final body ends with the follow-up sentinel followed by a blank line followed by the `addressed` sentinel. `grep cb-babysit:followup` finds the deferral and `grep cb-babysit:addressed` still marks the thread handled for dedupe.
|
|
253
253
|
|
|
@@ -263,7 +263,7 @@ The PR-level summary should:
|
|
|
263
263
|
|
|
264
264
|
- Group by source. Use `## Review-body findings` for step-7 work and `## Conversation-tab comments` for step-6b work. Omit a section if its list is empty.
|
|
265
265
|
- Inside each section, group verdicts under **Agree / Disagree / Already fixed / Deferred (out of scope)** subheadings. Omit a subheading if its list is empty.
|
|
266
|
-
- Under **Deferred (out of scope)**, list each deferred item as a bullet, followed on its own line by `<sub>π€ <code>cb-babysit:followup v1 core@3.
|
|
266
|
+
- Under **Deferred (out of scope)**, list each deferred item as a bullet, followed on its own line by `<sub>π€ <code>cb-babysit:followup v1 core@3.17.0</code></sub>` so grep catches them individually.
|
|
267
267
|
- Include the commit URL for fixes.
|
|
268
268
|
- End with a fenced fingerprint block listing every current fingerprint (addressed and deferred) one per line. Include both `reviewBodyComments[].fingerprint` (whole-body, one per automated review) and `activeIssueComments[].fingerprint` (per Conversation-tab comment). Future runs dedupe by matching these against `priorBabysitSentinels`.
|
|
269
269
|
|
|
@@ -10,7 +10,7 @@
|
|
|
10
10
|
|
|
11
11
|
SENTINEL_PREFIX='cb-babysit:addressed v1 '
|
|
12
12
|
LEGACY_SENTINEL_PREFIX='babysit-pr:addressed v1 '
|
|
13
|
-
SENTINEL='<sub>π€ <code>cb-babysit:addressed v1 core@3.
|
|
13
|
+
SENTINEL='<sub>π€ <code>cb-babysit:addressed v1 core@3.17.0</code></sub>'
|
|
14
14
|
|
|
15
15
|
# Bot author allowlist (JSON array literal). Used by unresolvedPrComments.sh
|
|
16
16
|
# as a fallback when GraphQL's `author.__typename == "Bot"` misses a GitHub
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: cb-review
|
|
3
|
-
description: Code review of a diff, branch, or PR, with findings posted as anchored PR comments. Use when the user asks to review a diff, branch, or PR, asks to check a change against its ticket/spec/PRD, or runs /cb-review [pr-number-or-url] [--effort low|high] [--report].
|
|
4
|
-
argument-hint: "[pr-number-or-url] [--effort low|high] [--report]"
|
|
3
|
+
description: Code review of a diff, branch, or PR, with findings posted as anchored PR comments. Use when the user asks to review a diff, branch, or PR, asks to check a change against its ticket/spec/PRD, or runs /cb-review [pr-number-or-url] [--effort low|high] [--report] [--spec-context <path-or-reference-or-text>].
|
|
4
|
+
argument-hint: "[pr-number-or-url] [--effort low|high] [--report] [--spec-context <path-or-reference-or-text>]"
|
|
5
5
|
---
|
|
6
6
|
|
|
7
7
|
# CB Review
|
|
@@ -17,6 +17,7 @@ Review a diff against one rubric, filter to the few findings worth raising, gate
|
|
|
17
17
|
- `/cb-review <pr-number-or-url>` β review that PR without checking it out; forces reviewer mode. Accepts a bare number (current repo) or full GitHub URL (identifies owner/repo).
|
|
18
18
|
- `--effort low|high` β pick the engine explicitly. Phrases also select: "quick"/"fast" β low; "deep"/"thorough"/"multi-perspective" β high.
|
|
19
19
|
- `--report` β non-interactive: stop after Synthesize and return the findings to the caller. No user gates, no posting, no implementing. For agent callers (e.g. cb-ship's review step).
|
|
20
|
+
- `--spec-context <path-or-reference-or-text>` β use the value only as the originating source of truth, never as the PR selector. Read a local path in full, fetch a ticket or spec reference, or preserve natural-language text verbatim. Use the resolved content for the Spec lens instead of reconstructing intent from commit or PR prose.
|
|
20
21
|
|
|
21
22
|
**Effort auto-select** (no flag, no phrase): `high` when the diff exceeds 20 changed files or 600 changed lines, else `low`. Before reviewing, print one line β `Effort: <low|high> (<N> files, <M> lines; override with --effort <other>)` β so the user can interrupt.
|
|
22
23
|
|
|
@@ -50,7 +51,9 @@ Determine **mode**:
|
|
|
50
51
|
- PR exists and authors differ β **reviewer mode**.
|
|
51
52
|
- No PR β **author mode**.
|
|
52
53
|
|
|
53
|
-
**Persistence:** both efforts persist for subagents into a fresh per-run directory β `RUN_DIR=$(mktemp -d "${TMPDIR:-/tmp}/cb-review.XXXXXX")` β so concurrent sessions never clobber each other: diff β `$RUN_DIR/diff.patch`, context β `$RUN_DIR/context.md`, changed files β `$RUN_DIR/files.txt`, metadata (PR number/url/base/author, viewer, head SHA, owner/repo, mode, `context_ref`) β `$RUN_DIR/meta.json`.
|
|
54
|
+
**Persistence:** both efforts persist for subagents into a fresh per-run directory β `RUN_DIR=$(mktemp -d "${TMPDIR:-/tmp}/cb-review.XXXXXX")` β so concurrent sessions never clobber each other: diff β `$RUN_DIR/diff.patch`, context β `$RUN_DIR/context.md`, changed files β `$RUN_DIR/files.txt`, metadata (PR number/url/base/author, viewer, head SHA, owner/repo, mode, `context_ref`, `spec_source`) β `$RUN_DIR/meta.json`.
|
|
55
|
+
|
|
56
|
+
When the caller supplied source-of-truth context, include it verbatim in `$RUN_DIR/context.md` and set `spec_source` so every dispatched reviewer uses it.
|
|
54
57
|
|
|
55
58
|
## Freshness preflight (mandatory before reading code)
|
|
56
59
|
|
|
@@ -121,7 +124,7 @@ Walk the changed-file list. Activate lenses that match:
|
|
|
121
124
|
- **Security** triggers on: `routes/`, `controllers/`, `middleware*/`, files matching `auth*`/`*permission*`/`*acl*`/`*token*`/`*session*`, response serializers, OpenAPI/contract definitions, new API endpoint files.
|
|
122
125
|
- **Database** triggers on: `migrations/`, `*.sql`, files matching `schema*`, Mongoose/Prisma model files (`models/`, `*.model.ts`, `*.schema.ts`), repository/DAL files, query builders.
|
|
123
126
|
- **Frontend** triggers on: `*.tsx`, `*.jsx`, `*.css`, `*.scss`, `pages/`, `components/`, `hooks/`, or anything importing from `react`, `@tanstack/react-query`, or a design-system package.
|
|
124
|
-
- **Spec** triggers when a spec source exists. Look in order: (1) issue/ticket references in the PR body or commit messages (`#123`, `Closes #45`, Linear/Jira keys) β fetch via `gh` or the tracker; (
|
|
127
|
+
- **Spec** triggers when a spec source exists. Look in order: (1) source-of-truth context supplied by the caller; (2) issue/ticket references in the PR body or commit messages (`#123`, `Closes #45`, Linear/Jira keys) β fetch via `gh` or the tracker; (3) a plan/PRD file under `docs/`, `specs/`, or `.scratch/` matching the branch or feature name. Nothing found β skip the lens and note "no spec available" in Summary.
|
|
125
128
|
|
|
126
129
|
Always-on lenses: **Engineering**, **Minimalism**, **Conventions**, **AntiSlop**.
|
|
127
130
|
|
|
@@ -131,7 +134,7 @@ The rubric β severity ladder, `failure_mode` contract, do-not-raise list, NIT
|
|
|
131
134
|
|
|
132
135
|
### Low effort
|
|
133
136
|
|
|
134
|
-
Dispatch **one** reviewer subagent β fresh eyes on the diff, and the bulk content stays out of your context. Its prompt carries the persisted file paths, the absolute path to references/review-rubric.md with the instruction to read it in full, the active lens list, and the two contracts from multi-agent.md Β§Dispatch mechanics (context-read, cross-repo evidence) with `<context_ref>` substituted and the moderator/Round-2 sentence replaced by: emit `evidence_required` findings capped at MAJOR; the dispatching agent resolves them in the Filter's cross-repo audit. Finding ids use an `R` prefix in place of roster letters. The subagent walks the diff, applying every active lens β the walk is done only when every hunk has been read under each active lens. Exhaustive reading, selective output: it returns _the smallest number of high-signal findings_ in the Round 1 output shape (multi-agent.md Β§Round 1), flagging anything that needs deeper investigation than it can do confidently rather than guessing. If the host cannot run subagents, do that same single pass yourself inline.
|
|
137
|
+
Dispatch **one** reviewer subagent β fresh eyes on the diff, and the bulk content stays out of your context. Its prompt carries the persisted file paths, the absolute path to references/review-rubric.md with the instruction to read it in full, the active lens list, the spec source when the Spec lens is active, and the two contracts from multi-agent.md Β§Dispatch mechanics (context-read, cross-repo evidence) with `<context_ref>` substituted and the moderator/Round-2 sentence replaced by: emit `evidence_required` findings capped at MAJOR; the dispatching agent resolves them in the Filter's cross-repo audit. Finding ids use an `R` prefix in place of roster letters. The subagent walks the diff, applying every active lens β the walk is done only when every hunk has been read under each active lens. Exhaustive reading, selective output: it returns _the smallest number of high-signal findings_ in the Round 1 output shape (multi-agent.md Β§Round 1), flagging anything that needs deeper investigation than it can do confidently rather than guessing. If the host cannot run subagents, do that same single pass yourself inline.
|
|
135
138
|
|
|
136
139
|
### High effort
|
|
137
140
|
|
package/skills/cb-ship/SKILL.md
CHANGED
|
@@ -1,12 +1,13 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: cb-ship
|
|
3
3
|
description: Ship changes. Simplify the diff, commit, review, push, and open or update a PR. Use when the user says 'ship it', 'commit and push', or wants a PR created or updated.
|
|
4
|
-
argument-hint: "[--draft]"
|
|
4
|
+
argument-hint: "[--draft] [--spec-context <path-or-reference-or-text>]"
|
|
5
5
|
---
|
|
6
6
|
|
|
7
7
|
## Setup
|
|
8
8
|
|
|
9
9
|
- If `gh auth status` fails, stop and tell the user.
|
|
10
|
+
- Treat `--spec-context` as source-of-truth input for review, never as a PR selector. Preserve and forward its value unchanged; `cb-review` owns resolution.
|
|
10
11
|
- If `git fetch` or `git push` fails with public key or agent errors, retry the same operation with `git -c credential.helper='!gh auth git-credential'` while preserving the original remote and branch mapping. For `git fetch`, keep the configured remote (e.g., `git -c credential.helper='!gh auth git-credential' fetch origin <same arguments>`) so `origin/*` refs update; for `git push`, use the same destination and branch mapping, substituting `https://github.com/<org>/<repo>.git` only when the configured remote URL is SSH.
|
|
11
12
|
- If `git rev-parse --verify origin/HEAD` fails, `origin/HEAD` is unset. Stop and tell the user to run `git remote set-head origin -a`.
|
|
12
13
|
- If `git status --short`, `git log --oneline origin/HEAD..HEAD`, and `gh pr view --json url --jq .url 2>/dev/null` are all empty, stop and reply "nothing to ship."
|
|
@@ -18,7 +19,7 @@ Resolve bundled "./references" paths relative to SKILL.md.
|
|
|
18
19
|
1. Create a new branch if on the default branch (e.g., `feat/add-user-validation`, `fix/null-check-in-parser`).
|
|
19
20
|
2. Use ./references/simplify.md on the full PR diff: `git diff $(git merge-base HEAD origin/HEAD)..HEAD` and any uncommitted changes.
|
|
20
21
|
3. Inspect `git status --short`, identify intended files, ask if ambiguous, then git add them. If uncommitted changes, create a conventional commit with `git commit --no-gpg-sign`.
|
|
21
|
-
4. Invoke the `cb-review` skill with `--effort low --report`
|
|
22
|
+
4. Invoke the `cb-review` skill with `--effort low --report`, forwarding `--spec-context` unchanged when supplied so the Spec lens runs against the actual request. Triage each returned finding yourself: if it's real and in scope, apply it, rerun the repo's relevant checks, and commit; otherwise dismiss it with a one-line reason for the output. Skip this step when the session already ran cb-review over these same changes and the same source of truth.
|
|
22
23
|
5. Push changes to origin.
|
|
23
24
|
6. Create or update the PR using ./references/pr-template.md:
|
|
24
25
|
a. If the host exposes a session ID (e.g., CODEX_THREAD_ID, CLAUDE_CODE_SESSION_ID), include the resume command in the PR body in backticks: ``Agent session: `codex resume <id>` `` or ``Agent session: `claude --resume <id>` ``. Preserve existing `Agent session:` lines, append only.
|
|
@@ -17,5 +17,5 @@ Concisely explain the user intent from session history and the meaningful behavi
|
|
|
17
17
|
|
|
18
18
|
Optional, don't fabricate: ticket links, rollout plan, residual risk, or specific areas for reviewers to focus.
|
|
19
19
|
|
|
20
|
-
<sub>π€ <code>cb-ship:created v1 core@3.
|
|
20
|
+
<sub>π€ <code>cb-ship:created v1 core@3.17.0</code></sub>
|
|
21
21
|
```
|
package/skills/cb-work/SKILL.md
CHANGED
|
@@ -27,12 +27,21 @@ The plan or request is the source of truth for scope:
|
|
|
27
27
|
|
|
28
28
|
Run the relevant checks after each meaningful unit of work, not only at the end.
|
|
29
29
|
|
|
30
|
+
For behavior changes in repositories with automated tests, use vertical red-green slices:
|
|
31
|
+
|
|
32
|
+
1. Derive the public test seam from the source of truth. If the seam is unclear and the choice would materially affect scope or architecture, stop and ask; otherwise state the seam in a progress update and proceed.
|
|
33
|
+
2. Write one behavior test through that seam. Avoid private interfaces, internal-collaborator assertions, and expected values computed with the implementation's algorithm.
|
|
34
|
+
3. Run the focused test and confirm it fails for the intended reason.
|
|
35
|
+
4. Implement only enough to pass, rerun the focused test, and repeat one slice at a time.
|
|
36
|
+
|
|
37
|
+
Mock only at system boundaries such as third-party APIs, time, randomness, or external I/O. Prefer real controlled collaborators, including a test database when practical. Run the narrowest applicable typecheck regularly; if the repository exposes only a slow workspace-wide typecheck, leave it to CI unless the plan or user requires it. For documentation, metadata, or other changes without runtime behavior, do not invent a test; use the relevant validation instead.
|
|
38
|
+
|
|
30
39
|
## Validate
|
|
31
40
|
|
|
32
|
-
If the plan names specific checks, use those
|
|
41
|
+
If the plan names specific checks, use those. Otherwise find relevant validation commands in, e.g., `AGENTS.md`, `CLAUDE.md`, `CONTRIBUTING.md`, or pre-commit/pre-push hooks. Prefer checks scoped to the touched projects or files; skip clearly slow, resource-intensive, or CI-only suites unless the plan or user requires them.
|
|
33
42
|
|
|
34
|
-
Done when every check passes or the user has explicitly accepted a failure.
|
|
43
|
+
Done when every selected check passes or the user has explicitly accepted a failure.
|
|
35
44
|
|
|
36
45
|
## Hand off
|
|
37
46
|
|
|
38
|
-
Invoke the `cb-ship` skill, passing `--draft` if it was passed to this skill. Relay `cb-ship`'s reply to the user.
|
|
47
|
+
Invoke the `cb-ship` skill, passing `--draft` if it was passed to this skill. Pass `--spec-context` with the source of truth: the resolved absolute plan path, ticket or spec reference, or the natural-language implementation request. Preserve the original context instead of relying on the eventual commit or PR description to reconstruct it. Relay `cb-ship`'s reply to the user.
|
|
@@ -8,7 +8,7 @@ This rubric turns Rocky's observed review standards into checkable rules for fla
|
|
|
8
8
|
|
|
9
9
|
Findings from automated reviewers (Mendral, CodeRabbit, etc.) are **advisory inputs, not binding rules**: the critic evaluates each on rubric merits and may decline one with stated reasoning, exactly as cb-babysit's Disagree replies do.
|
|
10
10
|
|
|
11
|
-
**Statement-missing vs substance-missing (backtest amendment, 2026-06-11):** B5/B6/B7 each require an explicit statement in the plan (dedup-check results, confidence score plus observability section, current-main status). When the _statement_ is absent but the violation is not demonstrated β no actual duplicate found, no actual low-confidence code fix, no actual staleness β the verdict is reject with disposition **amend-and-resubmit**: one bounce asking for exactly the missing sections, never escalation. A demonstrated violation (a real duplicate, a real β€2/5 code fix) is a substantive reject. The 2026-06-11 backtest showed 6 of 7 false rejections were statement-missing on plans predating these conventions; live plans carry the sections by template, but the distinction stays load-bearing for the agreement metric.
|
|
11
|
+
**Statement-missing vs substance-missing (backtest amendment, 2026-06-11):** B5/B6/B7/B8 each require an explicit statement in the plan (dedup-check results, confidence score plus observability section, current-main status, causal-chain section). When the _statement_ is absent but the violation is not demonstrated β no actual duplicate found, no actual low-confidence code fix, no actual staleness, no demonstrably symptom-terminated diagnosis β the verdict is reject with disposition **amend-and-resubmit**: one bounce asking for exactly the missing sections, never escalation. A demonstrated violation (a real duplicate, a real β€2/5 code fix, a diagnosis that demonstrably stops at a symptom) is a substantive reject. The 2026-06-11 backtest showed 6 of 7 false rejections were statement-missing on plans predating these conventions; live plans carry the sections by template, but the distinction stays load-bearing for the agreement metric.
|
|
12
12
|
|
|
13
13
|
## 2. Banned patterns
|
|
14
14
|
|
|
@@ -72,6 +72,21 @@ Implementing a plan against code that has changed on `main`, or for a flake alre
|
|
|
72
72
|
|
|
73
73
|
**Staleness clause (backtest amendment, 2026-06-11):** a plan more than ~14 days old, or one whose fingerprint has new sightings since the plan was written, requires re-verification against current main before approval β treat unverified staleness as B7. (Two backtest "rejections" were plans that sat unworked until a sweep canceled them; freshness is part of a plan being gateable.)
|
|
74
74
|
|
|
75
|
+
### B8 β Unterminated causal chain (symptom-level diagnosis)
|
|
76
|
+
|
|
77
|
+
Proposing a code fix from a diagnosis that stops at the failure symptom β "the request 500ed/timed out," "setup was throttled," "the element never appeared" β without establishing _why_ at a specific cause in the owning code, config, or infrastructure.
|
|
78
|
+
|
|
79
|
+
**Why (amendment, 2026-07-07):** post-Done recurrence is the pipeline's dominant failure mode: in the Jun 2βJul 6 window, 21 fingerprint families re-flaked after a Done closure, and the fullLifecycle family (33deef731a10) was closed Done six times. Each recurrence traces to a diagnosis that dead-ended at the repo boundary or at a status code and patched the symptom locally. Local-symptom plans are the cheapest to produce, so as long as the gate accepts them, agents keep producing them; this rule makes boundary-crossing the only way to pass.
|
|
80
|
+
|
|
81
|
+
**Required structure:** the plan must contain a **Causal chain** section: failing assertion β app/UI state β network/trace/log evidence (trace ID, or run-window log queries when the causal event is outside the test's request path: seeding, deploys, CDC/async jobs) β owning service and repo, resolved via the groundtruth ownership registry (`ClipboardHealth/groundtruth`, `registry/services.json` β `registry/repos.json`) β terminal cause cited at file/line, config key, or specific log line. The link types adapt to the failure surface: CI/setup, fixture, component, and unit failures substitute build logs, fixture state, or runner artifacts for network evidence and the owning package/repo for a deployed service β the critic checks the terminus, not a fixed link shape. Two valid terminal states:
|
|
82
|
+
|
|
83
|
+
1. **Terminated:** the chain ends at a cause. A status code, timeout, or throttle is a link, never a terminus.
|
|
84
|
+
2. **Explicitly broken:** the chain names the link where evidence ran out and the observability that would extend it; confidence is capped at 2/5 and B6 applies β the deliverable is the instrumentation, not a code fix.
|
|
85
|
+
|
|
86
|
+
**Confidence ceiling:** 5/5 additionally requires reproduction by inducing the blamed cause β fault injection in the harness (delay or fail the specific response, disable the seed step) or a focused lower-level test that deterministically reproduces the race. Without reproduction, the ceiling is 4/5. One inferred _intermediate_ link with an evidenced terminus is still a terminated chain (β€4/5); an unevidenced terminus is a broken chain (cap 2/5).
|
|
87
|
+
|
|
88
|
+
**Detection heuristic:** plan lacks a Causal chain section (statement-missing β amend-and-resubmit per Β§1); the terminal link is a symptom rather than a cause; the chain implicates another service or repo but cites no code/log evidence from it and doesn't route per C5/D5; or the plan claims 5/5 without reproduction evidence.
|
|
89
|
+
|
|
75
90
|
## 3. Allowed/safe patterns
|
|
76
91
|
|
|
77
92
|
These have strong precedent (counts from the 74 merged `flaky-test-fix` PRs; classification approximate) and are auto-merge candidates **only** when the listed evidence is present in the PR body.
|
|
@@ -6,20 +6,32 @@ Shared tail of the flaky-debug planning phase, used by both [`plan-e2e.md`](./pl
|
|
|
6
6
|
|
|
7
7
|
Rate your confidence in the root cause on a 1-5 scale. Report this score alongside your evidence.
|
|
8
8
|
|
|
9
|
-
| Score | Meaning | Criteria
|
|
10
|
-
| ----- | ------------------- |
|
|
11
|
-
| **5** | Certain | Root cause is directly visible in artifacts
|
|
12
|
-
| **4** | High confidence |
|
|
13
|
-
| **3** | Moderate confidence | Evidence is consistent with the diagnosis but alternative explanations remain plausible. Flag the alternatives explicitly
|
|
14
|
-
| **2** | Low confidence | Limited evidence, mostly reasoning from code patterns rather than observed artifacts. Recommend gathering more data before committing to a fix
|
|
15
|
-
| **1** | Speculative | No direct evidence for the root cause. The fix is a best guess. Recommend reproducing the failure locally or adding instrumentation before proceeding
|
|
9
|
+
| Score | Meaning | Criteria |
|
|
10
|
+
| ----- | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
11
|
+
| **5** | Certain | Root cause is directly visible in artifacts AND reproduced by inducing it β fault injection in the harness (delay/fail the blamed response or step) or a focused lower-level test that deterministically triggers the race. Without reproduction, the ceiling is 4 |
|
|
12
|
+
| **4** | High confidence | The chain terminates at an evidenced cause (file/line, config key, or log line) but one intermediate link is inferred rather than observed. An unevidenced terminal cause is a broken chain, capped at 2 (see [Causal Chain](#causal-chain)) |
|
|
13
|
+
| **3** | Moderate confidence | Evidence is consistent with the diagnosis but alternative explanations remain plausible. Flag the alternatives explicitly |
|
|
14
|
+
| **2** | Low confidence | Limited evidence, mostly reasoning from code patterns rather than observed artifacts. Recommend gathering more data before committing to a fix |
|
|
15
|
+
| **1** | Speculative | No direct evidence for the root cause. The fix is a best guess. Recommend reproducing the failure locally or adding instrumentation before proceeding |
|
|
16
16
|
|
|
17
17
|
Apply the score:
|
|
18
18
|
|
|
19
19
|
- **If >2:** continue to [Decide Fix Approach](#decide-fix-approach).
|
|
20
|
-
- **If less than 5/5:** the plan must include the frontend and/or backend observability changes that would make the next occurrence
|
|
20
|
+
- **If less than 5/5:** the plan must include the frontend and/or backend observability changes that would make the next occurrence's root cause directly visible in artifacts (the artifact half of 5/5; reaching 5/5 additionally requires reproduction per [Causal Chain](#causal-chain)). Scope recommendations to the repositories and services on the causal chain; resolve owners via the groundtruth registry rather than assuming the failure belongs to the repo the test lives in.
|
|
21
21
|
- **If confidence is 2 or below:** do not propose a code fix. Instead, recommend specific instrumentation or reproduction steps to raise confidence.
|
|
22
22
|
|
|
23
|
+
## Causal Chain
|
|
24
|
+
|
|
25
|
+
Every plan must trace the failure to a terminal _cause_, not a symptom:
|
|
26
|
+
|
|
27
|
+
failing assertion β app/UI state β network/trace/log evidence β owning service and repo β cause at a file/line, config key, or specific log line.
|
|
28
|
+
|
|
29
|
+
- A status code, timeout, or throttle is a link in the chain, never the terminus. "The request 500ed" or "setup was throttled" is where the investigation continues, not where it stops.
|
|
30
|
+
- The link types adapt to the failure surface. CI/setup, fixture, component, and unit failures substitute build logs, fixture state, or runner artifacts for network evidence, and the owning package/repo for a deployed service. The invariant is the terminus β a cause, not a symptom β not the specific link types.
|
|
31
|
+
- When a network request or deployed service is implicated, resolve its owning service and repo via the groundtruth ownership registry (`ClipboardHealth/groundtruth`: `registry/services.json` β `registry/repos.json`; per-repo context in `context/devin-wiki/<repo>/`). Follow the chain into that repo's code β do not stop at the repo the test lives in.
|
|
32
|
+
- When the causal event may be outside the test's own request path (seeding, deploys, CDC lag, async jobs), trace-ID lookup cannot reach it. Use time-window log queries scoped to the run instead.
|
|
33
|
+
- If evidence runs out, state exactly which link breaks and what observability would extend it. That caps confidence at 2, and the instrumentation becomes the deliverable.
|
|
34
|
+
|
|
23
35
|
## Decide Fix Approach
|
|
24
36
|
|
|
25
37
|
Applies to all test types.
|
|
@@ -84,9 +96,10 @@ Produce the plan with these fields:
|
|
|
84
96
|
- **Current main status:** whether the failing commit's code path still exists on current `main`, has already been fixed, or has changed enough that the plan must be adjusted
|
|
85
97
|
- **Symptom:** what failed and where
|
|
86
98
|
- **Root cause:** concise technical explanation
|
|
99
|
+
- **Causal chain:** each link from failing assertion to terminal cause with its evidence, or the explicit break point and the observability that would extend it (see [Causal Chain](#causal-chain))
|
|
87
100
|
- **Evidence:** artifacts supporting the diagnosis (traces, network, error messages, screenshots as applicable)
|
|
88
101
|
- **Proposed fix:** test harness, product, or both β with the specific file(s) and the change you would make
|
|
89
|
-
- **Observability to reach 5/5:** required when confidence is less than 5/5. List the frontend and/or backend telemetry, logging, tracing, reporter, or metric changes that would make this flake
|
|
102
|
+
- **Observability to reach 5/5:** required when confidence is less than 5/5. List the frontend and/or backend telemetry, logging, tracing, reporter, or metric changes that would make this flake's root cause directly visible in artifacts next time (reproduction then completes 5/5). Include another repository only when the evidence implicates it. Use "N/A -- confidence is 5/5" only for a 5/5 plan.
|
|
90
103
|
- **Sibling candidates:** files that appear to share the same anti-pattern, for the reviewer (or fix.md) to confirm. Or "N/A -- fix is test-specific" if the issue is one-off (see [`fix.md`](./fix.md) for what counts as a structural anti-pattern worth searching for).
|
|
91
104
|
- **Validation plan:** lint/typecheck commands and test commands to run after applying the fix
|
|
92
105
|
- **Open questions:** anything that needs human input before fixing
|