@clipboard-health/ai-rules 2.38.0 → 2.38.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
"use strict";
|
|
2
2
|
Object.defineProperty(exports, "__esModule", { value: true });
|
|
3
3
|
exports.toErrorMessage = toErrorMessage;
|
|
4
|
-
// Prefer stack over message for Error instances to preserve debugging context
|
|
4
|
+
// Prefer stack over message for Error instances to preserve debugging context!
|
|
5
5
|
function toErrorMessage(error) {
|
|
6
6
|
return error instanceof Error ? (error.stack ?? error.message) : String(error);
|
|
7
7
|
}
|
|
@@ -8,7 +8,7 @@ This rubric turns Rocky's observed review standards into checkable rules for fla
|
|
|
8
8
|
|
|
9
9
|
Findings from automated reviewers (Mendral, CodeRabbit, etc.) are **advisory inputs, not binding rules**: the critic evaluates each on rubric merits and may decline one with stated reasoning, exactly as cb-babysit's Disagree replies do.
|
|
10
10
|
|
|
11
|
-
**Statement-missing vs substance-missing (backtest amendment, 2026-06-11):** B5/B6/B7 each require an explicit statement in the plan (dedup-check results, confidence score plus observability section, current-main status). When the _statement_ is absent but the violation is not demonstrated — no actual duplicate found, no actual low-confidence code fix, no actual staleness — the verdict is reject with disposition **amend-and-resubmit**: one bounce asking for exactly the missing sections, never escalation. A demonstrated violation (a real duplicate, a real ≤2/5 code fix) is a substantive reject. The 2026-06-11 backtest showed 6 of 7 false rejections were statement-missing on plans predating these conventions; live plans carry the sections by template, but the distinction stays load-bearing for the agreement metric.
|
|
11
|
+
**Statement-missing vs substance-missing (backtest amendment, 2026-06-11):** B5/B6/B7/B8 each require an explicit statement in the plan (dedup-check results, confidence score plus observability section, current-main status, causal-chain section). When the _statement_ is absent but the violation is not demonstrated — no actual duplicate found, no actual low-confidence code fix, no actual staleness, no demonstrably symptom-terminated diagnosis — the verdict is reject with disposition **amend-and-resubmit**: one bounce asking for exactly the missing sections, never escalation. A demonstrated violation (a real duplicate, a real ≤2/5 code fix, a diagnosis that demonstrably stops at a symptom) is a substantive reject. The 2026-06-11 backtest showed 6 of 7 false rejections were statement-missing on plans predating these conventions; live plans carry the sections by template, but the distinction stays load-bearing for the agreement metric.
|
|
12
12
|
|
|
13
13
|
## 2. Banned patterns
|
|
14
14
|
|
|
@@ -72,6 +72,21 @@ Implementing a plan against code that has changed on `main`, or for a flake alre
|
|
|
72
72
|
|
|
73
73
|
**Staleness clause (backtest amendment, 2026-06-11):** a plan more than ~14 days old, or one whose fingerprint has new sightings since the plan was written, requires re-verification against current main before approval — treat unverified staleness as B7. (Two backtest "rejections" were plans that sat unworked until a sweep canceled them; freshness is part of a plan being gateable.)
|
|
74
74
|
|
|
75
|
+
### B8 — Unterminated causal chain (symptom-level diagnosis)
|
|
76
|
+
|
|
77
|
+
Proposing a code fix from a diagnosis that stops at the failure symptom — "the request 500ed/timed out," "setup was throttled," "the element never appeared" — without establishing _why_ at a specific cause in the owning code, config, or infrastructure.
|
|
78
|
+
|
|
79
|
+
**Why (amendment, 2026-07-07):** post-Done recurrence is the pipeline's dominant failure mode: in the Jun 2–Jul 6 window, 21 fingerprint families re-flaked after a Done closure, and the fullLifecycle family (33deef731a10) was closed Done six times. Each recurrence traces to a diagnosis that dead-ended at the repo boundary or at a status code and patched the symptom locally. Local-symptom plans are the cheapest to produce, so as long as the gate accepts them, agents keep producing them; this rule makes boundary-crossing the only way to pass.
|
|
80
|
+
|
|
81
|
+
**Required structure:** the plan must contain a **Causal chain** section: failing assertion → app/UI state → network/trace/log evidence (trace ID, or run-window log queries when the causal event is outside the test's request path: seeding, deploys, CDC/async jobs) → owning service and repo, resolved via the groundtruth ownership registry (`ClipboardHealth/groundtruth`, `registry/services.json` → `registry/repos.json`) → terminal cause cited at file/line, config key, or specific log line. The link types adapt to the failure surface: CI/setup, fixture, component, and unit failures substitute build logs, fixture state, or runner artifacts for network evidence and the owning package/repo for a deployed service — the critic checks the terminus, not a fixed link shape. Two valid terminal states:
|
|
82
|
+
|
|
83
|
+
1. **Terminated:** the chain ends at a cause. A status code, timeout, or throttle is a link, never a terminus.
|
|
84
|
+
2. **Explicitly broken:** the chain names the link where evidence ran out and the observability that would extend it; confidence is capped at 2/5 and B6 applies — the deliverable is the instrumentation, not a code fix.
|
|
85
|
+
|
|
86
|
+
**Confidence ceiling:** 5/5 additionally requires reproduction by inducing the blamed cause — fault injection in the harness (delay or fail the specific response, disable the seed step) or a focused lower-level test that deterministically reproduces the race. Without reproduction, the ceiling is 4/5. One inferred _intermediate_ link with an evidenced terminus is still a terminated chain (≤4/5); an unevidenced terminus is a broken chain (cap 2/5).
|
|
87
|
+
|
|
88
|
+
**Detection heuristic:** plan lacks a Causal chain section (statement-missing → amend-and-resubmit per §1); the terminal link is a symptom rather than a cause; the chain implicates another service or repo but cites no code/log evidence from it and doesn't route per C5/D5; or the plan claims 5/5 without reproduction evidence.
|
|
89
|
+
|
|
75
90
|
## 3. Allowed/safe patterns
|
|
76
91
|
|
|
77
92
|
These have strong precedent (counts from the 74 merged `flaky-test-fix` PRs; classification approximate) and are auto-merge candidates **only** when the listed evidence is present in the PR body.
|
|
@@ -6,20 +6,32 @@ Shared tail of the flaky-debug planning phase, used by both [`plan-e2e.md`](./pl
|
|
|
6
6
|
|
|
7
7
|
Rate your confidence in the root cause on a 1-5 scale. Report this score alongside your evidence.
|
|
8
8
|
|
|
9
|
-
| Score | Meaning | Criteria
|
|
10
|
-
| ----- | ------------------- |
|
|
11
|
-
| **5** | Certain | Root cause is directly visible in artifacts
|
|
12
|
-
| **4** | High confidence |
|
|
13
|
-
| **3** | Moderate confidence | Evidence is consistent with the diagnosis but alternative explanations remain plausible. Flag the alternatives explicitly
|
|
14
|
-
| **2** | Low confidence | Limited evidence, mostly reasoning from code patterns rather than observed artifacts. Recommend gathering more data before committing to a fix
|
|
15
|
-
| **1** | Speculative | No direct evidence for the root cause. The fix is a best guess. Recommend reproducing the failure locally or adding instrumentation before proceeding
|
|
9
|
+
| Score | Meaning | Criteria |
|
|
10
|
+
| ----- | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
11
|
+
| **5** | Certain | Root cause is directly visible in artifacts AND reproduced by inducing it — fault injection in the harness (delay/fail the blamed response or step) or a focused lower-level test that deterministically triggers the race. Without reproduction, the ceiling is 4 |
|
|
12
|
+
| **4** | High confidence | The chain terminates at an evidenced cause (file/line, config key, or log line) but one intermediate link is inferred rather than observed. An unevidenced terminal cause is a broken chain, capped at 2 (see [Causal Chain](#causal-chain)) |
|
|
13
|
+
| **3** | Moderate confidence | Evidence is consistent with the diagnosis but alternative explanations remain plausible. Flag the alternatives explicitly |
|
|
14
|
+
| **2** | Low confidence | Limited evidence, mostly reasoning from code patterns rather than observed artifacts. Recommend gathering more data before committing to a fix |
|
|
15
|
+
| **1** | Speculative | No direct evidence for the root cause. The fix is a best guess. Recommend reproducing the failure locally or adding instrumentation before proceeding |
|
|
16
16
|
|
|
17
17
|
Apply the score:
|
|
18
18
|
|
|
19
19
|
- **If >2:** continue to [Decide Fix Approach](#decide-fix-approach).
|
|
20
|
-
- **If less than 5/5:** the plan must include the frontend and/or backend observability changes that would make the next occurrence
|
|
20
|
+
- **If less than 5/5:** the plan must include the frontend and/or backend observability changes that would make the next occurrence's root cause directly visible in artifacts (the artifact half of 5/5; reaching 5/5 additionally requires reproduction per [Causal Chain](#causal-chain)). Scope recommendations to the repositories and services on the causal chain; resolve owners via the groundtruth registry rather than assuming the failure belongs to the repo the test lives in.
|
|
21
21
|
- **If confidence is 2 or below:** do not propose a code fix. Instead, recommend specific instrumentation or reproduction steps to raise confidence.
|
|
22
22
|
|
|
23
|
+
## Causal Chain
|
|
24
|
+
|
|
25
|
+
Every plan must trace the failure to a terminal _cause_, not a symptom:
|
|
26
|
+
|
|
27
|
+
failing assertion → app/UI state → network/trace/log evidence → owning service and repo → cause at a file/line, config key, or specific log line.
|
|
28
|
+
|
|
29
|
+
- A status code, timeout, or throttle is a link in the chain, never the terminus. "The request 500ed" or "setup was throttled" is where the investigation continues, not where it stops.
|
|
30
|
+
- The link types adapt to the failure surface. CI/setup, fixture, component, and unit failures substitute build logs, fixture state, or runner artifacts for network evidence, and the owning package/repo for a deployed service. The invariant is the terminus — a cause, not a symptom — not the specific link types.
|
|
31
|
+
- When a network request or deployed service is implicated, resolve its owning service and repo via the groundtruth ownership registry (`ClipboardHealth/groundtruth`: `registry/services.json` → `registry/repos.json`; per-repo context in `context/devin-wiki/<repo>/`). Follow the chain into that repo's code — do not stop at the repo the test lives in.
|
|
32
|
+
- When the causal event may be outside the test's own request path (seeding, deploys, CDC lag, async jobs), trace-ID lookup cannot reach it. Use time-window log queries scoped to the run instead.
|
|
33
|
+
- If evidence runs out, state exactly which link breaks and what observability would extend it. That caps confidence at 2, and the instrumentation becomes the deliverable.
|
|
34
|
+
|
|
23
35
|
## Decide Fix Approach
|
|
24
36
|
|
|
25
37
|
Applies to all test types.
|
|
@@ -84,9 +96,10 @@ Produce the plan with these fields:
|
|
|
84
96
|
- **Current main status:** whether the failing commit's code path still exists on current `main`, has already been fixed, or has changed enough that the plan must be adjusted
|
|
85
97
|
- **Symptom:** what failed and where
|
|
86
98
|
- **Root cause:** concise technical explanation
|
|
99
|
+
- **Causal chain:** each link from failing assertion to terminal cause with its evidence, or the explicit break point and the observability that would extend it (see [Causal Chain](#causal-chain))
|
|
87
100
|
- **Evidence:** artifacts supporting the diagnosis (traces, network, error messages, screenshots as applicable)
|
|
88
101
|
- **Proposed fix:** test harness, product, or both — with the specific file(s) and the change you would make
|
|
89
|
-
- **Observability to reach 5/5:** required when confidence is less than 5/5. List the frontend and/or backend telemetry, logging, tracing, reporter, or metric changes that would make this flake
|
|
102
|
+
- **Observability to reach 5/5:** required when confidence is less than 5/5. List the frontend and/or backend telemetry, logging, tracing, reporter, or metric changes that would make this flake's root cause directly visible in artifacts next time (reproduction then completes 5/5). Include another repository only when the evidence implicates it. Use "N/A -- confidence is 5/5" only for a 5/5 plan.
|
|
90
103
|
- **Sibling candidates:** files that appear to share the same anti-pattern, for the reviewer (or fix.md) to confirm. Or "N/A -- fix is test-specific" if the issue is one-off (see [`fix.md`](./fix.md) for what counts as a structural anti-pattern worth searching for).
|
|
91
104
|
- **Validation plan:** lint/typecheck commands and test commands to run after applying the fix
|
|
92
105
|
- **Open questions:** anything that needs human input before fixing
|