@clipboard-health/ai-rules 2.37.1 → 2.37.3

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@clipboard-health/ai-rules",
-  "version": "2.37.1",
+  "version": "2.37.3",
   "description": "Pre-built AI agent rules for consistent coding standards.",
   "keywords": [
     "ai",

package/rules/backend/restApiDesign.md

@@ -56,8 +56,10 @@ GET /shifts?filter[verified]=true&sort=startDate,-urgency&page[cursor]=abc&page[
 - Add contracts to `contract-<microservice-name>` package
 - Use `ts-rest` with composable Zod schemas (enforced by `enforce-ts-rest-in-controllers`)
 - Own the shape of your inputs and outputs — do not depend on other contract packages (`contract-*`, `api-contract-*`, `flag-*`); `@clipboard-health/contract-core` is the only shared contract dependency
+- Contracts validate the shape of the request and response, not the business logic
 - Duplicate schemas from other contracts locally when needed — type-checking and response validation catch drift
 - Do not re-export or pass through another contract's schemas or endpoints
+- Do not directly test contracts; use a service test to verify the contract is working as expected
 
 ### Schema rules
 

package/skills/blind-backtest/SKILL.md

@@ -0,0 +1,46 @@
+---
+name: blind-backtest
+description: Backtest an automated judgment stage (clustering, verdicts, routing, gating) against historical decisions with known outcomes. Use before any judgment skill goes live or into shadow operation, or when asked to grade an agent's decisions against a human's.
+---
+
+# Blind Backtest
+
+Before an automated judgment stage touches production state, prove it against history. The trap this skill exists to avoid: a single agent that can see how cases actually resolved will leak outcomes into its judgments and produce a worthless score. Blinding is enforced by splitting roles across separate agents that never share context.
+
+The coordinator (you, the session running this skill) holds the answer key and grades. Two subagents do the work:
+
+## Phase 1: Assembler (sees outcomes — never judges)
+
+Spawn a subagent that may read the historical record (Linear, GitHub, logs) to:
+
+- **Select a balanced cohort** (~20 cases is usually enough to expose error structure): both outcome classes represented, ideally ~60/40. If one class is rare, say so rather than padding with weak cases.
+- **Verify outcomes, don't assume them.** "Ticket Done" is weaker than "Done AND the PR merged"; "Canceled" is weaker than "canceled by a reviewer's judgment" (bulk sweeps and staleness cancellations are noisy labels — flag them in the key).
+- **Sanitize each case into an input file** containing only what existed _before_ the human verdict: the plan/content under judgment and its cited evidence. Strip everything outcome-revealing: states, completion/cancel dates, merge status, post-decision comments, and identifiers (ticket/PR numbers → `[redacted]`) that the judge could look up.
+- **Write** `inputs/` files (shuffled, opaque/neutral filenames like `case-01`, no class grouping) and `answer-key.json` (`{file, source, outcome, basis}`) to a scratch directory, then grep the inputs case-insensitively (`grep -i`) — contents, filenames, and any frontmatter — for leak terms (merged, closed, canceled, done, wontfix, approved, rejected, released, accepted, resolved) so capitalized status labels like `Done` or `Merged` aren't missed, and justify any survivor.
+- Report cohort size, balance, weak-label caveats, and any correlated cases (same family/mechanism) that could let the judge score unearned points.
+
+## Phase 2: Blind judge (sees nothing but the inputs)
+
+Spawn a separate subagent with an explicit blind protocol in its prompt:
+
+- May read ONLY: the judgment skill under test (and its references), and the `inputs/` files.
+- Forbidden, named explicitly: the answer key, the scratch directory's other files, Linear/GitHub/network, and anything that could reveal resolutions. Writes nothing.
+- Must judge each case per the skill under test, outputting exactly one structured line per case, keyed by the input's opaque filename: verdict, confidence, classification, cited rules, one-line justification quoting the decisive content.
+- Must flag cases it found genuinely borderline — those are where the grading discussion matters.
+
+Never reuse the assembler as the judge; an agent cannot unsee outcomes.
+
+## Phase 3: Grade (coordinator)
+
+- Join each verdict to `answer-key.json` by its opaque filename, never by output order. Require exactly one well-formed verdict per input before grading; missing, duplicate, or malformed records are protocol failures — re-run the affected cases under the same blind protocol, up to a small fixed retry budget (e.g., 2 attempts per case). If a case still lacks exactly one well-formed verdict after that, stop and report the run as a protocol failure rather than looping or grading a partial cohort. Never drop or exclude a case after seeing its verdict; outcome-dependent exclusion biases the matrix. Cohort membership is fixed before judging.
+- Build the confusion matrix against the answer key. Escalations (needs-human) on a correct-outcome case count as _safe non-matches_, not errors — bouncing up is designed behavior — but track them as a distinct outcome and report the escalation rate separately, so an agent that escalates everything cannot look artificially safe.
+- **Decompose every error before trusting the headline number.** The three buckets that recur:
+  1. **Era artifacts** — the skill enforces conventions that postdate the historical cases (will vanish on live traffic; consider scoring without them, but say so).
+  2. **Weak ground truth** — sweep-canceled, stale, or bulk-handled cases where the label doesn't reflect a judgment on quality.
+  3. **Genuine misses** — the only bucket that demands a rule change.
+- **Convert findings into amendments**: every genuine miss becomes a concrete edit to the skill/rubric under test, applied immediately and noted with date + "backtest amendment" provenance. Era artifacts are reported separately and become edits only when independently confirmed as live rule gaps — never amend the rubric for a systematic artifact alone, or you encode obsolete conventions and overfit future judgments.
+- A backtest validates the _mechanism_ (verdicts cite checkable rules) and surfaces rule gaps.
+
+## Output
+
+Report, in order: cohort summary (size, balance, label caveats); confusion matrix — raw score first, then the decomposed/adjusted view with its assumptions; error decomposition by bucket; amendments applied; borderline cases worth a human look; and what this backtest does and does not establish — **promotion to live enforcement still requires live shadow agreement**, since historical labels are too noisy to clear a high bar alone.

package/skills/cb-babysit/SKILL.md

@@ -19,9 +19,9 @@ Resolve bundled "./scripts" paths relative to SKILL.md.
 
 The skill uses two sentinels with visible footer lines.
 
-**Addressed sentinel**: `<sub>🤖 <code>cb-babysit:addressed v1 core@3.16.2</code></sub>`. Appended on its own line at the end of every reply the skill posts so re-runs know which threads and review-body comments are already handled. Dedupe also recognizes the legacy `babysit-pr:addressed v1` prefix from before this skill was renamed.
+**Addressed sentinel**: `<sub>🤖 <code>cb-babysit:addressed v1 core@3.16.3</code></sub>`. Appended on its own line at the end of every reply the skill posts so re-runs know which threads and review-body comments are already handled. Dedupe also recognizes the legacy `babysit-pr:addressed v1` prefix from before this skill was renamed.
 
-**Follow-up sentinel**: `<sub>🤖 <code>cb-babysit:followup v1 core@3.16.2</code></sub>`. Attached to replies that defer an out-of-scope comment as a tracked follow-up. The sentinel is additive: the post-reply scripts still append the `addressed` sentinel at the end.
+**Follow-up sentinel**: `<sub>🤖 <code>cb-babysit:followup v1 core@3.16.3</code></sub>`. Attached to replies that defer an out-of-scope comment as a tracked follow-up. The sentinel is additive: the post-reply scripts still append the `addressed` sentinel at the end.
 
 **Sentinel recency rules.** The script emits a per-thread `activityState` with three values. Step 6a owns the handling rules for each state.
 
@@ -247,7 +247,7 @@ Body templates (the script appends the `addressed` sentinel if missing):
 - **Agree**: `Addressed in <commit-url>. <one-line what-changed>.`
 - **Disagree**: `Leaving current behavior. <reasoning>.`
 - **Already fixed**: `Already handled by <commit-url-or-file:line>. <brief pointer>.`
-- **Defer**: `Out of scope for this PR; this looks like follow-up work rather than something introduced or required by this change. <one-line rationale or pointer if useful>.\n\n<sub>🤖 <code>cb-babysit:followup v1 core@3.16.2</code></sub>`
+- **Defer**: `Out of scope for this PR; this looks like follow-up work rather than something introduced or required by this change. <one-line rationale or pointer if useful>.\n\n<sub>🤖 <code>cb-babysit:followup v1 core@3.16.3</code></sub>`
 
 For Defer replies, include the follow-up sentinel on its own line as shown. The script will append the `addressed` sentinel after it on its own line, so the final body ends with the follow-up sentinel followed by a blank line followed by the `addressed` sentinel. `grep cb-babysit:followup` finds the deferral and `grep cb-babysit:addressed` still marks the thread handled for dedupe.
 
@@ -263,7 +263,7 @@ The PR-level summary should:
 
 - Group by source. Use `## Review-body findings` for step-7 work and `## Conversation-tab comments` for step-6b work. Omit a section if its list is empty.
 - Inside each section, group verdicts under **Agree / Disagree / Already fixed / Deferred (out of scope)** subheadings. Omit a subheading if its list is empty.
-- Under **Deferred (out of scope)**, list each deferred item as a bullet, followed on its own line by `<sub>🤖 <code>cb-babysit:followup v1 core@3.16.2</code></sub>` so grep catches them individually.
+- Under **Deferred (out of scope)**, list each deferred item as a bullet, followed on its own line by `<sub>🤖 <code>cb-babysit:followup v1 core@3.16.3</code></sub>` so grep catches them individually.
 - Include the commit URL for fixes.
 - End with a fenced fingerprint block listing every current fingerprint (addressed and deferred) one per line. Include both `reviewBodyComments[].fingerprint` (whole-body, one per automated review) and `activeIssueComments[].fingerprint` (per Conversation-tab comment). Future runs dedupe by matching these against `priorBabysitSentinels`.
 

package/skills/cb-babysit/scripts/_sentinel.sh

@@ -10,7 +10,7 @@
 
 SENTINEL_PREFIX='cb-babysit:addressed v1 '
 LEGACY_SENTINEL_PREFIX='babysit-pr:addressed v1 '
-SENTINEL='<sub>🤖 <code>cb-babysit:addressed v1 core@3.16.2</code></sub>'
+SENTINEL='<sub>🤖 <code>cb-babysit:addressed v1 core@3.16.3</code></sub>'
 
 # Bot author allowlist (JSON array literal). Used by unresolvedPrComments.sh
 # as a fallback when GraphQL's `author.__typename == "Bot"` misses a GitHub

package/skills/cb-ship/references/pr-template.md

@@ -17,5 +17,5 @@ Concisely explain the user intent from session history and the meaningful behavi
 
 Optional, don't fabricate: ticket links, rollout plan, residual risk, or specific areas for reviewers to focus.
 
-<sub>🤖 <code>cb-ship:created v1 core@3.16.2</code></sub>
+<sub>🤖 <code>cb-ship:created v1 core@3.16.3</code></sub>
 ```