@assertrx/dino 1.0.0

package/skills/validate-evaluator/SKILL.md

@@ -0,0 +1,215 @@
+---
+name: validate-evaluator
+description: >
+  Calibrate an LLM judge against human labels using data splits, TPR/TNR, and
+  bias correction. Use after writing a judge prompt (write-judge-prompt) when you
+  need to verify alignment before trusting its outputs. Do NOT use for code-based
+  evaluators (those are deterministic; test with standard unit tests).
+---
+
+# Validate Evaluator
+
+Calibrate an LLM judge against human judgment.
+
+## Overview
+
+1. Split human-labeled data into train (10-20%), dev (40-45%), test (40-45%)
+2. Run judge on dev set and measure TPR/TNR
+3. Iterate on the judge until TPR and TNR > 90% on dev set
+4. Run once on held-out test set for final TPR/TNR
+5. Apply bias correction formula to production data
+
+## Prerequisites
+
+- A built LLM judge prompt (from write-judge-prompt)
+- Human-labeled data: ~100 traces with binary Pass/Fail labels per failure mode
+  - Aim for ~50 Pass and ~50 Fail (balanced, even if real distribution is skewed)
+  - Labels must come from a domain expert, not outsourced annotators
+- Candidate few-shot examples from your labeled data
+
+## Core Instructions
+
+### Step 1: Create Data Splits
+
+Split human-labeled data into three disjoint sets:
+
+| Split | Size | Purpose | Rules |
+|-------|------|---------|-------|
+| **Training** | 10-20% (~10-20 examples) | Source of few-shot examples for the judge prompt | Only clear-cut Pass and Fail cases. Used directly in the prompt. |
+| **Dev** | 40-45% (~40-45 examples) | Iterative evaluator refinement | Never include in the prompt. Evaluate against repeatedly. |
+| **Test** | 40-45% (~40-45 examples) | Final unbiased accuracy measurement | Do NOT look at during development. Used once at the end. |
+
+Target: 30-50 examples of each class (Pass and Fail) across dev and test combined. Use balanced splits even if real-world prevalence is skewed — you need enough Fail examples to measure TNR reliably.
+
+```python
+from sklearn.model_selection import train_test_split
+
+# First split: separate test set
+train_dev, test = train_test_split(
+    labeled_data, test_size=0.4, stratify=labeled_data['label'], random_state=42
+)
+# Second split: separate training examples from dev set
+train, dev = train_test_split(
+    train_dev, test_size=0.75, stratify=train_dev['label'], random_state=42
+)
+# Result: ~15% train, ~45% dev, ~40% test
+```
+
+### Step 2: Run Evaluator on Dev Set
+
+Run the judge on every example in the dev set. Compare predictions to human labels.
+
+### Step 3: Measure TPR and TNR
+
+**TPR (True Positive Rate):** When a human says Pass, how often does the judge also say Pass?
+
+```
+TPR = (judge says Pass AND human says Pass) / (human says Pass)
+```
+
+**TNR (True Negative Rate):** When a human says Fail, how often does the judge also say Fail?
+
+```
+TNR = (judge says Fail AND human says Fail) / (human says Fail)
+```
+
+```python
+from sklearn.metrics import confusion_matrix
+
+tn, fp, fn, tp = confusion_matrix(human_labels, evaluator_labels,
+                                   labels=['Fail', 'Pass']).ravel()
+tpr = tp / (tp + fn)
+tnr = tn / (tn + fp)
+```
+
+Use TPR/TNR, not Precision/Recall or raw accuracy. These two metrics directly map to the bias correction formula. Use Cohen's Kappa only for measuring agreement between two human annotators, not for judge-vs-ground-truth.
+
+### Step 4: Inspect Disagreements
+
+Examine every case where the judge disagrees with human labels:
+
+| Disagreement Type | Judge | Human | Fix |
+|-------------------|-------|-------|-----|
+| **False Pass** | Pass | Fail | Judge is too lenient. Strengthen Fail definitions or add edge-case examples. |
+| **False Fail** | Fail | Pass | Judge is too strict. Clarify Pass definitions or adjust examples. |
+
+For each disagreement, determine whether to:
+- Clarify wording in the judge prompt
+- Swap or add few-shot examples from the training set
+- Add explicit rules for the edge case
+- Split the criterion into more specific sub-checks
+
+### Step 5: Iterate
+
+Refine the judge prompt and re-run on the dev set. Repeat until TPR and TNR stabilize.
+
+**Stopping criteria:**
+- **Target:** TPR > 90% AND TNR > 90%
+- **Minimum acceptable:** TPR > 80% AND TNR > 80%
+
+**If alignment stalls:**
+
+| Problem | Solution |
+|---------|---------|
+| TPR and TNR both low | Use a more capable LLM for the judge |
+| One metric low, one acceptable | Inspect disagreements for the low metric specifically |
+| Both plateau below target | Decompose the criterion into smaller, more atomic checks |
+| Consistently wrong on certain input types | Add targeted few-shot examples from training set |
+| Labels themselves seem inconsistent | Re-examine human labels; the rubric may need refinement |
+
+### Step 6: Final Measurement on Test Set
+
+Run the judge **exactly once** on the held-out test set. Record final TPR and TNR.
+
+Do not iterate after seeing test set results. Go back to step 4 with new dev data if needed.
+
+### Step 7 (Optional): Estimate True Success Rate (Rogan-Gladen Correction)
+
+Raw judge scores on unlabeled production data are biased. If you need an accurate aggregate pass rate, correct for known judge errors:
+
+```
+theta_hat = (p_obs + TNR - 1) / (TPR + TNR - 1)
+```
+
+Where:
+- `p_obs` = fraction of unlabeled traces the judge scored as Pass
+- `TPR`, `TNR` = from test set measurement
+- `theta_hat` = corrected estimate of true success rate
+
+Clip to [0, 1]. Invalid when TPR + TNR - 1 is near 0 (judge is no better than random).
+
+**Example:**
+- Judge TPR = 0.92, TNR = 0.88
+- 500 production traces: 400 scored Pass -> p_obs = 0.80
+- theta_hat = (0.80 + 0.88 - 1) / (0.92 + 0.88 - 1) = 0.68 / 0.80 = **0.85**
+- True success rate is ~85%, not the raw 80%
+
+### Step 8: Confidence Interval
+
+Compute a bootstrap confidence interval. A point estimate alone is not enough.
+
+```python
+import numpy as np
+
+def bootstrap_ci(human_labels, eval_labels, p_obs, n_bootstrap=2000):
+    """Bootstrap 95% CI for corrected success rate."""
+    n = len(human_labels)
+    estimates = []
+    for _ in range(n_bootstrap):
+        idx = np.random.choice(n, size=n, replace=True)
+        h = np.array(human_labels)[idx]
+        e = np.array(eval_labels)[idx]
+
+        tp = ((h == 'Pass') & (e == 'Pass')).sum()
+        fn = ((h == 'Pass') & (e == 'Fail')).sum()
+        tn = ((h == 'Fail') & (e == 'Fail')).sum()
+        fp = ((h == 'Fail') & (e == 'Pass')).sum()
+
+        tpr_b = tp / (tp + fn) if (tp + fn) > 0 else 0
+        tnr_b = tn / (tn + fp) if (tn + fp) > 0 else 0
+        denom = tpr_b + tnr_b - 1
+
+        if abs(denom) < 1e-6:
+            continue
+        theta = (p_obs + tnr_b - 1) / denom
+        estimates.append(np.clip(theta, 0, 1))
+
+    return np.percentile(estimates, 2.5), np.percentile(estimates, 97.5)
+
+lower, upper = bootstrap_ci(test_human, test_eval, p_obs=0.80)
+print(f"95% CI: [{lower:.2f}, {upper:.2f}]")
+```
+
+Or use `judgy` (`pip install judgy`):
+
+```python
+from judgy import estimate_success_rate
+
+# judgy expects 0/1 integer labels (1 = Pass, 0 = Fail)
+test_labels = [1 if l == 'Pass' else 0 for l in test_human_labels]
+test_preds = [1 if l == 'Pass' else 0 for l in test_eval_labels]
+unlabeled_preds = [1 if l == 'Pass' else 0 for l in prod_eval_labels]
+
+theta_hat, lower, upper = estimate_success_rate(
+    test_labels, test_preds, unlabeled_preds
+)
+print(f"Corrected rate: {theta_hat:.2f}")
+print(f"95% CI: [{lower:.2f}, {upper:.2f}]")
+```
+
+## Practical Guidance
+
+- **Pin exact model versions** for LLM judges (a dated snapshot id like `<model>-<YYYY-MM-DD>`, not a floating alias). Providers update models without notice, causing silent drift.
+- **Re-validate** after changing the judge prompt, switching models, or when production confidence intervals widen unexpectedly.
+- Use ~100 labeled examples (50 Pass, 50 Fail). Below 60, confidence intervals become wide.
+- **One trusted domain expert** is the most efficient labeling path. If not feasible, have two annotators label 20-50 traces independently and resolve disagreements before proceeding.
+- **Improving TPR narrows the confidence interval more than improving TNR.** The correction divides by `(TPR + TNR - 1)`, so a low TPR shrinks the denominator and amplifies estimation errors into wide CIs.
+
+## Anti-Patterns
+
+- **Assuming judges "just work" without validation.** A judge may consistently miss failures or flag passing traces.
+- **Using raw accuracy or percent agreement.** Use TPR and TNR. With class imbalance, raw accuracy is misleading.
+- **Dev/test examples as few-shot examples.** This is data leakage.
+- **Reporting dev set performance as final accuracy.** Dev numbers are optimistic. The test set gives the unbiased estimate.
+- **Raw judge scores without bias correction.** If you report an aggregate pass rate, apply the Rogan-Gladen formula (Step 7).
+- **Point estimates without confidence intervals.** A corrected rate of 85% could easily be 78-92% with small test sets. Report the range so stakeholders know how much to trust the number.

package/skills/write-judge-prompt/SKILL.md

@@ -0,0 +1,144 @@
+---
+name: write-judge-prompt
+description: >
+  Design LLM-as-Judge evaluators for subjective criteria that code-based checks
+  cannot handle. Use when a failure mode requires interpretation (tone,
+  faithfulness, relevance, completeness). Do NOT use when the failure mode can be
+  checked with code (regex, schema validation, execution tests). Do NOT use when
+  you need to validate or calibrate the judge — use validate-evaluator instead.
+---
+
+# Write LLM-as-Judge Prompt
+
+Design a binary Pass/Fail LLM-as-Judge evaluator for one specific failure mode. Each judge checks exactly one thing.
+
+## Prerequisites
+
+- Error analysis is complete. The failure mode is identified.
+- You have human-labeled traces for this failure mode (at least 20 Pass and 20 Fail examples).
+- A code-based evaluator cannot check this failure mode. Exhaust code-based options before reaching for a judge — many failure modes that seem subjective reduce to keyword checks, regex, or API calls when you understand the domain. Example: detecting whether an AI interviewing coach suggests "general" questions (asking about typical behavior instead of a specific past event) seems to require semantic understanding, but in practice a keyword check for words like "usually," "typical," and "normally" could work quite well.
+
+## The Four Components
+
+Every judge prompt requires exactly four components:
+
+### 1. Task and Evaluation Criterion
+
+State what the judge evaluates. One failure mode per judge.
+
+```
+You are an evaluator assessing whether a real estate assistant's email
+uses the appropriate tone for the client's persona.
+```
+
+Not: "Evaluate whether the email is good" or "Rate the email quality from 1-5."
+
+### 2. Pass/Fail Definitions
+
+Outcomes are strictly binary: Pass or Fail. No Likert scales, no letter grades, no partial credit. Define exactly what constitutes Pass and Fail. These definitions come from your error analysis failure mode descriptions.
+
+```
+## Definitions
+
+PASS: The email matches the expected communication style for the client persona:
+- Luxury Buyers: formal language, emphasis on exclusive features, premium
+  market positioning, no casual slang
+- First-Time Homebuyers: warm and encouraging tone, educational explanations,
+  avoids jargon, patient and supportive
+- Investors: data-driven language, ROI-focused, market analytics, concise
+  and professional
+
+FAIL: The email uses a tone mismatched to the client persona. Examples:
+- Using casual slang ("hey, check out this pad!") for a luxury buyer
+- Using heavy financial jargon for a first-time homebuyer
+- Using overly emotional language for an investor
+```
+
+### 3. Few-Shot Examples
+
+Include labeled Pass and Fail examples from your human-labeled data.
+
+```
+## Examples
+
+### Example 1: PASS
+Client Persona: Luxury Buyer
+Email: "Dear Mr. Harrington, I am pleased to present an exclusive listing
+at 1200 Pacific Heights Drive. This distinguished property features..."
+Critique: The email opens with a formal salutation and uses language
+consistent with luxury positioning — "exclusive listing," "distinguished
+property." No casual slang or informal phrasing. The tone matches the
+luxury buyer persona throughout.
+Result: Pass
+
+### Example 2: FAIL
+Client Persona: Luxury Buyer
+Email: "Hey! Just found this awesome place you might like. It's got a
+pool and stuff, super cool neighborhood..."
+Critique: The greeting "Hey!" is informal. Phrases like "awesome place,"
+"got a pool and stuff," and "super cool" are casual slang inappropriate
+for a luxury buyer. The email reads like a text message, not a
+professional communication for a high-end client.
+Result: Fail
+
+### Example 3: PASS (borderline)
+Client Persona: First-Time Homebuyer
+Email: "Hi Sarah, I found a property that might be a great fit for your
+first home. The neighborhood has good schools nearby, and the monthly
+payment would be similar to what you're currently paying in rent..."
+Critique: The greeting is warm but not overly casual. The email explains
+the property in relatable terms — comparing mortgage to rent, mentioning
+schools — which is educational without being condescending. It avoids
+jargon like "amortization" or "LTV ratio." While not deeply technical,
+this matches the supportive tone expected for a first-time buyer.
+Result: Pass
+```
+
+**Rules for selecting examples:**
+- Include at least one clear Pass, one clear Fail, and one borderline case. Borderline examples are the most valuable — they teach nuance.
+- Draw examples from the training split (10-20% of labeled data set aside for this purpose).
+- Any example used in the judge prompt must be excluded from dev and test sets. Using dev/test examples is data leakage.
+- 2-4 examples is typical. Performance plateaus after 4-8.
+
+### 4. Structured Output Format
+
+Enforce structured output using your LLM provider's schema enforcement (e.g., `response_format` in OpenAI, tool definitions in Anthropic) or a library like Instructor or Outlines. If the provider doesn't support schema enforcement, specify the JSON schema in the prompt.
+
+The output must include a critique before the verdict. Placing the critique first forces the judge to articulate its assessment before committing to a decision.
+
+```json
+{
+  "critique": "string — detailed assessment of the output against the criterion",
+  "result": "Pass or Fail"
+}
+```
+
+Critiques must be detailed, not terse. A good critique explains what specifically was correct or incorrect and references concrete evidence from the output. The critiques in your few-shot examples set the bar for the level of detail the judge will produce.
+
+## Choosing What to Pass to the Judge
+
+Feed only what the judge needs for an accurate decision:
+
+| Failure Mode | What the Judge Needs |
+|-------------|---------------------|
+| Tone mismatch | Client persona + generated email |
+| Answer faithfulness | Retrieved context + generated answer |
+| SQL correctness | User query + generated SQL + schema |
+| Instruction following | System prompt rules + generated response |
+| Tool call justification | Conversation history + tool call + tool result |
+
+For long documents, feed only the relevant snippet, not the entire document.
+
+## Model Selection
+
+Start with the most capable model available. The same model used for the main task works as judge (the judge performs a different, narrower task). Optimize for cost later once alignment is confirmed.
+
+## Anti-Patterns
+
+- **Vague criteria like "is this helpful?"** Target a specific, observable failure mode from error analysis.
+- **Holistic judge for the entire trace.** A single judge covering multiple dimensions produces unactionable verdicts.
+- **No few-shot examples.** Without examples, the model won't know what counts as a failure in your application.
+- **Dev/test examples used as few-shot.** This is data leakage. Use only the training split.
+- **Likert scales (1-5, letter grades, etc.).** Binary pass/fail only. Likert scales produce scores that sound precise but can't be calibrated: annotators disagree on the difference between a 3 and a 4, and the judge inherits that noise. Binary forces you to define a clear decision boundary upfront, which makes inter-annotator agreement measurable and the judge's errors actionable. If you need to capture severity, use multiple binary judges (e.g., "factually wrong" and "dangerously wrong") rather than one ordinal scale.
+- **Skipping validation.** Measure alignment with human labels using validate-evaluator before trusting the judge.
+- **Judges for specification failures without fixing the prompt first.** If the prompt never asked for the behavior, add the instruction before building an evaluator. For critical requirements, a judge can still serve as a regression guard.

package/themes/dino.json

@@ -0,0 +1,66 @@
+{
+  "$schema": "https://raw.githubusercontent.com/earendil-works/pi/main/packages/coding-agent/src/modes/interactive/theme/theme-schema.json",
+  "name": "dino",
+  "vars": {
+    "purple": "#7d4e8c",
+    "lightPurple": "#b98cc4",
+    "green": "#78c74d",
+    "yellow": "#f5d742",
+    "dark": "#2a1d35",
+    "gray": 242
+  },
+  "colors": {
+    "accent": "lightPurple",
+    "border": "purple",
+    "borderAccent": "lightPurple",
+    "borderMuted": "gray",
+    "success": "green",
+    "error": "#ff5555",
+    "warning": "yellow",
+    "muted": "gray",
+    "dim": 240,
+    "text": "",
+    "thinkingText": "gray",
+    "selectedBg": "dark",
+    "userMessageBg": "dark",
+    "userMessageText": "",
+    "customMessageBg": "dark",
+    "customMessageText": "",
+    "customMessageLabel": "lightPurple",
+    "toolPendingBg": "#221a29",
+    "toolSuccessBg": "#1a291a",
+    "toolErrorBg": "#291a1a",
+    "toolTitle": "lightPurple",
+    "toolOutput": "",
+    "mdHeading": "yellow",
+    "mdLink": "lightPurple",
+    "mdLinkUrl": "gray",
+    "mdCode": "green",
+    "mdCodeBlock": "",
+    "mdCodeBlockBorder": "gray",
+    "mdQuote": "gray",
+    "mdQuoteBorder": "gray",
+    "mdHr": "gray",
+    "mdListBullet": "green",
+    "toolDiffAdded": "green",
+    "toolDiffRemoved": "#ff5555",
+    "toolDiffContext": "gray",
+    "syntaxComment": "gray",
+    "syntaxKeyword": "lightPurple",
+    "syntaxFunction": "yellow",
+    "syntaxVariable": "green",
+    "syntaxString": "green",
+    "syntaxNumber": "yellow",
+    "syntaxType": "lightPurple",
+    "syntaxOperator": "lightPurple",
+    "syntaxPunctuation": "gray",
+    "thinkingOff": "gray",
+    "thinkingMinimal": "lightPurple",
+    "thinkingLow": "purple",
+    "thinkingMedium": "lightPurple",
+    "thinkingHigh": "yellow",
+    "thinkingXhigh": "green",
+    "thinkingMax": "#ff88cc",
+    "bashMode": "yellow"
+  }
+}