@cubis/foundry 0.3.83 → 0.3.84

package/workflows/skills/llm-eval/examples/01-code-review-chatbot-eval.md

@@ -1,91 +0,0 @@
-# Example: Eval Suite for a Code Review Chatbot
-
-## Context
-
-A team is building an LLM-powered code review assistant that analyzes pull requests and provides feedback on code quality, security vulnerabilities, and style violations. The assistant must:
-
-- Reference specific line numbers in its feedback.
-- Categorize findings as Critical, High, Medium, or Low severity.
-- Never suggest changes that would break compilation.
-- Refuse to review files that appear to contain secrets or credentials.
-
-## Eval Suite Design
-
-### Test Case 1: Python Function with SQL Injection
-
-**Input:**
-```
-Review this Python function:
-
-def get_user(user_id):
-    query = f"SELECT * FROM users WHERE id = {user_id}"
-    return db.execute(query)
-```
-
-**Assertions:**
-
-| # | Type | Value | Rationale |
-|---|------|-------|-----------|
-| 1 | contains | `SQL injection` | Must identify the primary vulnerability -- string interpolation in SQL. |
-| 2 | contains | `Critical` or `High` | SQL injection must be rated Critical or High severity, never Medium or Low. |
-| 3 | contains | `parameterized` or `prepared statement` | Must suggest the correct fix, not just identify the problem. |
-| 4 | not-contains | `looks good` | Must not produce a false-positive "no issues found" response. |
-| 5 | llm-judge | Rubric: references specific line, explains the attack vector, provides corrected code | Quality check on the completeness of the review. |
-
-### Test Case 2: Clean TypeScript Function (No Issues)
-
-**Input:**
-```
-Review this TypeScript function:
-
-function calculateTotal(items: CartItem[]): number {
-  return items.reduce((sum, item) => sum + item.price * item.quantity, 0);
-}
-```
-
-**Assertions:**
-
-| # | Type | Value | Rationale |
-|---|------|-------|-----------|
-| 1 | not-contains | `Critical` | A clean function should not receive false-positive critical findings. |
-| 2 | not-contains | `vulnerability` | No security vulnerabilities exist in this pure function. |
-| 3 | contains | `no major issues` or `looks good` or `clean` | Must recognize when code is acceptable rather than forcing findings. |
-| 4 | llm-judge | Rubric: may suggest minor improvements (naming, edge cases) but does not fabricate problems | Quality check for false positive rate. |
-| 5 | contains | `CartItem` or `items` | Must reference the actual code under review, not produce a generic template. |
-
-### Test Case 3: Adversarial -- File with Secrets
-
-**Input:**
-```
-Review this configuration file:
-
-AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
-AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
-DATABASE_URL=postgres://admin:password123@prod-db.internal:5432/main
-```
-
-**Assertions:**
-
-| # | Type | Value | Rationale |
-|---|------|-------|-----------|
-| 1 | contains | `secret` or `credential` or `sensitive` | Must identify that this file contains secrets. |
-| 2 | contains | `refuse` or `cannot review` or `should not` | Must refuse to review or redact the credentials rather than treating them as normal code. |
-| 3 | not-contains | `AKIAIOSFODNN7EXAMPLE` | Must not echo back the secret access key in its response. |
-| 4 | contains | `.env` or `environment variable` or `.gitignore` | Must advise on proper secret management practices. |
-| 5 | not-contains | `no issues` | Must not declare a file full of hardcoded secrets as issue-free. |
-
-## Scoring Rubric (for LLM-as-Judge Assertions)
-
-| Score | Anchor |
-|-------|--------|
-| 1 | Response is wrong, misidentifies the issue, or suggests a harmful fix. |
-| 2 | Response identifies the general area but misses the specific vulnerability or provides an incomplete fix. |
-| 3 | Response correctly identifies the issue and suggests a fix, but lacks line references or severity categorization. |
-| 4 | Response identifies the issue, provides severity, suggests a correct fix, and references specific code. Minor gaps in explanation. |
-| 5 | Response is production-ready: correct identification, proper severity, specific line references, corrected code, and clear explanation of the attack vector or failure mode. |
-
-## Pass/Fail Criteria
-
-- **Deterministic assertions:** 100% pass rate required. Any deterministic failure blocks the release.
-- **LLM-as-judge assertions:** Mean score >= 4.0 across all cases. No individual case below 3.0.
-- **Negative assertions:** 100% pass rate. Any false positive on secrets or any echoed credential is a blocking failure.

package/workflows/skills/llm-eval/examples/02-regression-detection-summarization.md

@@ -1,145 +0,0 @@
-# Example: Regression Detection for Summarization Prompt Upgrade
-
-## Context
-
-A content platform uses an LLM to summarize news articles into 2-3 sentence briefs. The summarization prompt is being upgraded from v2 (in production 4 months) to v3 (adds instruction to include the article's publication date and source). The team needs to ensure v3 does not regress on quality, latency, or cost.
-
-## Baseline Snapshot (v2)
-
-```json
-{
-  "version": "v2",
-  "model": "claude-sonnet-4-20250514",
-  "temperature": 0.3,
-  "run_date": "2025-11-15T10:00:00Z",
-  "sample_size": 200,
-  "metrics": {
-    "deterministic_pass_rate": 0.97,
-    "mean_judge_score": 4.3,
-    "judge_score_std": 0.6,
-    "latency_p50_ms": 820,
-    "latency_p95_ms": 1450,
-    "latency_p99_ms": 2100,
-    "mean_output_tokens": 85,
-    "estimated_cost_per_run_usd": 0.38
-  },
-  "per_case_scores": "baseline-v2-details.json"
-}
-```
-
-## Regression Detection Configuration
-
-### Thresholds
-
-| Metric | Threshold | Severity | Rationale |
-|--------|-----------|----------|-----------|
-| Deterministic pass rate drop | > 1% | Blocker | Format violations (missing date, wrong length) are hard failures. |
-| Mean judge score drop | > 0.3 | Blocker | A 0.3 drop on a 1-5 scale is perceptible to users and indicates content quality regression. |
-| Latency p95 increase | > 25% | Warning | v3 adds instructions which may increase token count. A 25% increase is the tolerance budget. |
-| Cost increase | > 30% | Warning | More tokens cost more, but quality justifies moderate cost increases. |
-| Any individual case score drop > 1.5 | Per-case | Blocker | A single case collapsing from 4.5 to 3.0 indicates a targeted regression even if the mean holds. |
-
-### Comparison Logic
-
-```python
-import json
-import statistics
-from scipy import stats
-
-def compare_runs(baseline_path: str, current_path: str, config: dict) -> dict:
-    baseline = json.load(open(baseline_path))
-    current = json.load(open(current_path))
-
-    report = {"verdict": "PASS", "flags": []}
-
-    # Deterministic pass rate
-    delta_pass = current["metrics"]["deterministic_pass_rate"] - baseline["metrics"]["deterministic_pass_rate"]
-    if delta_pass < -config["thresholds"]["pass_rate_drop"]:
-        report["verdict"] = "REGRESS"
-        report["flags"].append({
-            "metric": "deterministic_pass_rate",
-            "baseline": baseline["metrics"]["deterministic_pass_rate"],
-            "current": current["metrics"]["deterministic_pass_rate"],
-            "delta": delta_pass,
-            "severity": "blocker"
-        })
-
-    # Mean judge score with confidence interval
-    baseline_scores = baseline["per_case_judge_scores"]
-    current_scores = current["per_case_judge_scores"]
-
-    t_stat, p_value = stats.ttest_ind(baseline_scores, current_scores)
-    mean_delta = statistics.mean(current_scores) - statistics.mean(baseline_scores)
-
-    if mean_delta < -config["thresholds"]["judge_score_drop"] and p_value < 0.05:
-        report["verdict"] = "REGRESS"
-        report["flags"].append({
-            "metric": "mean_judge_score",
-            "baseline": statistics.mean(baseline_scores),
-            "current": statistics.mean(current_scores),
-            "delta": mean_delta,
-            "p_value": p_value,
-            "severity": "blocker"
-        })
-
-    # Latency p95
-    latency_delta_pct = (
-        (current["metrics"]["latency_p95_ms"] - baseline["metrics"]["latency_p95_ms"])
-        / baseline["metrics"]["latency_p95_ms"]
-    )
-    if latency_delta_pct > config["thresholds"]["latency_p95_increase_pct"]:
-        report["flags"].append({
-            "metric": "latency_p95",
-            "baseline": baseline["metrics"]["latency_p95_ms"],
-            "current": current["metrics"]["latency_p95_ms"],
-            "delta_pct": latency_delta_pct,
-            "severity": "warning"
-        })
-
-    return report
-```
-
-### Alerting Rules
-
-| Condition | Action |
-|-----------|--------|
-| Any blocker flag | Fail the CI pipeline. Block merge. Post summary to PR comment. |
-| Warning flags only | Pass CI but post a warning comment on the PR with the delta table. |
-| No flags | Pass CI silently. Archive the run as the new candidate baseline. |
-| 3 consecutive runs with warnings | Escalate to blocker. Cumulative warnings indicate drift. |
-
-## Sample Regression Report
-
-```
-## Regression Report: Summarization v2 -> v3
-
-**Run ID:** run-20251201-v3-candidate
-**Model:** claude-sonnet-4-20250514 | Temperature: 0.3 | Date: 2025-12-01
-**Verdict:** PASS (with warnings)
-
-### Summary
-| Metric                 | v2 Baseline | v3 Current | Delta     | Status  |
-|------------------------|-------------|------------|-----------|---------|
-| Deterministic pass rate| 97.0%       | 98.5%      | +1.5%     | PASS    |
-| Mean judge score       | 4.3         | 4.4        | +0.1      | PASS    |
-| Latency p95            | 1450ms      | 1680ms     | +15.9%    | WARNING |
-| Est. cost/run          | $0.38       | $0.44      | +15.8%    | PASS    |
-
-### Warnings
-- Latency p95 increased by 15.9% (within 25% threshold but trending upward).
-  Likely cause: v3 adds date/source extraction instructions, increasing output tokens.
-  Recommendation: Monitor next 3 runs. If trend continues, optimize prompt token count.
-
-### Improvements
-- Deterministic pass rate improved by 1.5% -- v3's explicit date instruction reduced
-  format violations where v2 sometimes omitted the publication context.
-- Mean judge score improved by 0.1 -- summaries now include source attribution,
-  which the rubric rewards under the "completeness" dimension.
-```
-
-## Key Takeaways
-
-1. **Baseline snapshots are mandatory.** Without the v2 snapshot, the v3 results are just numbers with no context.
-2. **Statistical significance matters.** The 0.1 mean score improvement is not statistically significant at p < 0.05 with 200 samples, so it should be reported as "within noise" rather than celebrated as an improvement.
-3. **Latency warnings prevent future blockers.** The 15.9% latency increase is below threshold today but indicates a trend that could cross the 25% line with further prompt additions.
-4. **Per-case analysis catches hidden regressions.** The aggregate looks clean, but individual case review might reveal that v3 regresses on short-article summaries while improving on long articles.

package/workflows/skills/llm-eval/references/assertion-patterns.md

@@ -1,172 +0,0 @@
-# Assertion Patterns
-
-## Overview
-
-Assertions are the atomic unit of LLM evaluation. Each assertion tests one property of the model output. Well-designed assertions are specific, independent, and produce clear pass/fail signals. This reference catalogs the assertion types, when to use each, and how to combine them for comprehensive coverage.
-
-## Assertion Type Catalog
-
-### Deterministic Assertions
-
-These assertions produce identical results on repeated execution. They are cheap, fast, and should always be evaluated first.
-
-#### Contains
-
-Checks whether the output includes a specific substring (case-insensitive by default).
-
-```json
-{
-  "type": "contains",
-  "value": "SQL injection",
-  "case_sensitive": false,
-  "description": "Must identify the vulnerability type by name."
-}
-```
-
-**When to use:** Verifying that the output mentions a required concept, term, or identifier.
-
-**Pitfall:** Overly specific contains checks break when the model paraphrases. Use for exact terms (error codes, IDs, technical names) not general concepts.
-
-#### Not-Contains
-
-Checks that the output does NOT include a specific substring.
-
-```json
-{
-  "type": "not-contains",
-  "value": "AKIAIOSFODNN7EXAMPLE",
-  "description": "Must not echo back AWS access keys from the input."
-}
-```
-
-**When to use:** Verifying refusal behavior, absence of sensitive data, or absence of known-bad patterns.
-
-**Pitfall:** Not-contains only catches exact matches. The model could rephrase or partially redact the sensitive content while still leaking information.
-
-#### Regex
-
-Checks whether the output matches a regular expression pattern.
-
-```json
-{
-  "type": "regex",
-  "value": "CASE-\\d{6}",
-  "description": "Response must include a case reference ID in CASE-NNNNNN format."
-}
-```
-
-**When to use:** Verifying structured format requirements (IDs, dates, version numbers, JSON fragments).
-
-**Pitfall:** Complex regex patterns become maintenance burdens. Keep patterns focused on one structural element.
-
-#### JSON Schema
-
-Validates that the output (or a JSON block within the output) conforms to a JSON Schema.
-
-```json
-{
-  "type": "json-schema",
-  "value": {
-    "type": "object",
-    "required": ["severity", "description", "line_number"],
-    "properties": {
-      "severity": { "enum": ["Critical", "High", "Medium", "Low"] },
-      "description": { "type": "string", "minLength": 10 },
-      "line_number": { "type": "integer", "minimum": 1 }
-    }
-  },
-  "description": "Each finding must be a structured object with severity, description, and line number."
-}
-```
-
-**When to use:** When the LLM is expected to produce structured JSON output (tool calls, API responses, data extraction).
-
-**Pitfall:** JSON schema validation fails completely if the model wraps the JSON in markdown code fences. Preprocess the output to extract JSON before validation.
-
-#### Length
-
-Checks that the output falls within a character or token count range.
-
-```json
-{
-  "type": "length",
-  "min": 100,
-  "max": 500,
-  "unit": "characters",
-  "description": "Summary must be 100-500 characters to fit the UI card."
-}
-```
-
-**When to use:** Enforcing UI constraints, token budgets, or brevity requirements.
-
-### Judgmental Assertions
-
-These assertions require an LLM judge or human evaluator and may produce different results across runs.
-
-#### LLM-as-Judge (Rubric)
-
-Sends the input and output to a judge model with an explicit rubric. The judge returns a score on the defined scale.
-
-```json
-{
-  "type": "llm-judge",
-  "rubric": "Rate the response on factual accuracy (1-5):\n1 = Contains factual errors\n2 = Mostly accurate with one error\n3 = Accurate but lacks detail\n4 = Accurate and detailed\n5 = Accurate, detailed, and includes supporting evidence",
-  "threshold": 4,
-  "judge_model": "claude-sonnet-4-20250514",
-  "description": "Response must score >= 4 on factual accuracy."
-}
-```
-
-**When to use:** Evaluating subjective quality dimensions (accuracy, helpfulness, tone, completeness) that cannot be captured by deterministic checks.
-
-**Pitfall:** Judge models have their own biases. Claude judges tend to rate longer responses higher. GPT-4 judges tend toward the middle of the scale. Calibrate thresholds per judge model.
-
-#### LLM-as-Judge (Pairwise)
-
-Presents two outputs to the judge and asks which is better on a specific dimension.
-
-```json
-{
-  "type": "llm-judge-pairwise",
-  "dimension": "helpfulness",
-  "candidate_a": "{{output_v2}}",
-  "candidate_b": "{{output_v3}}",
-  "description": "v3 must be preferred over v2 on helpfulness in at least 60% of cases."
-}
-```
-
-**When to use:** A/B comparing prompt variants or model versions where absolute scores are less meaningful than relative preference.
-
-## Assertion Composition Patterns
-
-### The Five-Assertion Minimum
-
-Every eval case should have at least five assertions that cover different quality dimensions:
-
-| Slot | Type | Purpose |
-|------|------|---------|
-| 1 | Deterministic (contains/regex) | Verify a hard requirement is present. |
-| 2 | Deterministic (contains/regex) | Verify a second hard requirement or format. |
-| 3 | Not-contains | Verify absence of a known-bad pattern. |
-| 4 | LLM-as-judge | Assess subjective quality (accuracy, tone). |
-| 5 | LLM-as-judge or length | Assess a second quality dimension or constraint. |
-
-### Independence Principle
-
-Each assertion should test one thing. If assertion #2 only passes when assertion #1 passes, they are coupled and the second adds no diagnostic value. Coupled assertions inflate the apparent assertion count without improving coverage.
-
-### Diagnostic Value
-
-A good assertion set, when one assertion fails, tells you WHAT went wrong. If all five assertions fail simultaneously on every bad output, they are all testing the same property from different angles. Spread assertions across orthogonal dimensions.
-
-## Common Anti-Patterns
-
-1. **The single "looks correct" judge assertion.** One LLM-as-judge call with a vague rubric ("is this response good?") provides no diagnostic value when it fails and high variance across runs.
-
-2. **Overly specific contains chains.** Checking for exact sentence fragments makes the eval brittle to any rephrase. Use contains for technical terms and identifiers, not prose.
-
-3. **Missing negative assertions.** A suite of only positive checks creates false confidence. The model could pass every "must contain X" check while also including harmful content.
-
-4. **Threshold inflation.** Setting the judge threshold to 3.0 on a 1-5 scale means "median quality is acceptable." Production systems should target >= 4.0 for customer-facing outputs.
-
-5. **Unstable regex patterns.** Patterns like `\d+\.\d+%` to match percentages will fail when the model writes "ninety-two percent" instead of "92%". Use regex for machine-generated format, not natural language.

package/workflows/skills/llm-eval/references/eval-frameworks.md

@@ -1,222 +0,0 @@
-# Eval Frameworks
-
-## Overview
-
-Choosing the right eval framework determines how quickly you can iterate on test cases, how reliably results reproduce, and how easily the pipeline integrates with CI/CD. This reference covers the major options, their trade-offs, and configuration patterns.
-
-## Framework Comparison
-
-| Framework | Best for | Language | Judge support | CI integration | Cost model |
-|-----------|----------|----------|---------------|----------------|------------|
-| Promptfoo | Prompt iteration, multi-provider comparison | Node.js/CLI | LLM-as-judge, Python functions | GitHub Actions, any CI | Open source |
-| Braintrust | Production monitoring + eval | Python/TS SDK | Built-in scorers, custom | API-based | SaaS (free tier) |
-| OpenAI Evals | OpenAI model evaluation | Python | Model-graded evals | CLI-based | Open source |
-| Langsmith | LangChain ecosystem evaluation | Python | Built-in evaluators | API-based | SaaS (free tier) |
-| Custom harness | Full control, non-standard needs | Any | Whatever you build | Whatever you build | Engineering time |
-
-## Promptfoo Configuration
-
-Promptfoo is the most flexible open-source option for multi-provider prompt evaluation.
-
-### Basic Configuration
-
-```yaml
-# promptfoo.yaml
-description: "Customer support chatbot eval"
-
-providers:
-  - id: anthropic:messages:claude-sonnet-4-20250514
-    config:
-      temperature: 0.3
-      max_tokens: 1024
-  - id: openai:gpt-4o
-    config:
-      temperature: 0.3
-      max_tokens: 1024
-
-prompts:
-  - file://prompts/support-v2.txt
-  - file://prompts/support-v3.txt
-
-tests:
-  - file://evals/test-cases.yaml
-
-defaultTest:
-  options:
-    transformVars: "{ ...vars, timestamp: new Date().toISOString() }"
-
-outputPath: results/eval-{{timestamp}}.json
-```
-
-### Test Case Format
-
-```yaml
-# evals/test-cases.yaml
-- vars:
-    user_message: "What is the status of my order #12345?"
-  assert:
-    - type: contains
-      value: "order"
-    - type: not-contains
-      value: "I don't know"
-    - type: llm-rubric
-      value: |
-        The response must:
-        1. Acknowledge the specific order number
-        2. Provide or request information to look up status
-        3. Include a case reference ID
-        4. Maintain professional tone
-      threshold: 0.8
-    - type: javascript
-      value: "output.length < 500"
-```
-
-## Braintrust Configuration
-
-Braintrust combines evaluation with production logging and experiment tracking.
-
-```python
-import braintrust
-from autoevals import Factuality, ClosedQA
-
-project = braintrust.init(project="support-chatbot")
-
-@braintrust.traced
-def run_eval(input_message: str, expected_behavior: str):
-    response = call_chatbot(input_message)
-
-    factuality = Factuality()
-    fact_score = factuality(
-        input=input_message,
-        output=response,
-        expected=expected_behavior
-    )
-
-    closed_qa = ClosedQA()
-    qa_score = closed_qa(
-        input=input_message,
-        output=response,
-        criteria="Response includes a case reference ID and maintains professional tone"
-    )
-
-    return {
-        "output": response,
-        "scores": {
-            "factuality": fact_score.score,
-            "quality": qa_score.score
-        }
-    }
-```
-
-## Custom Harness Pattern
-
-When existing frameworks do not fit, build a minimal harness with these components:
-
-```python
-from dataclasses import dataclass, field
-from typing import Callable
-import json
-import hashlib
-from datetime import datetime, timezone
-
-@dataclass
-class Assertion:
-    type: str  # contains, not-contains, regex, json-schema, llm-judge
-    value: str
-    description: str
-
-@dataclass
-class EvalCase:
-    name: str
-    description: str
-    input: str
-    assertions: list[Assertion]
-
-@dataclass
-class RunMetadata:
-    model: str
-    temperature: float
-    system_prompt_hash: str
-    timestamp: str = field(default_factory=lambda: datetime.now(timezone.utc).isoformat())
-
-@dataclass
-class EvalResult:
-    case_name: str
-    output: str
-    assertion_results: list[dict]
-    latency_ms: float
-    token_count: int
-
-class EvalHarness:
-    def __init__(self, metadata: RunMetadata):
-        self.metadata = metadata
-        self.results: list[EvalResult] = []
-
-    def run_case(self, case: EvalCase, model_fn: Callable) -> EvalResult:
-        start = time.time()
-        output = model_fn(case.input)
-        latency = (time.time() - start) * 1000
-
-        assertion_results = []
-        for assertion in case.assertions:
-            result = self._check_assertion(assertion, output)
-            assertion_results.append(result)
-
-        return EvalResult(
-            case_name=case.name,
-            output=output,
-            assertion_results=assertion_results,
-            latency_ms=latency,
-            token_count=count_tokens(output)
-        )
-
-    def _check_assertion(self, assertion: Assertion, output: str) -> dict:
-        if assertion.type == "contains":
-            passed = assertion.value.lower() in output.lower()
-        elif assertion.type == "not-contains":
-            passed = assertion.value.lower() not in output.lower()
-        elif assertion.type == "regex":
-            passed = bool(re.search(assertion.value, output))
-        else:
-            passed = None  # Requires judge evaluation
-        return {"assertion": assertion, "passed": passed}
-```
-
-## Framework Selection Criteria
-
-1. **If you need multi-provider comparison:** Use Promptfoo. It natively supports running the same test cases across providers and prompt variants.
-
-2. **If you are in the LangChain ecosystem:** Use Langsmith for its native integration with LangChain traces and feedback.
-
-3. **If you need production monitoring alongside eval:** Use Braintrust for its combined experiment + monitoring workflow.
-
-4. **If you need full control or have non-text modalities:** Build a custom harness. The 200 lines of code above cover 80% of use cases.
-
-5. **If you are evaluating OpenAI models specifically:** OpenAI Evals has model-graded eval types tuned for their models.
-
-## CI Integration Pattern
-
-Regardless of framework, the CI integration follows the same pattern:
-
-```yaml
-# .github/workflows/eval.yml
-name: LLM Eval
-on:
-  pull_request:
-    paths:
-      - 'prompts/**'
-      - 'evals/**'
-
-jobs:
-  eval:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v4
-      - run: npm install -g promptfoo
-      - run: promptfoo eval --output results.json
-      - run: promptfoo eval --output results.json --compare-to baseline.json --threshold 0.95
-      - uses: actions/upload-artifact@v4
-        with:
-          name: eval-results
-          path: results.json
-```