@cubis/foundry 0.3.83 → 0.3.84

package/workflows/workflows/agent-environment-setup/platforms/copilot/skills/llm-eval/references/eval-frameworks.md

@@ -1,222 +0,0 @@
-# Eval Frameworks
-
-## Overview
-
-Choosing the right eval framework determines how quickly you can iterate on test cases, how reliably results reproduce, and how easily the pipeline integrates with CI/CD. This reference covers the major options, their trade-offs, and configuration patterns.
-
-## Framework Comparison
-
-| Framework | Best for | Language | Judge support | CI integration | Cost model |
-|-----------|----------|----------|---------------|----------------|------------|
-| Promptfoo | Prompt iteration, multi-provider comparison | Node.js/CLI | LLM-as-judge, Python functions | GitHub Actions, any CI | Open source |
-| Braintrust | Production monitoring + eval | Python/TS SDK | Built-in scorers, custom | API-based | SaaS (free tier) |
-| OpenAI Evals | OpenAI model evaluation | Python | Model-graded evals | CLI-based | Open source |
-| Langsmith | LangChain ecosystem evaluation | Python | Built-in evaluators | API-based | SaaS (free tier) |
-| Custom harness | Full control, non-standard needs | Any | Whatever you build | Whatever you build | Engineering time |
-
-## Promptfoo Configuration
-
-Promptfoo is the most flexible open-source option for multi-provider prompt evaluation.
-
-### Basic Configuration
-
-```yaml
-# promptfoo.yaml
-description: "Customer support chatbot eval"
-
-providers:
-  - id: anthropic:messages:claude-sonnet-4-20250514
-    config:
-      temperature: 0.3
-      max_tokens: 1024
-  - id: openai:gpt-4o
-    config:
-      temperature: 0.3
-      max_tokens: 1024
-
-prompts:
-  - file://prompts/support-v2.txt
-  - file://prompts/support-v3.txt
-
-tests:
-  - file://evals/test-cases.yaml
-
-defaultTest:
-  options:
-    transformVars: "{ ...vars, timestamp: new Date().toISOString() }"
-
-outputPath: results/eval-{{timestamp}}.json
-```
-
-### Test Case Format
-
-```yaml
-# evals/test-cases.yaml
-- vars:
-    user_message: "What is the status of my order #12345?"
-  assert:
-    - type: contains
-      value: "order"
-    - type: not-contains
-      value: "I don't know"
-    - type: llm-rubric
-      value: |
-        The response must:
-        1. Acknowledge the specific order number
-        2. Provide or request information to look up status
-        3. Include a case reference ID
-        4. Maintain professional tone
-      threshold: 0.8
-    - type: javascript
-      value: "output.length < 500"
-```
-
-## Braintrust Configuration
-
-Braintrust combines evaluation with production logging and experiment tracking.
-
-```python
-import braintrust
-from autoevals import Factuality, ClosedQA
-
-project = braintrust.init(project="support-chatbot")
-
-@braintrust.traced
-def run_eval(input_message: str, expected_behavior: str):
-    response = call_chatbot(input_message)
-
-    factuality = Factuality()
-    fact_score = factuality(
-        input=input_message,
-        output=response,
-        expected=expected_behavior
-    )
-
-    closed_qa = ClosedQA()
-    qa_score = closed_qa(
-        input=input_message,
-        output=response,
-        criteria="Response includes a case reference ID and maintains professional tone"
-    )
-
-    return {
-        "output": response,
-        "scores": {
-            "factuality": fact_score.score,
-            "quality": qa_score.score
-        }
-    }
-```
-
-## Custom Harness Pattern
-
-When existing frameworks do not fit, build a minimal harness with these components:
-
-```python
-from dataclasses import dataclass, field
-from typing import Callable
-import json
-import hashlib
-from datetime import datetime, timezone
-
-@dataclass
-class Assertion:
-    type: str  # contains, not-contains, regex, json-schema, llm-judge
-    value: str
-    description: str
-
-@dataclass
-class EvalCase:
-    name: str
-    description: str
-    input: str
-    assertions: list[Assertion]
-
-@dataclass
-class RunMetadata:
-    model: str
-    temperature: float
-    system_prompt_hash: str
-    timestamp: str = field(default_factory=lambda: datetime.now(timezone.utc).isoformat())
-
-@dataclass
-class EvalResult:
-    case_name: str
-    output: str
-    assertion_results: list[dict]
-    latency_ms: float
-    token_count: int
-
-class EvalHarness:
-    def __init__(self, metadata: RunMetadata):
-        self.metadata = metadata
-        self.results: list[EvalResult] = []
-
-    def run_case(self, case: EvalCase, model_fn: Callable) -> EvalResult:
-        start = time.time()
-        output = model_fn(case.input)
-        latency = (time.time() - start) * 1000
-
-        assertion_results = []
-        for assertion in case.assertions:
-            result = self._check_assertion(assertion, output)
-            assertion_results.append(result)
-
-        return EvalResult(
-            case_name=case.name,
-            output=output,
-            assertion_results=assertion_results,
-            latency_ms=latency,
-            token_count=count_tokens(output)
-        )
-
-    def _check_assertion(self, assertion: Assertion, output: str) -> dict:
-        if assertion.type == "contains":
-            passed = assertion.value.lower() in output.lower()
-        elif assertion.type == "not-contains":
-            passed = assertion.value.lower() not in output.lower()
-        elif assertion.type == "regex":
-            passed = bool(re.search(assertion.value, output))
-        else:
-            passed = None  # Requires judge evaluation
-        return {"assertion": assertion, "passed": passed}
-```
-
-## Framework Selection Criteria
-
-1. **If you need multi-provider comparison:** Use Promptfoo. It natively supports running the same test cases across providers and prompt variants.
-
-2. **If you are in the LangChain ecosystem:** Use Langsmith for its native integration with LangChain traces and feedback.
-
-3. **If you need production monitoring alongside eval:** Use Braintrust for its combined experiment + monitoring workflow.
-
-4. **If you need full control or have non-text modalities:** Build a custom harness. The 200 lines of code above cover 80% of use cases.
-
-5. **If you are evaluating OpenAI models specifically:** OpenAI Evals has model-graded eval types tuned for their models.
-
-## CI Integration Pattern
-
-Regardless of framework, the CI integration follows the same pattern:
-
-```yaml
-# .github/workflows/eval.yml
-name: LLM Eval
-on:
-  pull_request:
-    paths:
-      - 'prompts/**'
-      - 'evals/**'
-
-jobs:
-  eval:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v4
-      - run: npm install -g promptfoo
-      - run: promptfoo eval --output results.json
-      - run: promptfoo eval --output results.json --compare-to baseline.json --threshold 0.95
-      - uses: actions/upload-artifact@v4
-        with:
-          name: eval-results
-          path: results.json
-```

package/workflows/workflows/agent-environment-setup/platforms/copilot/skills/llm-eval/references/human-eval.md

@@ -1,212 +0,0 @@
-# Human Evaluation
-
-## Overview
-
-Human evaluation remains the gold standard for subjective quality dimensions that LLM judges cannot reliably assess: nuance, cultural appropriateness, domain expertise, and user satisfaction. This reference covers campaign design, annotator calibration, agreement metrics, and integration with automated pipelines.
-
-## When Human Eval is Necessary
-
-| Scenario | Why automated eval falls short |
-|----------|-------------------------------|
-| Launching a new product vertical | No existing rubric or baseline to calibrate a judge model against. |
-| Assessing cultural sensitivity | LLM judges lack the cultural context to detect subtle inappropriateness. |
-| Domain-expert validation | Medical, legal, or financial accuracy requires credentialed reviewers. |
-| Calibrating LLM-as-judge | Human scores provide the ground truth needed to validate and tune judge prompts. |
-| High-stakes decisions | Automated pass rates should not be the sole evidence for decisions affecting user safety. |
-
-## Campaign Design
-
-### Task Definition
-
-Write the task description as if explaining to a new annotator who has never seen the product. Include:
-
-1. **Context:** What the LLM is doing and who uses it.
-2. **Input format:** What the annotator will see (user query + LLM response).
-3. **Scoring dimensions:** Exactly what to evaluate and what each score means.
-4. **Examples:** At least 2 scored examples per dimension with explanations.
-
-```markdown
-## Task: Rate Customer Support Responses
-
-You will review responses from an AI customer support assistant. For each
-response, you will see the customer's message and the AI's reply.
-
-Rate each response on two dimensions:
-
-### Accuracy (1-5)
-- 1: Response contains factual errors or contradicts the query.
-- 2: Response is partially accurate but missing key information.
-- 3: Response is accurate but generic, not tailored to the query.
-- 4: Response is accurate and specifically addresses the query.
-- 5: Response is accurate, specific, and provides additional helpful context.
-
-### Tone (1-5)
-- 1: Response is rude, dismissive, or robotic.
-- 2: Response is neutral but impersonal.
-- 3: Response is polite but formulaic.
-- 4: Response is warm, professional, and empathetic.
-- 5: Response demonstrates genuine understanding and proactive care.
-```
-
-### Sample Size
-
-Calculate the required sample size based on desired confidence and expected variance:
-
-```python
-import math
-
-def required_sample_size(
-    confidence: float = 0.95,
-    margin_of_error: float = 0.3,
-    estimated_std: float = 1.0
-) -> int:
-    """For a 1-5 scale with std ~1.0, margin of error 0.3, need ~43 samples."""
-    z = 1.96 if confidence == 0.95 else 2.576  # 95% or 99%
-    n = (z * estimated_std / margin_of_error) ** 2
-    return math.ceil(n)
-```
-
-**Rules of thumb:**
-- 50 samples: Sufficient for directional signal (is v3 better than v2?).
-- 200 samples: Sufficient for confident per-dimension scoring.
-- 500+ samples: Required for sub-group analysis (performance by input type).
-
-### Annotator Selection
-
-| Type | Use for | Cost | Quality |
-|------|---------|------|---------|
-| Internal team | Rapid iteration, domain knowledge | Low (staff time) | High but potential bias |
-| Contracted annotators | Scale, objectivity | Medium ($15-40/hr) | Medium (requires training) |
-| Crowdsource (MTurk, Prolific) | Large-scale, diverse perspectives | Low ($0.10-1.00/task) | Variable (requires QC) |
-| Domain experts | Medical, legal, financial accuracy | High ($50-200/hr) | Highest for domain tasks |
-
-## Annotator Calibration
-
-### Golden Set
-
-Create 10-20 pre-scored examples (the "golden set") that annotators must score before starting the real task. Compare their scores against the gold scores.
-
-```python
-def calibration_score(annotator_scores: list[int], gold_scores: list[int]) -> float:
-    """Returns the mean absolute deviation from gold scores."""
-    deviations = [abs(a - g) for a, g in zip(annotator_scores, gold_scores)]
-    return statistics.mean(deviations)
-
-# Acceptable: MAD < 0.5 on a 1-5 scale
-# Marginal: MAD 0.5-1.0, requires discussion
-# Fail: MAD > 1.0, annotator needs retraining or removal
-```
-
-### Calibration Session
-
-Before large campaigns, hold a 30-minute calibration session:
-
-1. Present 5 examples from the golden set.
-2. Have annotators score independently.
-3. Reveal gold scores and discuss disagreements.
-4. Re-score 3 additional examples to verify alignment.
-5. Document any rubric clarifications that emerged.
-
-### Ongoing Quality Control
-
-Embed golden-set items (10-15% of the total) throughout the real task. Monitor annotator scores on these items to detect drift.
-
-```python
-def monitor_quality(annotator_id: str, golden_results: list[dict]) -> dict:
-    recent_mad = calibration_score(
-        [r["annotator_score"] for r in golden_results[-10:]],
-        [r["gold_score"] for r in golden_results[-10:]]
-    )
-    return {
-        "annotator_id": annotator_id,
-        "recent_mad": recent_mad,
-        "status": "ok" if recent_mad < 0.5 else "flagged"
-    }
-```
-
-## Inter-Annotator Agreement
-
-### Cohen's Kappa (2 Annotators)
-
-Measures agreement between two annotators, adjusted for chance agreement.
-
-```python
-from sklearn.metrics import cohen_kappa_score
-
-def compute_kappa(scores_a: list[int], scores_b: list[int]) -> float:
-    return cohen_kappa_score(scores_a, scores_b, weights="linear")
-
-# Interpretation:
-# < 0.20: Poor agreement - rubric is ambiguous
-# 0.21-0.40: Fair - rubric needs refinement
-# 0.41-0.60: Moderate - acceptable for subjective tasks
-# 0.61-0.80: Substantial - good rubric clarity
-# > 0.80: Almost perfect - typical for objective tasks only
-```
-
-### Krippendorff's Alpha (3+ Annotators)
-
-Generalizes to any number of annotators and handles missing data.
-
-```python
-import krippendorff
-
-def compute_alpha(reliability_data: list[list]) -> float:
-    """
-    reliability_data: matrix where rows are annotators, columns are items.
-    Use None for missing annotations.
-    """
-    return krippendorff.alpha(
-        reliability_data=reliability_data,
-        level_of_measurement="ordinal"
-    )
-```
-
-### What to Do When Agreement is Low
-
-1. **Review the rubric.** Low agreement usually means the rubric is ambiguous, not that annotators are bad. Identify the specific score boundaries that cause disagreement.
-
-2. **Add examples to the rubric.** For every boundary that causes disagreement (is this a 3 or a 4?), add a concrete example that anchors the distinction.
-
-3. **Split the dimension.** If "quality" has low agreement, try splitting into "accuracy" and "helpfulness." Narrower dimensions produce higher agreement.
-
-4. **Accept the variance.** Some dimensions (e.g., "creativity") have inherently low agreement. In those cases, use more annotators and report the distribution rather than the mean.
-
-## Integrating Human Eval with Automated Pipelines
-
-### Hybrid Workflow
-
-1. **Automated eval runs on every PR** with deterministic assertions and LLM-as-judge.
-2. **Human eval runs weekly** on a stratified sample of production outputs.
-3. **Human scores calibrate the judge.** Compare human and judge scores on the same items. If they diverge, update the judge rubric.
-4. **Quarterly rubric review.** Revise the rubric, golden set, and judge prompts based on accumulated human eval data.
-
-### Judge Calibration from Human Data
-
-```python
-def calibrate_judge(human_scores: list[float], judge_scores: list[float]) -> dict:
-    correlation = statistics.correlation(human_scores, judge_scores)
-    bias = statistics.mean(judge_scores) - statistics.mean(human_scores)
-
-    return {
-        "correlation": correlation,
-        "bias": bias,
-        "recommendation": (
-            "Judge is well-calibrated" if correlation > 0.7 and abs(bias) < 0.3
-            else "Judge needs rubric adjustment"
-        )
-    }
-```
-
-**Target:** Pearson correlation > 0.7 and absolute bias < 0.3 between human and judge scores. Below these thresholds, the judge is not a reliable proxy for human evaluation on this dimension.
-
-## Cost Estimation
-
-| Scale | Annotators | Items | Dimensions | Est. Hours | Est. Cost |
-|-------|------------|-------|------------|------------|-----------|
-| Pilot | 3 | 50 | 2 | 5 | $200 |
-| Standard | 3 | 200 | 3 | 30 | $1,200 |
-| Large | 5 | 500 | 3 | 100 | $4,000 |
-| Enterprise | 5 | 1000 | 5 | 300 | $12,000 |
-
-These estimates assume contracted annotators at $40/hr, including calibration time. Crowdsource costs are 5-10x lower but require more QC overhead.

package/workflows/workflows/agent-environment-setup/platforms/copilot/skills/llm-eval/references/regression-detection.md

@@ -1,243 +0,0 @@
-# Regression Detection
-
-## Overview
-
-Regression detection answers one question: did the latest change make things worse? This reference covers baseline management, comparison algorithms, threshold configuration, drift detection, and alerting integration.
-
-## Baseline Management
-
-### What to Capture
-
-A baseline snapshot must contain everything needed to reproduce the run and compare future runs against it.
-
-```json
-{
-  "baseline_id": "baseline-2025-11-15-v2",
-  "created_at": "2025-11-15T10:00:00Z",
-  "prompt_version": "v2",
-  "model": "claude-sonnet-4-20250514",
-  "model_version": "20250514",
-  "temperature": 0.3,
-  "system_prompt_hash": "sha256:a1b2c3d4...",
-  "eval_suite_hash": "sha256:e5f6g7h8...",
-  "sample_size": 200,
-  "aggregate_metrics": {
-    "deterministic_pass_rate": 0.97,
-    "mean_judge_score": 4.3,
-    "judge_score_std": 0.6,
-    "latency_p50_ms": 820,
-    "latency_p95_ms": 1450,
-    "mean_output_tokens": 85,
-    "estimated_cost_usd": 0.38
-  },
-  "per_case_results": [
-    {
-      "case_name": "order-status-happy",
-      "deterministic_pass": true,
-      "judge_score": 4.5,
-      "latency_ms": 780
-    }
-  ]
-}
-```
-
-### Baseline Lifecycle
-
-1. **Candidate baseline:** Created after every eval run. Not yet promoted.
-2. **Active baseline:** The reference point for regression checks. Promoted manually or after N consecutive clean runs.
-3. **Archived baseline:** Previous active baselines, kept for historical trend analysis.
-
-### Storage
-
-Store baselines as versioned JSON files alongside the eval suite:
-
-```
-evals/
-  baselines/
-    baseline-2025-09-01-v1.json
-    baseline-2025-11-15-v2.json  <-- active
-  test-cases.yaml
-  evals.json
-```
-
-## Comparison Algorithms
-
-### Point Estimate Comparison
-
-The simplest approach: compute the delta between current and baseline metrics.
-
-```python
-def compare_metric(current: float, baseline: float) -> dict:
-    delta = current - baseline
-    delta_pct = (delta / baseline) * 100 if baseline != 0 else float('inf')
-    return {"delta": delta, "delta_pct": delta_pct}
-```
-
-**Limitation:** Point estimates ignore variance. A 2% drop might be noise on small samples.
-
-### Statistical Significance Testing
-
-For judge scores and other continuous metrics, use a t-test or Mann-Whitney U test to determine if the difference is statistically significant.
-
-```python
-from scipy import stats
-
-def compare_with_significance(
-    current_scores: list[float],
-    baseline_scores: list[float],
-    alpha: float = 0.05
-) -> dict:
-    t_stat, p_value = stats.ttest_ind(current_scores, baseline_scores)
-    mean_delta = statistics.mean(current_scores) - statistics.mean(baseline_scores)
-
-    return {
-        "mean_delta": mean_delta,
-        "p_value": p_value,
-        "significant": p_value < alpha,
-        "direction": "regression" if mean_delta < 0 else "improvement"
-    }
-```
-
-### Confidence Intervals
-
-Report the confidence interval of the delta, not just the point estimate.
-
-```python
-def confidence_interval(scores: list[float], confidence: float = 0.95) -> tuple:
-    n = len(scores)
-    mean = statistics.mean(scores)
-    se = statistics.stdev(scores) / (n ** 0.5)
-    t_crit = stats.t.ppf((1 + confidence) / 2, n - 1)
-    return (mean - t_crit * se, mean + t_crit * se)
-```
-
-A 95% CI of [4.1, 4.5] with a baseline of 4.3 means the current run is consistent with the baseline. A CI of [3.8, 4.1] with a baseline of 4.3 signals a likely regression.
-
-## Threshold Configuration
-
-### Per-Metric Thresholds
-
-| Metric | Threshold | Rationale |
-|--------|-----------|-----------|
-| Deterministic pass rate | > 1% drop | Format violations are hard failures. Even 1% is notable. |
-| Mean judge score | > 0.3 drop | On a 1-5 scale, 0.3 is the smallest perceptible quality difference. |
-| Latency p95 | > 25% increase | User experience degrades noticeably above this. |
-| Cost per run | > 50% increase | Cost is elastic but needs a cap. |
-| Any single case score | > 1.5 drop | Catches targeted regressions hidden by aggregate means. |
-
-### Severity Levels
-
-- **Blocker:** Fails the CI pipeline. Must be resolved before merge.
-- **Warning:** Passes CI but posts a visible alert. Must be reviewed.
-- **Info:** Logged for trend analysis. No immediate action required.
-
-### Dynamic Thresholds
-
-For long-running projects, compute thresholds from historical variance rather than static values:
-
-```python
-def dynamic_threshold(historical_scores: list[float], sigma_multiplier: float = 2.0) -> float:
-    mean = statistics.mean(historical_scores)
-    std = statistics.stdev(historical_scores)
-    return mean - sigma_multiplier * std
-```
-
-A score below `mean - 2*std` is an outlier with 95% confidence, indicating a real regression rather than normal variance.
-
-## Drift Detection
-
-Drift is gradual degradation that stays within per-run thresholds but accumulates over time.
-
-### Moving Average
-
-Track a rolling average of scores over the last N runs. A downward trend that persists for 5+ runs signals drift even if no single run breaches the threshold.
-
-```python
-def detect_drift(run_history: list[dict], window: int = 5) -> dict:
-    if len(run_history) < window:
-        return {"drift_detected": False, "reason": "insufficient history"}
-
-    recent = [r["mean_judge_score"] for r in run_history[-window:]]
-    older = [r["mean_judge_score"] for r in run_history[-2*window:-window]]
-
-    recent_mean = statistics.mean(recent)
-    older_mean = statistics.mean(older)
-
-    drift = older_mean - recent_mean
-    return {
-        "drift_detected": drift > 0.15,
-        "drift_magnitude": drift,
-        "window": window,
-        "recommendation": "Investigate prompt or data changes in last 5 runs" if drift > 0.15 else "No action"
-    }
-```
-
-### CUSUM (Cumulative Sum)
-
-A more sensitive drift detection algorithm that accumulates small deviations:
-
-```python
-def cusum(scores: list[float], target: float, threshold: float = 3.0, slack: float = 0.5) -> int:
-    """Returns the index where drift is detected, or -1 if no drift."""
-    cumsum_pos = 0
-    cumsum_neg = 0
-    for i, score in enumerate(scores):
-        cumsum_pos = max(0, cumsum_pos + (target - score) - slack)
-        cumsum_neg = max(0, cumsum_neg + (score - target) - slack)
-        if cumsum_pos > threshold or cumsum_neg > threshold:
-            return i
-    return -1
-```
-
-## Alerting Integration
-
-### CI Pipeline Gate
-
-```yaml
-# GitHub Actions example
-- name: Run regression check
-  run: |
-    python eval/regression.py \
-      --current results/current.json \
-      --baseline evals/baselines/active.json \
-      --thresholds evals/thresholds.yaml
-  continue-on-error: false  # Blockers fail the job
-
-- name: Post PR comment
-  if: always()
-  run: |
-    python eval/post_comment.py \
-      --report results/regression-report.md \
-      --pr ${{ github.event.pull_request.number }}
-```
-
-### Slack Notification
-
-```python
-def send_slack_alert(report: dict, webhook_url: str):
-    severity = "blocker" if report["verdict"] == "REGRESS" else "warning"
-    color = "#FF0000" if severity == "blocker" else "#FFA500"
-
-    payload = {
-        "attachments": [{
-            "color": color,
-            "title": f"Eval Regression: {report['verdict']}",
-            "text": format_report_summary(report),
-            "fields": [
-                {"title": "Run ID", "value": report["run_id"], "short": True},
-                {"title": "Flags", "value": str(len(report["flags"])), "short": True}
-            ]
-        }]
-    }
-    requests.post(webhook_url, json=payload)
-```
-
-## Sample Workflow
-
-1. Developer changes a prompt and opens a PR.
-2. CI runs the eval suite against the changed prompt.
-3. Regression checker loads the active baseline and compares.
-4. If blocker: CI fails, PR comment explains the regression.
-5. If warning: CI passes, PR comment flags the metric.
-6. If clean: CI passes, candidate baseline is archived for potential promotion.
-7. After merge, the next 3 clean production runs promote the candidate to active baseline.