mindforge-cc 10.0.3 → 10.7.0

package/.mindforge/skills/embedding-systems/SKILL.md

@@ -0,0 +1,55 @@
+---
+name: embedding-systems
+version: 1.0.0
+min_mindforge_version: 10.5.0
+status: stable
+triggers: embedding system design, vector database architecture, semantic search implementation, embedding pipeline, similarity algorithm, vector index optimization, vector embedding model selection, vector similarity search, dense vector retrieval, embedding dimension reduction, hybrid search design, vector store scaling
+compose:
+  - rag-architecture
+---
+
+# Embedding Systems & Vector Databases
+
+## When this skill activates
+
+This skill activates when designing semantic search systems, implementing vector databases, building dense retrieval pipelines, or optimizing embedding-based similarity search. It applies to any system where AI must find semantically related content (documents, images, code) using vector representations.
+
+## Mandatory actions when this skill is active
+
+### Before writing any code
+
+1. **Select embedding model** — Choose based on modality (text: sentence-transformers, OpenAI ada-002; code: CodeBERT, StarCoder; images: CLIP; multimodal: CLIP, ImageBind) and performance requirements (dimension size vs. accuracy trade-off, encoding speed, cost). Validate model on your domain data before committing.
+2. **Design vector schema** — Define embedding dimensions (384, 768, 1536), metadata fields (timestamps, tags, source IDs), and filtering constraints. Schema changes require full reindexing. Get it right upfront.
+3. **Choose vector database** — Evaluate based on scale (millions vs. billions of vectors), query latency requirements (<100ms for real-time, <1s for batch), indexing strategy (HNSW, IVF, Product Quantization), and ecosystem (Pinecone, Weaviate, Qdrant, Milvus, pgvector). Run benchmarks on your data before selecting.
+4. **Establish similarity metrics** — Select distance function: cosine similarity (most common, normalized), dot product (unnormalized, faster), Euclidean distance (position-sensitive). Different metrics produce different rankings. Test which aligns with human judgment.
+
+### During implementation
+
+- **Normalize embeddings consistently** — Always L2-normalize embeddings before storage if using cosine similarity. Unnormalized embeddings produce incorrect rankings. Validate normalization: check that ||embedding|| = 1.0 for all vectors.
+- **Batch encode for efficiency** — Encode embeddings in batches (32-256 examples) to maximize GPU utilization. Single-example encoding wastes 90%+ of GPU compute. Implement batching with dynamic padding to handle variable-length inputs.
+- **Design hybrid search** — Combine dense retrieval (semantic similarity) with sparse retrieval (BM25, TF-IDF keyword matching). Hybrid search outperforms either alone: semantic search finds conceptually similar content, keyword search ensures exact term matches are included. Fuse rankings with reciprocal rank fusion (RRF).
+- **Implement metadata filtering** — Support filtering by metadata (date ranges, categories, tags) before or after vector search. Pre-filtering (filter then search) is faster but less accurate. Post-filtering (search then filter) is more accurate but slower. Choose based on selectivity (how many vectors match the filter).
+- **Optimize index parameters** — Tune HNSW parameters (M, efConstruction, efSearch) for your accuracy/latency trade-off. Higher M and efConstruction improve accuracy but slow indexing. Higher efSearch improves query accuracy but slows search. Benchmark with your data.
+- **Handle embedding drift** — Embedding models change (model updates, fine-tuning). When embeddings change, you must reindex all vectors. Implement versioned indexes: maintain old index while building new index, then hot-swap. Monitor query quality after model changes.
+
+### After implementation
+
+- **Validate recall accuracy** — Measure recall@k: for a set of ground-truth similar pairs, what percentage are in the top-k search results? Target: recall@10 >90%. If lower, increase k, tune index parameters, or use a better embedding model.
+- **Benchmark query latency** — Measure p50, p95, p99 latency under realistic load (queries per second, index size). Target: p95 <100ms for real-time search. If higher, scale horizontally, optimize index, or use approximate search (lower accuracy, higher speed).
+- **Test edge cases** — Query with empty strings, extremely long texts, rare languages, special characters. Ensure system degrades gracefully (return empty results, not crashes). Validate that out-of-domain queries (content very different from training data) still return sensible results.
+- **Monitor index size and cost** — Track storage cost (GB per million vectors), indexing throughput (vectors/second), and query cost ($/1000 queries). Embedding systems can become expensive at scale. Optimize dimension size (use dimension reduction if accuracy is acceptable).
+
+## Self-check before task completion
+
+- [ ] Embedding model is selected and validated on domain-specific data
+- [ ] Vector schema (dimensions, metadata, filters) is defined and documented
+- [ ] Vector database is chosen based on benchmarks with realistic data scale and latency
+- [ ] Similarity metric (cosine/dot product/Euclidean) is selected and justified
+- [ ] Embeddings are L2-normalized before storage (if using cosine similarity)
+- [ ] Batch encoding is implemented with dynamic padding for efficiency
+- [ ] Hybrid search combines dense (semantic) and sparse (keyword) retrieval with RRF fusion
+- [ ] Metadata filtering is implemented with pre/post-filtering strategy documented
+- [ ] Recall@10 is validated at >90% on ground-truth similar pairs
+- [ ] Query latency p95 is <100ms under realistic load
+- [ ] Edge cases (empty queries, long texts, out-of-domain) are tested and handled gracefully
+- [ ] Storage cost, indexing throughput, and query cost are measured and acceptable

package/.mindforge/skills/environment-management/SKILL.md

@@ -0,0 +1,54 @@
+---
+name: environment-management
+version: 1.0.0
+min_mindforge_version: 10.7.0
+status: stable
+triggers: environment management platform, preview environment design, ephemeral environment, environment parity, configuration drift detection, ephemeral environment provisioning, staging environment, environment lifecycle, environment as code, environment cleanup, environment promotion workflow, branch environment
+---
+
+# Skill — Environment Management
+
+## When this skill activates
+
+This skill activates when the user is designing or implementing environment management capabilities. This includes preview environments (per-PR), ephemeral environments, environment parity (dev/staging/prod consistency), configuration drift detection, automated environment provisioning, staging environment design, environment lifecycle management, environment-as-code, automatic cleanup, environment promotion workflows, and branch-based environments.
+
+## Mandatory actions when this skill is active
+
+### Before writing any code
+
+1. Inventory existing environments: count, purpose, cost, utilization, and configuration drift from production.
+2. Define environment types: production, staging, QA, developer sandbox, preview (per-PR), load testing. Establish parity requirements for each.
+3. Assess environment provisioning time: how long from request to usable environment. Target: under 10 minutes.
+4. Identify configuration drift: where do dev/staging/prod differ (versions, feature flags, resource sizes, network topology). Quantify drift percentage.
+5. Establish environment cleanup policies: when to delete ephemeral environments (after PR merge, after N days of inactivity).
+
+### During implementation
+
+- **Preview Environments (Per-PR):** Automatically create an isolated environment for each pull request. Include: full application stack, seeded test data, unique URL. Preview environment should be ready in under 10 minutes. Delete automatically when PR is merged or closed.
+- **Ephemeral Environments:** Short-lived environments created on-demand and destroyed after use. Use for: testing, demos, training, experiments. Include cost cap ($50-$200) and auto-deletion after 7 days of inactivity.
+- **Environment Parity:** Dev, staging, and prod should be as similar as possible. Use same: container images, Terraform modules, network topology, resource sizes (scale down in non-prod, but maintain same architecture). Differ only in: data (synthetic in non-prod), scale (fewer replicas), and external integrations (use mocks in non-prod).
+- **Environment as Code:** All environments defined via IaC (Terraform, CloudFormation, Pulumi). No manual changes in cloud console. Code should be versioned and reviewed via PRs. Use modules to ensure consistency across environments.
+- **Configuration Drift Detection:** Use Terraform plan, CloudFormation drift detection, or Config Sentinel. Run drift detection daily and alert on any manual changes. Automatically remediate drift by reapplying IaC.
+- **Environment Provisioning:** Self-service provisioning via CLI, API, or portal. Provisioning should: validate inputs, estimate cost, apply IaC, seed data, run smoke tests, return URL. Complete in under 10 minutes.
+- **Environment Lifecycle:** Define stages: provisioning → active → idle → scheduled for deletion → deleted. Idle environments (no activity for 3+ days) should be flagged for review. Auto-delete after 7 days idle (with 24-hour warning).
+- **Environment Promotion:** Promote changes from dev → staging → production. Use GitOps workflow: commit to environment-specific branch triggers deployment. Include smoke tests and rollback on failure.
+- **Staging Environment Design:** Staging should mirror production as closely as possible. Use 20-30% of production scale. Include: same services, same network topology, same feature flags, same monitoring. Differ only in: data volume and external integrations (use mocks or sandbox APIs).
+
+### After implementation
+
+- Verify preview environments are created automatically for each PR and deleted on merge.
+- Confirm ephemeral environments include cost caps and auto-deletion after 7 days.
+- Validate environment parity: dev/staging/prod use same IaC modules and container images.
+- Ensure configuration drift detection runs daily and alerts on manual changes.
+- Check that environment provisioning completes in under 10 minutes with cost estimates.
+
+## Self-check before task completion
+
+- [ ] Preview environments are created automatically per PR and deleted on merge.
+- [ ] Ephemeral environments include cost caps and auto-deletion after 7 days idle.
+- [ ] Environment parity is maintained: dev/staging/prod use same IaC modules.
+- [ ] Configuration drift detection runs daily and alerts on manual changes.
+- [ ] Environment provisioning is self-service and completes in under 10 minutes.
+- [ ] Environment lifecycle includes idle detection and auto-cleanup after 7 days.
+- [ ] Environment promotion uses GitOps workflow with smoke tests and rollback.
+- [ ] Staging environment mirrors production at 20-30% scale with same architecture.

package/.mindforge/skills/error-handling-architecture/SKILL.md

@@ -0,0 +1,118 @@
+---
+name: error-handling-architecture
+version: 1.0.0
+min_mindforge_version: 10.0.8
+status: stable
+triggers: error handling architecture, error hierarchy, error boundary, retry policy, dead letter queue, error reporting, sentry integration, error classification, graceful error, error propagation, error context, structured error
+---
+
+# Error Handling Architecture
+
+## When this skill activates
+
+This skill activates when designing error hierarchies, implementing error boundaries, configuring retry policies, setting up error reporting (Sentry/Datadog/Bugsnag), building dead letter queue processing, or establishing error handling patterns across a codebase. It applies to both frontend and backend error architecture.
+
+## Mandatory actions when this skill is active
+
+### Before
+
+1. Map the system boundaries where errors cross layers (UI → API → service → DB → external).
+2. Identify the error reporting tool in use or to be adopted (Sentry, Datadog, Bugsnag, custom).
+3. Categorize existing errors: which are retryable, which are permanent, which are user-facing.
+4. Determine SLA for error detection and alerting (time from error to notification).
+5. Review current error handling for anti-patterns (swallowed errors, generic catches, leaked internals).
+
+### During
+
+**Error Hierarchy Design:**
+```
+BaseError (abstract)
+├── ValidationError        — Invalid input (400). User can fix and retry.
+├── NotFoundError          — Resource doesn't exist (404). Client should not retry.
+├── AuthenticationError    — Identity unknown (401). Redirect to login.
+├── AuthorizationError     — Identity known, permission denied (403). Contact admin.
+├── ConflictError          — State conflict (409). Client should refresh and retry.
+├── RateLimitError         — Too many requests (429). Client must back off.
+├── ExternalServiceError   — Upstream dependency failed (502/503). Retry with backoff.
+└── InternalError          — Unexpected failure (500). Alert engineers.
+```
+- Every error class carries: `code` (machine-readable), `message` (human-readable), `context` (debugging data), `retryable` (boolean).
+- NEVER use generic `Error` or `Exception` for domain errors. Always use typed errors.
+- Error codes follow a namespace: `AUTH_001`, `PAYMENT_002`, `INVENTORY_003`.
+
+**Error Boundaries (Defense in Depth):**
+- **React UI**: `ErrorBoundary` components at route level and critical widget level. Show user-friendly fallback, report to Sentry.
+- **API Controllers**: Catch all errors at the handler level. Transform to appropriate HTTP status and structured response.
+- **Service Layer**: Catch errors from dependencies, add context, re-throw as domain errors.
+- **Infrastructure**: Global uncaught exception handler as last resort (log + alert + graceful shutdown).
+- Each boundary: catch, enrich with context, decide (retry/propagate/absorb), report.
+
+**Retry Policies:**
+- **Exponential backoff**: `delay = baseDelay * 2^attempt` (e.g., 100ms, 200ms, 400ms, 800ms).
+- **Jitter**: Add random variance to prevent thundering herd (`delay + random(0, delay * 0.1)`).
+- **Max retries**: 3 for fast operations, 5 for critical operations, 0 for non-idempotent writes.
+- **Idempotency keys**: Required for retrying write operations (prevent double-processing).
+- **Circuit breaker**: After N consecutive failures, stop trying for a cooldown period. Prevents cascade.
+- **Retry only retryable errors**: Network timeouts (yes), validation errors (no), 500s (maybe, with limit).
+
+**Dead Letter Queues (Async Error Handling):**
+- Messages that fail processing after max retries go to the DLQ.
+- DLQ messages must retain: original message, error details, attempt count, timestamp of each failure.
+- Alerting: notify when DLQ depth exceeds threshold (e.g., >10 messages in 5 minutes).
+- Manual replay: tooling to inspect DLQ messages and replay them after fix is deployed.
+- Poisoned messages: after replay still fails, move to permanent failure store with investigation ticket.
+- Never silently drop messages. Every message must reach success or explicit human acknowledgment.
+
+**Error Reporting (Sentry Integration Pattern):**
+- **Breadcrumbs**: Log navigation, API calls, user actions leading up to the error.
+- **Tags**: environment, service, user_id, transaction_id, feature_flag.
+- **Context**: request body (sanitized), response status, database query (no PII).
+- **Grouping**: Configure fingerprinting so related errors group together (not 1000 separate issues).
+- **Alerting**: New issues → Slack. Regression (resolved issue re-opens) → PagerDuty.
+- **Sampling**: 100% for errors, 10-20% for transactions/performance in production.
+- **PII scrubbing**: Strip emails, tokens, passwords before sending to error reporting.
+
+**Structured Error Responses (API):**
+```json
+{
+  "error": {
+    "code": "VALIDATION_001",
+    "message": "Email address is invalid",
+    "details": [
+      { "field": "email", "constraint": "Must be a valid email format" }
+    ],
+    "retryable": false,
+    "request_id": "req_abc123"
+  }
+}
+```
+- Always include `request_id` for correlation with server logs.
+- Never expose stack traces, internal paths, or database details in API responses.
+- Use `details` array for field-level validation errors.
+- Include `retryable` flag so clients can implement automatic retry logic.
+
+**Error Propagation Rules:**
+- Errors propagate UP (from infrastructure to service to controller to client).
+- At each layer: catch, add context specific to that layer, re-throw as appropriate type.
+- NEVER swallow errors silently (`catch (e) {}`). At minimum: log and re-throw.
+- Transform errors at boundaries (internal DB error becomes generic 500 to the client).
+- Preserve the original error as `cause` for debugging (Error Cause chain / `cause` property).
+
+### After
+
+1. Error hierarchy covers all known failure modes in the system.
+2. Every boundary has explicit error handling (no unhandled promise rejections, no bare throws).
+3. Retry policies are configured with backoff, jitter, and max attempts.
+4. Error reporting captures sufficient context for debugging without exposing PII.
+5. Alerting is configured for new errors, regressions, and DLQ threshold breaches.
+
+## Self-check before task completion
+
+- [ ] All errors use typed error classes from the hierarchy (no generic `Error` throws).
+- [ ] Error boundaries exist at every system layer (UI, API, service, infrastructure).
+- [ ] Retry policies use exponential backoff with jitter and respect idempotency.
+- [ ] Dead letter queues are configured for async processing with alerting and replay tooling.
+- [ ] Error reporting includes breadcrumbs, tags, and context (without PII).
+- [ ] API error responses are structured with code, message, retryable flag, and request_id.
+- [ ] No errors are silently swallowed anywhere in the codebase.
+- [ ] Error messages are actionable for the audience (user-friendly in UI, detailed in logs).

package/.mindforge/skills/estimation-techniques/SKILL.md

@@ -0,0 +1,113 @@
+---
+name: estimation-techniques
+version: 1.0.0
+min_mindforge_version: 10.1.0
+status: stable
+triggers: estimation technique, story points, planning poker, reference class forecasting, cone of uncertainty, velocity projection, t-shirt sizing, three-point estimate, estimation accuracy, estimation calibration, effort estimation, time estimation
+---
+
+# Estimation Techniques
+
+## When this skill activates
+
+This skill activates when the team needs to estimate effort, duration, or complexity
+of work items. It provides multiple estimation techniques, guidance on when to use each,
+and frameworks for improving estimation accuracy over time through calibration and
+reference class forecasting.
+
+## Mandatory actions when this skill is active
+
+### Before
+
+1. **Clarify what is being estimated** — Effort (person-days), duration (calendar time),
+   complexity (relative sizing), or cost (dollars)? These are different things.
+2. **Identify the audience** — Who needs this estimate and what decision will it inform?
+   (Sprint planning vs. budget approval vs. roadmap planning require different precision.)
+3. **Gather reference data** — Pull historical velocity, past estimates vs. actuals for
+   similar work, and team availability for the period.
+
+### During
+
+4. **Select appropriate technique based on context:**
+
+   - **Story Points (relative sizing):**
+     Compare work items to known reference stories. "Is this bigger or smaller than
+     the login feature we built last sprint?" Use Fibonacci sequence (1, 2, 3, 5, 8, 13)
+     to force recognition that large items have more uncertainty.
+     Best for: Sprint planning, backlog prioritization.
+
+   - **Planning Poker:**
+     Each estimator independently selects a card. Reveal simultaneously. If estimates
+     diverge by >2x, the highest and lowest explain their reasoning. Re-estimate.
+     Convergence indicates shared understanding. Divergence reveals hidden complexity
+     or misunderstanding. Best for: Team alignment on scope understanding.
+
+   - **Three-Point Estimate:**
+     Optimistic (O) + Most Likely (M) + Pessimistic (P).
+     Weighted average: (O + 4M + P) / 6.
+     Standard deviation: (P - O) / 6.
+     Present as range, not single number. Best for: Executive communication,
+     project planning with uncertainty bounds.
+
+   - **T-Shirt Sizing (S / M / L / XL):**
+     Rough relative sizing for large backlogs. Fast, low-ceremony. Good for roadmap-
+     level planning where precision is not needed. Convert to approximate days only
+     when pressed (S=1-2d, M=3-5d, L=1-2w, XL=3-4w — calibrate to your team).
+     Best for: Roadmap planning, early-stage estimation.
+
+   - **Reference Class Forecasting:**
+     Do NOT ask "how long do I think this will take?"
+     Instead ask "how long did SIMILAR work actually take in the past?"
+     Search for completed work of similar scope, technology, and team composition.
+     Use the actual duration distribution as the forecast basis.
+     Best for: Overcoming optimism bias, project-level estimates.
+
+5. **Apply the Cone of Uncertainty:**
+   - Initial concept: estimate accuracy is 4x (could be 4x longer or 0.25x shorter)
+   - After requirements: 2x range
+   - After detailed design: 1.5x range
+   - After implementation started: 1.25x range
+   - Communicate which stage you are at and the corresponding uncertainty range.
+   - Never present early-stage estimates as precise numbers.
+
+6. **Velocity-based projection:**
+   - Average velocity over last 3-5 sprints (discard outliers)
+   - Remaining story points / average velocity = estimated sprints remaining
+   - Present as range: (remaining / max velocity) to (remaining / min velocity)
+   - Account for known disruptions (holidays, planned absences, onboarding)
+
+7. **Common estimation pitfalls to avoid:**
+   - Anchoring — First number mentioned biases all subsequent estimates
+   - Planning fallacy — People consistently underestimate (use reference class data)
+   - Scope creep — Estimate what is defined NOW, flag unknowns separately
+   - Precision theater — "14.5 days" implies false precision; use ranges
+   - Forgetting overhead — Testing, code review, deployment, documentation are work too
+   - Hero assumptions — Estimate for average team member, not the fastest
+
+8. **Calibration practices:**
+   - Track estimate vs. actual for every completed item
+   - Review accuracy quarterly — are you consistently over or under?
+   - Adjust multiplier based on historical bias (if you are consistently 1.5x under,
+     multiply estimates by 1.5)
+   - Share calibration data with the team to build collective estimation skill
+
+### After
+
+9. **Record the estimate with context** — Document: what was estimated, technique used,
+   assumptions made, uncertainty range, and who participated.
+10. **Track actuals** — When work completes, record actual effort/duration alongside the
+    estimate for future calibration.
+11. **Retrospect on accuracy** — Periodically review estimation accuracy trends. Celebrate
+    improvement in calibration, investigate systematic biases.
+
+## Self-check before task completion
+
+- [ ] Estimation technique selected intentionally (not just defaulting to one)
+- [ ] Estimate presented as range, not single number
+- [ ] Cone of uncertainty stage identified and communicated
+- [ ] Reference class data consulted where available
+- [ ] Common pitfalls actively guarded against (especially anchoring and planning fallacy)
+- [ ] Assumptions and unknowns documented alongside the estimate
+- [ ] Overhead included (testing, review, deployment, documentation)
+- [ ] Historical calibration data referenced if available
+- [ ] Tracking mechanism in place to compare estimate vs. actual

package/.mindforge/skills/eval-harness/SKILL.md

@@ -0,0 +1,180 @@
+---
+name: eval-harness
+version: 1.0.0
+min_mindforge_version: 10.0.4
+status: stable
+triggers: eval, evaluation, grading, pass at k, rubric, regression eval, capability eval, model judge, deterministic grading, LLM-as-judge, eval score, eval-driven, benchmark eval
+---
+
+# Skill — Eval Harness (Systematic Evaluation Framework)
+
+## When this skill activates
+When measuring, scoring, or validating system outputs against defined criteria.
+Use for capability evaluation (can the system do X?), regression evaluation (does
+a change break existing behavior?), or comparative evaluation (is version A better
+than version B?). The eval harness ensures you define success BEFORE implementing,
+not after.
+
+Core principle: **Define-before-code** — write evaluation criteria before writing
+the implementation they measure.
+
+## Mandatory actions when this skill is active
+
+### Before evaluation begins
+
+1. **Define the eval type:**
+   - **Capability eval**: Can the system perform task X at acceptable quality?
+   - **Regression eval**: Does this change preserve existing behavior?
+   - **Comparative eval**: Is output A better than output B on criteria C?
+
+2. **Write the eval config BEFORE implementation:**
+   ```
+   .mindforge/evals/[eval-name]/
+   ├── config.json      # eval metadata, parameters, thresholds
+   ├── rubric.md        # human-readable success criteria
+   ├── test-cases.json  # input/expected-output pairs
+   └── results.jsonl    # append-only results log
+   ```
+
+3. **Define success criteria in config.json:**
+   ```json
+   {
+     "name": "eval-name",
+     "type": "capability" | "regression" | "comparative",
+     "version": "1.0.0",
+     "created": "ISO-8601",
+     "thresholds": {
+       "pass_at_1": 0.8,
+       "pass_at_5": 0.95,
+       "pass_at_10": 0.99
+     },
+     "grader": "code" | "model" | "human",
+     "model_judge_config": {
+       "model": "claude-sonnet",
+       "rubric_path": "./rubric.md",
+       "temperature": 0.0
+     },
+     "test_case_count": 0,
+     "tags": []
+   }
+   ```
+
+4. **Write the rubric (rubric.md) with explicit scoring:**
+   - Each criterion gets a 1-5 scale with concrete examples at each level
+   - Define what a "pass" means (minimum score per criterion)
+   - Define what a "fail" looks like with specific examples
+   - Include edge cases that should be tested
+
+### During evaluation
+
+**Three Grader Types:**
+
+**1. Code-Based (Deterministic):**
+- Use when outputs have objectively verifiable properties
+- Write assertion functions that return PASS/FAIL with evidence
+- Examples: output matches regex, JSON schema validates, function returns expected value
+- No ambiguity — the grader is a function, not a judgment call
+- Always prefer code-based grading when possible (fastest, most reliable)
+
+```typescript
+// Example code grader
+function grade(output: string, expected: TestCase): GradeResult {
+  const parsed = JSON.parse(output);
+  return {
+    pass: parsed.status === expected.status && parsed.count >= expected.minCount,
+    evidence: `status=${parsed.status}, count=${parsed.count}`,
+    criterion: "structural-correctness"
+  };
+}
+```
+
+**2. Model-Based (LLM-as-Judge):**
+- Use when outputs require semantic understanding (prose quality, code correctness, reasoning)
+- Always provide the rubric in the judge prompt — never rely on implicit standards
+- Use temperature 0.0 for judge calls (determinism)
+- Run judge 3x per item and take majority vote (reduces noise)
+- Log the judge's reasoning alongside the score
+
+```
+Judge prompt structure:
+1. Task description (what was the system asked to do?)
+2. Rubric (what does good look like? what does bad look like?)
+3. The output to grade
+4. Instruction: score 1-5 per criterion, explain each score, give overall PASS/FAIL
+```
+
+**3. Human-Based (Flag for Review):**
+- Use when stakes are too high for automated judgment
+- Generate a review queue with: input, output, rubric, suggested-score
+- Human confirms or overrides the suggested score
+- Track inter-rater reliability if multiple humans review
+
+**pass@k Metrics:**
+- Generate k independent outputs for each test case
+- **pass@1**: Fraction of test cases where the first output passes
+- **pass@5**: Fraction where at least 1 of 5 outputs passes
+- **pass@10**: Fraction where at least 1 of 10 outputs passes
+- Formula: pass@k = 1 - C(n-c, k) / C(n, k) where n=total, c=correct
+- Always report pass@1 (baseline) and at least one higher-k metric
+- Use pass@1 for production readiness, pass@k for capability ceiling
+
+**Result logging (results.jsonl):**
+```json
+{
+  "timestamp": "ISO-8601",
+  "test_case_id": "tc-001",
+  "input": "...",
+  "output": "...",
+  "grader": "code",
+  "scores": {"criterion_a": 4, "criterion_b": 5},
+  "pass": true,
+  "evidence": "...",
+  "latency_ms": 0,
+  "model_version": "...",
+  "run_id": "uuid"
+}
+```
+
+### After evaluation
+
+1. **Compute aggregate metrics:**
+   - Overall pass rate (pass@1, pass@5, pass@10)
+   - Per-criterion score distribution
+   - Failure mode clustering (what patterns cause failures?)
+   - Comparison to previous run (regression detection)
+
+2. **Regression detection logic:**
+   - If pass@1 drops > 5% from previous run: FLAG as regression
+   - If any previously-passing test case now fails: FLAG as regression
+   - If new failure modes appear that didn't exist before: FLAG as regression
+   - Regressions block shipping until investigated
+
+3. **Store results:**
+   - Append to results.jsonl (never overwrite)
+   - Update config.json with latest run metadata
+   - If regression detected: create `.mindforge/evals/[name]/REGRESSION.md`
+
+4. **Report format:**
+   ```
+   ## Eval Report: [eval-name]
+   - Type: capability | regression | comparative
+   - Run: [run-id] at [timestamp]
+   - Test cases: N total, P passed, F failed
+   - pass@1: X% | pass@5: Y% | pass@10: Z%
+   - Threshold: pass@1 >= T% → [MET / NOT MET]
+   - Regressions: [none | list]
+   - Top failure modes: [list with counts]
+   ```
+
+## Self-check before task completion
+
+Before marking a task done when this skill was active:
+
+- [ ] Did I define success criteria BEFORE writing implementation code?
+- [ ] Did I choose the appropriate grader type (code > model > human preference)?
+- [ ] Did I track pass@k metrics (at minimum pass@1)?
+- [ ] Did I run regression evals against previous results?
+- [ ] Are results stored in `.mindforge/evals/[name]/results.jsonl`?
+- [ ] If model-based grading: did I use temperature 0.0 and majority vote?
+- [ ] Did I report failure modes, not just pass rates?
+- [ ] Is the rubric explicit enough that another reviewer could grade independently?