mindforge-cc 10.0.2 → 10.7.0

package/.mindforge/skills/eval-harness/SKILL.md

@@ -0,0 +1,180 @@
+---
+name: eval-harness
+version: 1.0.0
+min_mindforge_version: 10.0.4
+status: stable
+triggers: eval, evaluation, grading, pass at k, rubric, regression eval, capability eval, model judge, deterministic grading, LLM-as-judge, eval score, eval-driven, benchmark eval
+---
+
+# Skill — Eval Harness (Systematic Evaluation Framework)
+
+## When this skill activates
+When measuring, scoring, or validating system outputs against defined criteria.
+Use for capability evaluation (can the system do X?), regression evaluation (does
+a change break existing behavior?), or comparative evaluation (is version A better
+than version B?). The eval harness ensures you define success BEFORE implementing,
+not after.
+
+Core principle: **Define-before-code** — write evaluation criteria before writing
+the implementation they measure.
+
+## Mandatory actions when this skill is active
+
+### Before evaluation begins
+
+1. **Define the eval type:**
+   - **Capability eval**: Can the system perform task X at acceptable quality?
+   - **Regression eval**: Does this change preserve existing behavior?
+   - **Comparative eval**: Is output A better than output B on criteria C?
+
+2. **Write the eval config BEFORE implementation:**
+   ```
+   .mindforge/evals/[eval-name]/
+   ├── config.json      # eval metadata, parameters, thresholds
+   ├── rubric.md        # human-readable success criteria
+   ├── test-cases.json  # input/expected-output pairs
+   └── results.jsonl    # append-only results log
+   ```
+
+3. **Define success criteria in config.json:**
+   ```json
+   {
+     "name": "eval-name",
+     "type": "capability" | "regression" | "comparative",
+     "version": "1.0.0",
+     "created": "ISO-8601",
+     "thresholds": {
+       "pass_at_1": 0.8,
+       "pass_at_5": 0.95,
+       "pass_at_10": 0.99
+     },
+     "grader": "code" | "model" | "human",
+     "model_judge_config": {
+       "model": "claude-sonnet",
+       "rubric_path": "./rubric.md",
+       "temperature": 0.0
+     },
+     "test_case_count": 0,
+     "tags": []
+   }
+   ```
+
+4. **Write the rubric (rubric.md) with explicit scoring:**
+   - Each criterion gets a 1-5 scale with concrete examples at each level
+   - Define what a "pass" means (minimum score per criterion)
+   - Define what a "fail" looks like with specific examples
+   - Include edge cases that should be tested
+
+### During evaluation
+
+**Three Grader Types:**
+
+**1. Code-Based (Deterministic):**
+- Use when outputs have objectively verifiable properties
+- Write assertion functions that return PASS/FAIL with evidence
+- Examples: output matches regex, JSON schema validates, function returns expected value
+- No ambiguity — the grader is a function, not a judgment call
+- Always prefer code-based grading when possible (fastest, most reliable)
+
+```typescript
+// Example code grader
+function grade(output: string, expected: TestCase): GradeResult {
+  const parsed = JSON.parse(output);
+  return {
+    pass: parsed.status === expected.status && parsed.count >= expected.minCount,
+    evidence: `status=${parsed.status}, count=${parsed.count}`,
+    criterion: "structural-correctness"
+  };
+}
+```
+
+**2. Model-Based (LLM-as-Judge):**
+- Use when outputs require semantic understanding (prose quality, code correctness, reasoning)
+- Always provide the rubric in the judge prompt — never rely on implicit standards
+- Use temperature 0.0 for judge calls (determinism)
+- Run judge 3x per item and take majority vote (reduces noise)
+- Log the judge's reasoning alongside the score
+
+```
+Judge prompt structure:
+1. Task description (what was the system asked to do?)
+2. Rubric (what does good look like? what does bad look like?)
+3. The output to grade
+4. Instruction: score 1-5 per criterion, explain each score, give overall PASS/FAIL
+```
+
+**3. Human-Based (Flag for Review):**
+- Use when stakes are too high for automated judgment
+- Generate a review queue with: input, output, rubric, suggested-score
+- Human confirms or overrides the suggested score
+- Track inter-rater reliability if multiple humans review
+
+**pass@k Metrics:**
+- Generate k independent outputs for each test case
+- **pass@1**: Fraction of test cases where the first output passes
+- **pass@5**: Fraction where at least 1 of 5 outputs passes
+- **pass@10**: Fraction where at least 1 of 10 outputs passes
+- Formula: pass@k = 1 - C(n-c, k) / C(n, k) where n=total, c=correct
+- Always report pass@1 (baseline) and at least one higher-k metric
+- Use pass@1 for production readiness, pass@k for capability ceiling
+
+**Result logging (results.jsonl):**
+```json
+{
+  "timestamp": "ISO-8601",
+  "test_case_id": "tc-001",
+  "input": "...",
+  "output": "...",
+  "grader": "code",
+  "scores": {"criterion_a": 4, "criterion_b": 5},
+  "pass": true,
+  "evidence": "...",
+  "latency_ms": 0,
+  "model_version": "...",
+  "run_id": "uuid"
+}
+```
+
+### After evaluation
+
+1. **Compute aggregate metrics:**
+   - Overall pass rate (pass@1, pass@5, pass@10)
+   - Per-criterion score distribution
+   - Failure mode clustering (what patterns cause failures?)
+   - Comparison to previous run (regression detection)
+
+2. **Regression detection logic:**
+   - If pass@1 drops > 5% from previous run: FLAG as regression
+   - If any previously-passing test case now fails: FLAG as regression
+   - If new failure modes appear that didn't exist before: FLAG as regression
+   - Regressions block shipping until investigated
+
+3. **Store results:**
+   - Append to results.jsonl (never overwrite)
+   - Update config.json with latest run metadata
+   - If regression detected: create `.mindforge/evals/[name]/REGRESSION.md`
+
+4. **Report format:**
+   ```
+   ## Eval Report: [eval-name]
+   - Type: capability | regression | comparative
+   - Run: [run-id] at [timestamp]
+   - Test cases: N total, P passed, F failed
+   - pass@1: X% | pass@5: Y% | pass@10: Z%
+   - Threshold: pass@1 >= T% → [MET / NOT MET]
+   - Regressions: [none | list]
+   - Top failure modes: [list with counts]
+   ```
+
+## Self-check before task completion
+
+Before marking a task done when this skill was active:
+
+- [ ] Did I define success criteria BEFORE writing implementation code?
+- [ ] Did I choose the appropriate grader type (code > model > human preference)?
+- [ ] Did I track pass@k metrics (at minimum pass@1)?
+- [ ] Did I run regression evals against previous results?
+- [ ] Are results stored in `.mindforge/evals/[name]/results.jsonl`?
+- [ ] If model-based grading: did I use temperature 0.0 and majority vote?
+- [ ] Did I report failure modes, not just pass rates?
+- [ ] Is the rubric explicit enough that another reviewer could grade independently?

package/.mindforge/skills/event-driven-architecture/SKILL.md

@@ -0,0 +1,162 @@
+---
+name: event-driven-architecture
+version: 1.0.0
+min_mindforge_version: 0.1.0
+status: stable
+triggers: event driven architecture, event bus, pub sub pattern, event schema design, ordering guarantee, exactly once delivery, dead letter topic, event sourcing integration, event catalog, event versioning strategy, event replay strategy, event consumer group
+---
+
+# Skill — Event-Driven Architecture
+
+## When this skill activates
+Any task involving event bus design, pub/sub patterns, message ordering,
+delivery guarantees, dead letter handling, or event schema evolution.
+
+## Mandatory actions when this skill is active
+
+### Before writing any code
+1. Classify event types (domain, integration, or command events).
+2. Define delivery guarantees required for each event stream.
+3. Design the event schema with forward/backward compatibility in mind.
+
+### During implementation
+- Make all consumers idempotent (safe to process same event multiple times).
+- Implement dead letter topic handling with alerting.
+- Use partition keys to maintain ordering where required.
+
+### After implementation
+- Register events in the event catalog with schema and owner.
+- Add consumer lag monitoring.
+- Document retry and failure handling in ARCHITECTURE.md.
+
+## Event Types
+
+### Domain Events
+- Facts about what happened in a bounded context.
+- Named in past tense: `OrderPlaced`, `PaymentProcessed`, `UserRegistered`.
+- Owned by the producing domain — consumers must adapt.
+- Immutable once published.
+
+### Integration Events
+- Cross-boundary communication between services.
+- May be transformed from domain events (different schema, less detail).
+- Published on shared event bus (Kafka, SNS, EventBridge).
+
+### Command Events
+- Request for action (not a fact).
+- Named as imperative: `ProcessPayment`, `SendNotification`.
+- Exactly one consumer expected to handle.
+- Requires acknowledgment/response.
+
+## Delivery Guarantees
+
+### At-Most-Once
+- Fire and forget. No retries.
+- Use for: metrics, analytics, non-critical notifications.
+- Risk: message loss on failure.
+
+### At-Least-Once (Recommended Default)
+- Retry until acknowledged.
+- Consumers MUST be idempotent.
+- Use for: most business events.
+- Risk: duplicate processing (mitigated by idempotency).
+
+### Exactly-Once (Expensive)
+- Requires transactional outbox + deduplication.
+- Use for: financial transactions, inventory changes.
+- Implementation: idempotency key + processed event log.
+
+## Ordering Guarantees
+
+### Per-Partition Ordering
+- Events with the same partition key are ordered.
+- Partition key = entity ID (e.g., order_id, user_id).
+- Different entities may be processed out of order (acceptable).
+
+### Global Ordering
+- Extremely expensive — single partition = no parallelism.
+- Almost never needed — design around per-entity ordering instead.
+
+### Kafka Partition Key Design
+```
+topic: order-events
+partition_key: order_id
+result: all events for order-123 arrive in sequence
+```
+
+## Schema Evolution
+
+### Compatibility Modes (Avro/Protobuf)
+- **Backward compatible**: new schema can read old data (add optional fields).
+- **Forward compatible**: old schema can read new data (ignore unknown fields).
+- **Full compatible**: both directions (safest, most restrictive).
+
+### Rules for Safe Evolution
+- Adding optional fields: always safe.
+- Removing fields: only if no consumers depend on them.
+- Renaming fields: treat as remove + add (breaking).
+- Changing field types: always breaking.
+
+### Schema Registry
+- Central registry of all event schemas with version history.
+- Validates compatibility before allowing schema updates.
+- Consumers reference schema by ID (embedded in message header).
+
+## Consumer Groups
+
+### Competing Consumers (Scaling Pattern)
+- Multiple instances in same group share the load.
+- Each message processed by exactly one instance.
+- Use for: order processing, notification sending.
+- Scale by adding more consumers (up to partition count).
+
+### Broadcasting (Fan-Out Pattern)
+- Each consumer group gets every message.
+- Use for: audit logging, cache invalidation, analytics.
+- Different groups process independently at their own pace.
+
+## Dead Letter Topics (DLT)
+
+### Flow
+```
+message → consumer → FAIL → retry (3x with backoff) → FAIL → DLT → alert
+```
+
+### Requirements
+- Every consumer MUST have a DLT configured.
+- DLT messages retain full context (original message + error + attempt count).
+- Alert on first DLT message (don't silently accumulate).
+- Manual resolution workflow: inspect → fix → replay or discard.
+
+### Retry Strategy
+- Attempt 1: immediate.
+- Attempt 2: 1 second delay.
+- Attempt 3: 10 second delay.
+- After 3 failures: route to DLT.
+
+## Event Catalog
+
+Every event in the system must be registered:
+
+| Field | Description |
+|-------|-------------|
+| Event name | `OrderPlaced` |
+| Schema version | `v3` |
+| Owner (team) | Order Service team |
+| Producers | order-service |
+| Consumers | notification-svc, analytics-svc, fulfillment-svc |
+| Partition key | order_id |
+| Delivery guarantee | at-least-once |
+| Retention | 7 days |
+
+## Self-check before task completion
+
+Before marking a task done when this skill was active:
+
+- [ ] Did I read the full SKILL.md before starting? (Not just the triggers)
+- [ ] Are all consumers idempotent?
+- [ ] Is ordering guaranteed per entity via partition keys?
+- [ ] Is dead letter topic configured with alerting?
+- [ ] Are event schemas registered in the catalog?
+- [ ] Is schema evolution backward-compatible?
+- [ ] Are consumer groups configured correctly (competing vs broadcasting)?

package/.mindforge/skills/experiment-design/SKILL.md

@@ -0,0 +1,139 @@
+---
+name: experiment-design
+version: 1.0.0
+min_mindforge_version: 10.0.4
+status: stable
+triggers: experiment design, A/B testing architecture, statistical significance, sample size calculator, guardrail metric, experiment lifecycle, hypothesis testing, control variant, experiment analysis, metric sensitivity, experiment duration, randomization unit
+---
+
+# Skill — Experiment Design (Rigorous A/B Testing Architecture)
+
+## When this skill activates
+When designing, planning, or analyzing A/B tests, multivariate experiments, or
+any controlled experiment that requires statistical rigor. Use for feature rollout
+decisions, conversion optimization, pricing tests, or any scenario where you need
+to measure the causal impact of a change.
+
+Core principle: **Hypothesis-first** — never launch an experiment without a written
+hypothesis that specifies the expected effect direction, magnitude, and mechanism.
+
+## Mandatory actions when this skill is active
+
+### Before experiment begins
+
+1. **Write the hypothesis in structured format:**
+   ```
+   If we [change X],
+   then [metric Y] will [improve/degrade] by [Z amount]
+   because [causal mechanism].
+   ```
+   - The hypothesis must be falsifiable
+   - The expected effect size must be realistic (based on prior data or industry benchmarks)
+   - The causal mechanism must be articulated (not just "it will be better")
+
+2. **Calculate sample size:**
+   ```
+   Inputs:
+   - Baseline conversion rate (current metric value)
+   - Minimum Detectable Effect (MDE): smallest improvement worth detecting
+   - Statistical significance level (alpha): typically 0.05
+   - Statistical power (1-beta): typically 0.80
+   - Number of variants (control + treatments)
+
+   Output:
+   - Required sample size per variant
+   - Estimated duration = required_N / daily_traffic_per_variant
+   ```
+
+   Rules:
+   - MDE should be the smallest PRACTICALLY significant effect (not just statistically significant)
+   - If duration > 8 weeks: increase MDE or find higher-traffic surface
+   - Never compromise on power — underpowered experiments waste everyone's time
+
+3. **Define guardrail metrics:**
+   ```json
+   {
+     "primary_metric": "conversion_rate",
+     "secondary_metrics": ["revenue_per_user", "engagement_time"],
+     "guardrail_metrics": [
+       {"name": "page_load_time_p95", "threshold": "+200ms", "action": "stop"},
+       {"name": "error_rate", "threshold": "+0.5%", "action": "stop"},
+       {"name": "revenue_per_session", "threshold": "-2%", "action": "alert"}
+     ]
+   }
+   ```
+   - Guardrails are metrics that MUST NOT degrade beyond threshold
+   - Violation of a guardrail = experiment stopped regardless of primary metric
+   - Always include: performance, error rate, and revenue as guardrails
+
+4. **Choose randomization unit:**
+   - **User-level**: Default for most experiments (consistent experience across sessions)
+   - **Session-level**: For UI experiments where cross-session contamination is acceptable
+   - **Page-level**: Only for layout experiments with no carryover effects
+   - **Device-level**: When logged-out users are significant traffic
+   - Rule: randomization unit >= analysis unit (never analyze at user level if randomized at page level)
+
+### During experiment
+
+1. **Minimum duration rules:**
+   - Run for at least 1 full business cycle (typically 7 days minimum)
+   - Recommended: 2 full weeks to capture weekday/weekend variation
+   - NEVER stop early because results "look significant" (peeking problem)
+   - If using sequential testing: define stopping rules BEFORE launch
+
+2. **Monitoring protocol:**
+   - Check guardrail metrics daily
+   - Do NOT check primary metric significance until planned end date
+   - If peeking is necessary: use group sequential methods with alpha spending
+   - Log any system issues that may contaminate results (outages, bugs, other launches)
+
+3. **Sample Ratio Mismatch (SRM) check:**
+   - Verify variant assignment is balanced (chi-square test, p < 0.001 = SRM)
+   - SRM invalidates the experiment — do not trust results
+   - Common causes: bot filtering, redirect failures, bucketing bugs
+
+### After experiment (analysis)
+
+1. **Statistical analysis checklist:**
+   - [ ] Confirm no SRM
+   - [ ] Check primary metric: p-value < 0.05 AND confidence interval excludes 0
+   - [ ] Check practical significance: is the effect size large enough to matter?
+   - [ ] Check guardrail metrics: no violations
+   - [ ] Check segment consistency: does the effect hold across key segments?
+   - [ ] Check novelty/primacy effects: is the effect stable over time?
+
+2. **Decision framework:**
+   ```
+   IF p < 0.05 AND practical significance AND no guardrail violations:
+     → SHIP (roll out to 100%)
+   IF p < 0.05 BUT guardrail violation:
+     → ITERATE (fix guardrail issue, re-run)
+   IF p >= 0.05 AND confidence interval includes meaningful effects:
+     → EXTEND (underpowered, run longer or increase traffic)
+   IF p >= 0.05 AND confidence interval excludes meaningful effects:
+     → KILL (the change doesn't work, move on)
+   ```
+
+3. **Document the result:**
+   ```markdown
+   ## Experiment Result: [name]
+   - Hypothesis: [statement]
+   - Duration: [days] | Sample: [N per variant]
+   - Primary metric: [baseline] → [variant] ([+/-X%], p=[value])
+   - Guardrails: [all clear / violations]
+   - Decision: SHIP / ITERATE / EXTEND / KILL
+   - Learning: [what did we learn about user behavior?]
+   ```
+
+## Self-check before task completion
+
+Before marking a task done when this skill was active:
+
+- [ ] Did I write a structured hypothesis with expected effect size and mechanism?
+- [ ] Did I calculate required sample size based on MDE and baseline?
+- [ ] Did I define guardrail metrics with explicit thresholds?
+- [ ] Did I choose an appropriate randomization unit?
+- [ ] Did I set minimum duration (>= 1 business cycle)?
+- [ ] Did I plan for the peeking problem (no early stopping without sequential testing)?
+- [ ] Did I document the decision framework (ship/iterate/extend/kill)?
+- [ ] Is the experiment design reproducible by another engineer?

package/.mindforge/skills/experiment-platform/SKILL.md

@@ -0,0 +1,43 @@
+---
+name: experiment-platform
+version: 1.0.0
+min_mindforge_version: 10.6.0
+status: stable
+triggers: experiment platform design, experimentation infrastructure, statistical rigor experiment, guardrail metric design, experiment velocity, feature flag experiment, experiment analysis automation, sample size calculation, multi-variant testing, experiment platform lifecycle, experiment review process, sequential testing
+compose: experiment-design
+---
+
+# Skill — Experiment Platform
+
+## When this skill activates
+This skill activates when building experimentation infrastructure, implementing statistical testing frameworks, or designing A/B testing platforms. Use when organizations need to scale testing velocity while maintaining statistical rigor.
+
+## Mandatory actions when this skill is active
+
+### Before writing any code
+1. Define experiment framework components: randomization service, exposure logging, metric computation pipeline, and statistical analysis engine
+2. Establish statistical rigor standards: minimum sample size, power (80%+), significance level (5%), minimum detectable effect, and multiple comparison corrections
+3. Design guardrail metrics framework: business health (revenue, retention), user experience (latency, errors), and ecosystem health (partner impact)
+4. Plan experiment lifecycle states: draft, review, running, paused, completed, archived with transition criteria and approval gates
+
+### During implementation
+- Implement consistent randomization using stable hashing (user_id + experiment_id) ensuring users see same variant across sessions
+- Build exposure logging capturing: timestamp, user_id, experiment_id, variant, context for accurate sample size and covariate adjustment
+- Create metric computation pipeline with: numerator/denominator structure, winsorization for outliers, delta method for ratios, bootstrap for confidence intervals
+- Design sequential testing capability for early stopping: alpha spending functions, futility boundaries, and minimum runtime requirements
+- Implement stratified analysis for heterogeneous treatment effects: by platform, user segment, geography with interaction effect testing
+- Build automated guardrail checks: alert on significant negative movement in critical metrics with experiment auto-pause capability
+- Create experiment metadata repository: hypothesis, success criteria, related experiments, learnings, and decision outcome for institutional knowledge
+
+### After implementation
+- Generate automated experiment scorecards: primary metric movement, guardrail status, statistical significance, practical significance, recommendation
+- Build experiment catalog with search and discovery: hypothesis library, metric glossary, analysis templates, and historical results
+- Create experimentation health dashboard: velocity (experiments/week), quality (statistical power distribution), impact (significant wins), and coverage (features tested)
+- Document statistical methodology: test selection, variance reduction techniques, multiple comparison approach, and sequential testing procedures
+
+## Self-check before task completion
+- [ ] Randomization service ensures stable assignment and balanced allocation across variants
+- [ ] Exposure logging captures all necessary context for accurate analysis and debugging
+- [ ] Statistical analysis engine implements proper corrections for multiple comparisons and peeking
+- [ ] Guardrail metrics monitored automatically with alerting and experiment pause capability
+- [ ] Experiment lifecycle enforces minimum runtime and sample size before declaring results

package/.mindforge/skills/feature-engineering/SKILL.md

@@ -0,0 +1,42 @@
+---
+name: feature-engineering
+version: 1.0.0
+min_mindforge_version: 10.6.0
+status: stable
+triggers: ML feature engineering workflow, feature selection method, feature transformation, feature importance analysis, automated feature discovery, feature scaling normalization, feature interaction, temporal feature extraction, text feature engineering, categorical encoding strategy, feature validation, domain feature creation
+---
+
+# Skill — Feature Engineering
+
+## When this skill activates
+This skill activates when building ML pipelines that require feature creation, transformation, or selection. Use when designing feature stores, implementing automated feature discovery, or optimizing model input representation.
+
+## Mandatory actions when this skill is active
+
+### Before writing any code
+1. Conduct exploratory data analysis to understand feature distributions, missing patterns, correlations, and domain-specific relationships
+2. Define feature engineering strategy: target encoding risks, temporal leakage prevention, train-test split boundaries, and cross-validation approach
+3. Document business logic for derived features with domain expert validation and interpretability requirements
+4. Establish feature quality metrics: null rates, cardinality, stability over time, and correlation with target variable
+
+### During implementation
+- Implement feature transformations within sklearn Pipelines or similar frameworks to prevent train-test leakage
+- Use robust scaling methods appropriate to distribution (StandardScaler for normal, RobustScaler for outliers, quantile for non-parametric)
+- Create temporal features with proper lag handling: rolling windows, exponential smoothing, seasonal decomposition, time-since-event
+- Encode categorical variables with strategy matching cardinality (one-hot <10 categories, target encoding >50, embeddings for high-cardinality)
+- Generate interaction features guided by domain knowledge and feature importance: polynomial, ratio, difference, product features
+- Handle missing values explicitly with strategy documented: imputation (mean/median/mode), indicator variables, or model-based imputation
+- Validate feature importance using multiple methods: permutation importance, SHAP values, and univariate tests to identify top contributors
+
+### After implementation
+- Create feature documentation with schema definitions, transformation logic, expected ranges, and update frequency
+- Build feature monitoring dashboards tracking distribution drift, missing rate changes, and correlation stability over time
+- Generate feature store integration with versioning, metadata tracking, and point-in-time correctness for temporal joins
+- Validate feature pipeline performance: transformation latency, memory usage, and batch vs online serving consistency
+
+## Self-check before task completion
+- [ ] All features are computed within transformation pipelines to prevent train-test leakage
+- [ ] Feature importance analysis identifies top 20 contributors with interpretable business meaning
+- [ ] Temporal features respect time boundaries and use only historically available information
+- [ ] Feature documentation includes transformation logic, expected distributions, and monitoring thresholds
+- [ ] Feature validation tests confirm stability across different time periods and data segments