mindforge-cc 10.0.3 → 11.0.0

package/.mindforge/skills/a11y-testing/SKILL.md

@@ -0,0 +1,143 @@
+---
+name: a11y-testing
+version: 1.0.0
+min_mindforge_version: 0.3.0
+status: stable
+triggers: a11y testing, axe-core, automated accessibility, screen reader testing, keyboard navigation audit, WCAG compliance test, aria validation, focus management test, color contrast check, accessibility CI, accessibility report, assistive technology
+compose: accessibility
+---
+
+# Skill — Accessibility Testing
+
+## When this skill activates
+Any task involving accessibility testing, WCAG compliance, screen reader validation,
+keyboard navigation audits, or automated a11y CI pipelines.
+
+## Mandatory actions when this skill is active
+
+### Before testing accessibility
+1. Identify the target WCAG conformance level (A, AA, or AAA).
+2. Determine which automated tools are available in the project.
+3. Plan manual testing scenarios for what automation cannot catch.
+
+### Automated testing (~30% of issues)
+
+**Unit level (jest-axe):**
+```javascript
+import { axe, toHaveNoViolations } from 'jest-axe';
+expect.extend(toHaveNoViolations);
+
+it('has no accessibility violations', async () => {
+  const { container } = render(<Component />);
+  const results = await axe(container);
+  expect(results).toHaveNoViolations();
+});
+```
+
+**Integration level (Playwright + axe):**
+```javascript
+import AxeBuilder from '@axe-core/playwright';
+
+test('page has no a11y violations', async ({ page }) => {
+  await page.goto('/dashboard');
+  const results = await new AxeBuilder({ page }).analyze();
+  expect(results.violations).toEqual([]);
+});
+```
+
+**CI pipeline:**
+- Run axe-core on every PR against all critical routes.
+- Fail the build on any "critical" or "serious" violations.
+- Report "moderate" violations as warnings (fix in next sprint).
+- Track violation count over time — must trend downward.
+
+### Manual testing checklist
+
+**Keyboard navigation:**
+- [ ] Tab through ALL interactive elements in logical order.
+- [ ] Shift+Tab moves backwards correctly.
+- [ ] Enter/Space activates buttons and links.
+- [ ] Escape closes modals, dropdowns, popovers.
+- [ ] Arrow keys navigate within composite widgets (tabs, menus, grids).
+- [ ] No keyboard traps (can always Tab out, except modals).
+- [ ] Focus indicator is clearly visible on all elements.
+
+**Screen reader testing:**
+- [ ] VoiceOver (macOS/iOS) — full user flow.
+- [ ] NVDA (Windows) — full user flow.
+- [ ] All images have meaningful alt text (or alt="" for decorative).
+- [ ] Form inputs have associated labels.
+- [ ] Dynamic content changes are announced (aria-live regions).
+- [ ] Headings form a logical hierarchy (h1 > h2 > h3, no skips).
+
+**Visual testing:**
+- [ ] Zoom to 200% — no horizontal scroll, no overlapping content.
+- [ ] Zoom to 400% — content still readable and usable.
+- [ ] High contrast mode — all content visible.
+- [ ] Reduced motion — animations respect prefers-reduced-motion.
+
+### WCAG conformance levels
+
+**Level A (minimum, always required):**
+- All non-text content has text alternative.
+- Content is navigable by keyboard.
+- No content causes seizures.
+
+**Level AA (target for most applications):**
+- Color contrast ratio 4.5:1 for normal text, 3:1 for large text.
+- Text can be resized to 200% without loss of content.
+- Focus order is meaningful and logical.
+- Error messages identify the field and suggest correction.
+
+**Level AAA (specialized — not typically a blanket requirement):**
+- Color contrast ratio 7:1 for normal text.
+- Sign language interpretation for media.
+- Reading level accommodations.
+
+### Focus management
+
+**Modal dialogs:**
+- Move focus into the modal when opened.
+- Trap focus within the modal (Tab cycles inside).
+- Return focus to the trigger element when closed.
+
+**Route changes (SPA):**
+- Move focus to the main content heading on navigation.
+- Announce the new page to screen readers (aria-live or document.title).
+
+**Dynamic content:**
+- New content added below the current focus: no announcement needed.
+- New content that requires attention: use aria-live="polite".
+- Urgent alerts: use aria-live="assertive" (sparingly).
+
+**Skip links:**
+- First focusable element should be "Skip to main content."
+- Links to bypass repetitive navigation blocks.
+
+### Color contrast
+
+**Tools:**
+- Browser DevTools (Accessibility panel shows contrast ratios).
+- axe-core catches contrast violations automatically.
+- Contrast checker plugins for design tools (Figma, Sketch).
+
+**Ratios:**
+- Normal text (< 18px or < 14px bold): minimum 4.5:1.
+- Large text (>= 18px or >= 14px bold): minimum 3:1.
+- UI components and graphical objects: minimum 3:1.
+- Never convey information by color alone (add icons, patterns, text).
+
+### Reporting format
+
+When reporting accessibility issues, include:
+1. **What** — the specific WCAG criterion violated.
+2. **Where** — page URL and element selector/description.
+3. **Impact** — who is affected and how severely.
+4. **Fix** — specific remediation recommendation.
+5. **Priority** — critical (blocks usage) / serious (difficult) / moderate (inconvenient).
+
+## Self-check before task completion
+- [ ] Did I follow the mandatory actions for this skill?
+- [ ] Did I apply the patterns appropriate to the context?
+- [ ] Did I verify the implementation meets the criteria above?
+- [ ] Did I document decisions and trade-offs made?

package/.mindforge/skills/agent-evaluation-framework/SKILL.md

@@ -0,0 +1,227 @@
+---
+name: agent-evaluation-framework
+version: 1.0.0
+min_mindforge_version: 10.0.4
+status: stable
+triggers: agent evaluation, task completion rate, agent benchmark, reasoning quality, tool selection accuracy, agent cost efficiency, end-to-end agent test, agent regression, agent quality score, agent performance metric, evaluation harness design, agent grading
+compose: eval-harness
+---
+
+# Skill — Agent Evaluation Framework (End-to-End Agent Performance Measurement)
+
+## When this skill activates
+When measuring agent performance, designing agent benchmarks, tracking quality
+regressions, or comparing agent configurations. Use for any scenario where you
+need to answer: "Is this agent good enough?" or "Did this change make the agent
+better or worse?"
+
+Core principle: **Multi-dimensional quality** — agent quality is not a single number.
+A fast agent that's wrong is worse than a slow agent that's right. A cheap agent
+that hallucinates is worse than an expensive agent that's accurate. Measure ALL
+dimensions that matter.
+
+## Mandatory actions when this skill is active
+
+### Metric Taxonomy
+
+1. **Core agent metrics (measure ALL of these):**
+   ```
+   Correctness metrics:
+   - Task completion rate: % of tasks completed successfully (end-to-end)
+   - First-attempt success rate: % completed without retry or correction
+   - Factual accuracy: % of claims that are verifiable and correct
+   - Instruction adherence: % of explicit instructions followed correctly
+
+   Efficiency metrics:
+   - Cost per task: total API spend / successful completions
+   - Tokens per task: input + output tokens consumed
+   - Time per task: wall-clock time from task start to completion
+   - Tool calls per task: number of tool invocations (fewer = more efficient)
+
+   Quality metrics:
+   - Reasoning quality score: rubric-based assessment of reasoning chain
+   - Tool selection accuracy: % of tool calls that were appropriate
+   - Output quality score: rubric-based assessment of final output
+   - Hallucination rate: % of outputs containing ungrounded claims
+
+   Safety metrics:
+   - Harmful output rate: % of outputs flagged by safety classifiers
+   - Permission violation rate: % of actions exceeding authorized scope
+   - Information leakage rate: % of outputs exposing sensitive data
+   ```
+
+2. **Composite quality score:**
+   ```
+   Agent Quality Score = weighted combination:
+   - Correctness (40%): task_completion * 0.25 + first_attempt * 0.15
+   - Quality (30%): reasoning_quality * 0.15 + output_quality * 0.15
+   - Efficiency (20%): normalized(1/cost) * 0.10 + normalized(1/time) * 0.10
+   - Safety (10%): (1 - harmful_rate) * 0.05 + (1 - violation_rate) * 0.05
+
+   Weights are defaults — adjust per use case (safety-critical → increase safety weight)
+   ```
+
+### Benchmark Design
+
+3. **Evaluation dataset structure:**
+   ```
+   .mindforge/evals/agent-benchmark/
+   ├── config.json           # benchmark metadata and thresholds
+   ├── tasks/
+   │   ├── easy/             # baseline tasks (should be ~100% success)
+   │   │   ├── task-001.json
+   │   │   └── task-002.json
+   │   ├── medium/           # standard tasks (target: 80%+ success)
+   │   │   ├── task-010.json
+   │   │   └── task-011.json
+   │   └── hard/             # stretch tasks (target: 50%+ success)
+   │       ├── task-020.json
+   │       └── task-021.json
+   ├── rubrics/
+   │   ├── correctness.md    # how to grade correctness
+   │   ├── reasoning.md      # how to grade reasoning quality
+   │   └── output.md         # how to grade output quality
+   └── results/
+       └── results.jsonl     # append-only results log
+   ```
+
+4. **Task definition format:**
+   ```json
+   {
+     "task_id": "task-001",
+     "difficulty": "easy",
+     "category": "code-generation",
+     "description": "Write a function that reverses a string",
+     "input": "Create a TypeScript function reverseString(s: string): string",
+     "expected_behavior": [
+       "Returns reversed string",
+       "Handles empty string",
+       "Handles unicode correctly"
+     ],
+     "verification": {
+       "type": "code",
+       "test_cases": [
+         {"input": "hello", "expected": "olleh"},
+         {"input": "", "expected": ""},
+         {"input": "abc", "expected": "cba"}
+       ]
+     },
+     "metadata": {
+       "tools_available": ["Read", "Write", "Bash"],
+       "time_limit_seconds": 120,
+       "cost_limit_usd": 0.50
+     }
+   }
+   ```
+
+   Rules:
+   - Minimum 30 tasks per benchmark (10 easy, 15 medium, 5 hard)
+   - Tasks must be representative of real usage patterns
+   - Include both deterministic tasks (one right answer) and generative tasks (rubric-graded)
+   - Each task has explicit success criteria (not vague "good output")
+   - Stratify by difficulty to detect capability thresholds
+
+### Running Benchmarks
+
+5. **Execution protocol:**
+   ```
+   For each task in benchmark:
+   1. Initialize fresh agent context (no contamination between tasks)
+   2. Provide task input + available tools
+   3. Record: start_time, all tool calls, all outputs, end_time
+   4. Grade output against verification criteria
+   5. Log full result to results.jsonl
+
+   Run N times per task (N >= 3) to measure variance:
+   - Report mean and standard deviation per metric
+   - Flag high-variance tasks (inconsistent agent behavior)
+   - Use same random seed where possible for reproducibility
+   ```
+
+6. **Result logging:**
+   ```json
+   {
+     "run_id": "uuid",
+     "timestamp": "ISO-8601",
+     "task_id": "task-001",
+     "agent_config": {"model": "claude-sonnet", "temperature": 0.0},
+     "metrics": {
+       "completed": true,
+       "first_attempt": true,
+       "time_seconds": 15.3,
+       "cost_usd": 0.012,
+       "tokens_used": {"input": 1200, "output": 450},
+       "tool_calls": 3,
+       "reasoning_quality": 4,
+       "output_quality": 5
+     },
+     "grading": {
+       "method": "code",
+       "pass": true,
+       "evidence": "All 3 test cases passed"
+     }
+   }
+   ```
+
+### Regression Detection
+
+7. **Regression detection algorithm:**
+   ```
+   Compare current run vs baseline:
+
+   RED (regression detected — blocks deployment):
+   - Task completion rate drops > 5%
+   - Any previously-passing easy task now fails
+   - Cost per task increases > 50%
+   - Safety metric degrades at all
+
+   YELLOW (warning — investigate before deploying):
+   - Task completion rate drops 2-5%
+   - Medium/hard task pass rate drops > 10%
+   - Time per task increases > 30%
+   - New failure modes appear
+
+   GREEN (no regression):
+   - All metrics within 2% of baseline
+   - No new failure modes
+   - Cost/time stable or improved
+   ```
+
+   Rules:
+   - ALWAYS compare to a pinned baseline (not just previous run)
+   - Run regression suite before any agent config change ships
+   - Regression in EASY tasks is more alarming than regression in HARD tasks
+   - Store baseline with agent version (update baseline when intentionally accepting changes)
+
+### Cost Efficiency Analysis
+
+8. **Quality-per-dollar assessment:**
+   ```
+   Cost Efficiency Ratio = quality_score / cost_per_task
+
+   Comparison framework:
+   - Agent A: quality=0.92, cost=$0.05/task → efficiency=18.4
+   - Agent B: quality=0.88, cost=$0.01/task → efficiency=88.0
+
+   Decision: Agent B is 4.8x more cost-efficient.
+   Choose A only if the 4% quality gap causes real user-visible failures.
+   ```
+
+   Rules:
+   - A cheaper model that achieves 95% of the quality at 20% of the cost is usually better
+   - Factor in retry cost (low first-attempt rate = hidden cost multiplier)
+   - Include tool call costs in total cost (API calls, compute)
+   - Report cost efficiency alongside raw quality (both matter)
+
+## Self-check before task completion
+
+Before marking a task done when this skill was active:
+
+- [ ] Did I define metrics across all four dimensions (correctness, quality, efficiency, safety)?
+- [ ] Is the benchmark stratified by difficulty (easy/medium/hard)?
+- [ ] Did I run multiple times (N >= 3) to measure variance?
+- [ ] Is there a pinned baseline for regression detection?
+- [ ] Are regression thresholds defined (RED/YELLOW/GREEN)?
+- [ ] Did I report cost efficiency (quality/cost ratio), not just raw quality?
+- [ ] Are easy-task failures treated as more alarming than hard-task failures?
+- [ ] Are results appended to results.jsonl (never overwritten)?

package/.mindforge/skills/agent-memory-design/SKILL.md

@@ -0,0 +1,199 @@
+---
+name: agent-memory-design
+version: 1.0.0
+min_mindforge_version: 10.0.4
+status: stable
+triggers: agent memory design, short-term memory, long-term memory, memory retrieval, memory consolidation, memory decay, episodic memory, semantic memory, memory architecture, working memory, memory indexing, knowledge persistence
+---
+
+# Skill — Agent Memory Design (Multi-Layer Knowledge Persistence)
+
+## When this skill activates
+When designing memory systems for AI agents, implementing context persistence across
+sessions, building knowledge retrieval mechanisms, or architecting memory consolidation
+pipelines. Use for any agent that needs to remember information beyond a single context
+window.
+
+Core principle: **Memory is retrieval, not storage** — the value of a memory system
+is measured by: can the agent find the RIGHT information at the RIGHT time? Storage
+without retrieval is a graveyard.
+
+## Mandatory actions when this skill is active
+
+### Memory Layer Architecture
+
+1. **Four-tier memory model:**
+   ```
+   Layer 1 — Working Memory (context window)
+   - Scope: current conversation turn
+   - Capacity: model context limit (8K-200K tokens)
+   - Persistence: none (gone when context resets)
+   - Access: immediate (already in context)
+   - Priority: highest — this is what the agent "sees"
+
+   Layer 2 — Short-Term Memory (session scratchpad)
+   - Scope: current session/task
+   - Capacity: 10-50 key facts
+   - Persistence: session duration
+   - Access: explicit retrieval (agent requests it)
+   - Format: JSONL scratchpad file
+   - Use for: intermediate results, task progress, recent user preferences
+
+   Layer 3 — Medium-Term Memory (project knowledge)
+   - Scope: current project/workspace
+   - Capacity: hundreds of entries
+   - Persistence: project lifetime
+   - Access: semantic search + key lookup
+   - Format: structured markdown files, JSON indexes
+   - Use for: architectural decisions, user patterns, project conventions
+
+   Layer 4 — Long-Term Memory (permanent knowledge)
+   - Scope: cross-project, cross-session
+   - Capacity: unbounded
+   - Persistence: permanent (with decay)
+   - Access: semantic similarity search + knowledge graph traversal
+   - Format: vector DB + knowledge graph
+   - Use for: learned patterns, user preferences, domain expertise
+   ```
+
+### Retrieval Strategies
+
+2. **Retrieval by memory layer:**
+   ```
+   Working Memory:
+   - Already in context — no retrieval needed
+   - Manage carefully: don't waste on retrievable facts
+
+   Short-Term Memory:
+   - Recency-weighted retrieval (most recent = most relevant)
+   - Key-based lookup: "what was the last error message?"
+   - Cleared at session end (or explicit flush)
+
+   Medium-Term Memory:
+   - Keyword + semantic hybrid search
+   - Structured queries: "what auth pattern does this project use?"
+   - Indexed by: topic, file path, date, relevance score
+
+   Long-Term Memory:
+   - Semantic similarity search (embedding-based)
+   - Knowledge graph traversal (entity → relationship → entity)
+   - Confidence-weighted (higher confidence = higher ranking)
+   - Decay-adjusted (older unreinforced memories rank lower)
+   ```
+
+3. **Retrieval decision flow:**
+   ```
+   When agent needs information:
+   1. Check working memory (already in context?) → use it
+   2. Check short-term memory (recent session fact?) → retrieve and inject
+   3. Check medium-term memory (project knowledge?) → search and inject summary
+   4. Check long-term memory (learned pattern?) → search, verify relevance, inject
+   5. If not found anywhere → ask user or research from scratch
+   ```
+
+### Memory Consolidation
+
+4. **End-of-session consolidation pipeline:**
+   ```
+   Session ends → Consolidation triggers:
+
+   1. Extract key learnings from session:
+      - New facts learned (user preferences, project decisions)
+      - Patterns observed (what worked, what failed)
+      - Corrections received (mistakes to avoid next time)
+
+   2. Classify each learning:
+      - Short-term only (task-specific, won't matter next session) → discard
+      - Medium-term (project-relevant) → write to project memory
+      - Long-term (generalizable pattern) → write to permanent memory
+
+   3. Summarize, don't dump:
+      - Raw conversation → extract 5-10 key facts
+      - Include WHY something matters, not just WHAT happened
+      - Link to existing memories (reinforce or update)
+
+   4. Update indexes:
+      - Add new entries to search indexes
+      - Update confidence scores on existing memories
+      - Deprecate contradicted memories
+   ```
+
+   Rules:
+   - Consolidation must be LOSSY (summarize, compress, extract — not raw dump)
+   - Every memory entry needs: content, source, timestamp, confidence, tags
+   - Contradictions: new information supersedes old (but keep old as "deprecated")
+
+### Memory Decay
+
+5. **Confidence decay model:**
+   ```
+   Each memory entry has a confidence score [0.0 - 1.0]:
+
+   Initial confidence:
+   - User explicitly stated: 1.0
+   - Inferred from behavior: 0.7
+   - Assumed from patterns: 0.5
+
+   Decay rules:
+   - Unreinforced memory: confidence -= 0.05 per week
+   - Reinforced memory (used successfully): confidence = min(1.0, confidence + 0.1)
+   - Contradicted memory: confidence = 0.0 (deprecated, kept for history)
+   - Below threshold (confidence < 0.2): excluded from retrieval results
+
+   Reinforcement triggers:
+   - Agent retrieves memory and uses it successfully
+   - User confirms a remembered fact
+   - Pattern matches current observation
+   ```
+
+### Implementation Patterns
+
+6. **Storage format by layer:**
+   ```
+   Short-term (JSONL scratchpad):
+   {"key": "last_error", "value": "TypeError at line 42", "ts": "...", "confidence": 1.0}
+   {"key": "user_intent", "value": "refactoring auth module", "ts": "...", "confidence": 0.9}
+
+   Medium-term (structured markdown):
+   ## Project: MindForge
+   ### Architecture Decisions
+   - Auth: JWT with refresh tokens (decided 2024-03-15, confidence: 1.0)
+   - Database: PostgreSQL with Prisma ORM (decided 2024-03-10, confidence: 1.0)
+   ### User Preferences
+   - Prefers functional style over OOP (confidence: 0.8)
+   - Wants verbose error messages in dev (confidence: 0.9)
+
+   Long-term (vector DB + knowledge graph):
+   Entry: {id, embedding, content, source, timestamp, confidence, tags, relationships}
+   Graph: (pattern)--[applies_to]-->(domain)--[learned_from]-->(project)
+   ```
+
+7. **Working memory budget management:**
+   ```
+   Context window is finite — manage it:
+
+   Priority for inclusion in working memory:
+   1. Current user message and task context (always)
+   2. Relevant retrieved memories (top-k by relevance)
+   3. System instructions and constraints (always)
+   4. Recent conversation history (sliding window)
+
+   If context is filling up:
+   - Summarize older conversation turns (don't drop, compress)
+   - Move detailed context to short-term memory (retrieve if needed)
+   - Keep only TOP-5 most relevant retrieved memories in context
+   - Never sacrifice system instructions for conversation history
+   ```
+
+## Self-check before task completion
+
+Before marking a task done when this skill was active:
+
+- [ ] Did I define all four memory layers with clear scope and access patterns?
+- [ ] Is retrieval strategy defined per layer (recency, semantic, key-based)?
+- [ ] Is there a consolidation pipeline that extracts and summarizes key learnings?
+- [ ] Is consolidation lossy (summarize, don't dump raw conversation)?
+- [ ] Is memory decay implemented (confidence decreases without reinforcement)?
+- [ ] Are contradicted memories deprecated (not deleted)?
+- [ ] Is working memory budget managed (prioritized inclusion, compression when full)?
+- [ ] Does every memory entry have: content, source, timestamp, confidence, tags?