mindforge-cc 10.0.3 → 11.0.0

package/.mindforge/skills/agent-orchestration-patterns/SKILL.md

@@ -0,0 +1,129 @@
+---
+name: agent-orchestration-patterns
+version: 1.0.0
+min_mindforge_version: 10.0.7
+status: stable
+triggers: agent orchestration pattern, supervisor worker pattern, agent pipeline topology, agent debate pattern, agent consensus protocol, map reduce agents, handoff protocol design, multi-agent coordination, agent topology design, swarm pattern design, agent composition, failure propagation pattern
+compose:
+  - tool-design
+---
+
+# Agent Orchestration Patterns
+
+## When this skill activates
+
+This skill activates when designing multi-agent systems, choosing coordination topologies, implementing handoff protocols, or debugging agent-to-agent communication failures. It applies to any system where two or more autonomous agents must collaborate, compete, or chain their outputs to accomplish a goal.
+
+## Mandatory actions when this skill is active
+
+### Before
+
+1. **Map the problem space** — Identify all subtasks. Determine which require sequential execution (dependencies) and which are independent (parallelizable).
+2. **Assess complexity** — Single-agent tasks masquerading as multi-agent problems waste coordination overhead. Only orchestrate when genuine specialization or parallelism is needed.
+3. **Define boundaries** — Each agent must have a clear responsibility boundary. Overlapping responsibilities cause conflicts. Gaps cause dropped tasks.
+4. **Choose state strategy** — Decide upfront: shared state (agents read/write common store) or isolated state (agents communicate only via messages).
+
+### During
+
+#### Pattern Catalog
+
+**1. Supervisor/Worker (Hub and Spoke)**
+- **Topology** — One coordinator agent decomposes the task and dispatches subtasks to N worker agents. Workers report results back to the supervisor.
+- **When to use** — Task is decomposable into independent units. Workers are interchangeable or specialized but non-overlapping.
+- **Supervisor responsibilities** — Task decomposition, worker assignment, result aggregation, error handling, timeout enforcement.
+- **Worker responsibilities** — Execute assigned subtask, report structured results, signal failure early.
+- **Pitfall** — Supervisor becomes bottleneck. Mitigate with async dispatch and parallel worker execution.
+
+**2. Pipeline (Sequential Chain)**
+- **Topology** — Agent A's output becomes Agent B's input. Linear flow through N stages.
+- **When to use** — Tasks have natural ordering (research → draft → review → publish). Each stage transforms or enriches the previous output.
+- **Stage contract** — Each stage must define its input schema and output schema. Type mismatch between stages is the most common pipeline failure.
+- **Error handling** — Fail the pipeline on any stage failure. Partial results from earlier stages should be preserved for debugging.
+- **Optimization** — Streaming between stages reduces latency. Agent B can begin processing as Agent A emits output.
+
+**3. Debate (Adversarial)**
+- **Topology** — Two or more agents argue opposing positions. A synthesizer agent evaluates arguments and produces a final decision.
+- **When to use** — High-stakes decisions where bias is a risk. Architecture choices, security reviews, strategic decisions.
+- **Protocol** — Round 1: each debater states position with evidence. Round 2: each debater rebuts opponent's position. Round 3: synthesizer produces verdict with reasoning.
+- **Constraint** — Debaters must not see each other's initial positions until after Round 1. Prevents anchoring.
+- **Pitfall** — Debates can be unproductive without strict structure. Always time-box rounds.
+
+**4. Consensus (Agreement Required)**
+- **Topology** — All agents must agree on the output. Disagreement triggers re-evaluation.
+- **When to use** — Safety-critical decisions. Deployment approvals. Security assessments. Changes where false positives are acceptable but false negatives are dangerous.
+- **Protocol** — Each agent independently evaluates. If all approve: proceed. If any reject: block and surface the dissenting reasoning.
+- **Threshold variants** — Unanimous (all agree), Majority (>50%), Supermajority (>66%), Quorum (minimum N must vote).
+- **Pitfall** — Consensus is expensive. Reserve for decisions where the cost of a wrong answer far exceeds the cost of deliberation.
+
+**5. MapReduce (Parallel Processing)**
+- **Topology** — Map phase: split input into N chunks, dispatch to N parallel agents. Reduce phase: aggregate results into final output.
+- **When to use** — Large inputs that can be processed independently (code review across files, document analysis, test execution).
+- **Map function** — Must produce non-overlapping chunks. Overlap causes duplicate work or conflicting results.
+- **Reduce function** — Must handle partial failures gracefully. If 1 of 10 map workers fails, the reduce should still produce useful output from the other 9.
+- **Scaling** — Add more workers linearly. Bottleneck is the reduce step, not the map step.
+
+#### Handoff Protocol Design
+
+Every agent-to-agent handoff must include a structured message:
+
+```json
+{
+  "task_id": "unique-identifier",
+  "from_agent": "agent-name",
+  "to_agent": "agent-name",
+  "task": "clear description of what to do",
+  "context": "relevant background (minimal, not full history)",
+  "constraints": ["must not modify X", "timeout 30s"],
+  "acceptance_criteria": ["output matches schema Y", "all tests pass"],
+  "artifacts": ["file paths or data references"]
+}
+```
+
+- **Minimal context** — Send only what the receiving agent needs. Full history causes confusion and wastes tokens.
+- **Explicit acceptance criteria** — The receiving agent must know when it has succeeded.
+- **Typed artifacts** — Reference files or data by path/ID, not by embedding content in the message.
+
+#### Failure Propagation Strategies
+
+| Strategy | Behavior | Use When |
+|----------|----------|----------|
+| Fail-fast | Abort immediately, surface error | Critical path, no recovery possible |
+| Retry | Repeat N times with backoff | Transient failures (network, rate limits) |
+| Escalate | Notify supervisor, request human input | Ambiguous failures, policy decisions |
+| Degrade | Continue with partial results, flag gaps | Non-critical subtasks, best-effort acceptable |
+| Circuit-break | Stop retrying after N failures, return cached/default | Dependency is unreliable |
+
+#### State Management
+
+- **Shared state** — All agents read/write a common store (database, shared memory). Simpler but requires conflict resolution (optimistic locking, CRDTs).
+- **Isolated state** — Agents maintain private state, communicate only via messages. Safer but requires explicit state transfer in handoffs.
+- **Hybrid** — Shared read-only state (project context, configuration) + isolated write state (each agent's working memory). Best balance for most systems.
+
+#### Decision Matrix: When to Use Which Pattern
+
+| Scenario | Pattern |
+|----------|---------|
+| Task decomposes into independent subtasks | MapReduce or Supervisor/Worker |
+| Tasks must execute in order | Pipeline |
+| High-stakes decision needs scrutiny | Debate or Consensus |
+| One coordinator manages many executors | Supervisor/Worker |
+| System must tolerate partial failures | MapReduce with degraded reduce |
+| Speed is critical, tasks are independent | MapReduce with max parallelism |
+
+### After
+
+1. **Validate handoff contracts** — Test that each agent produces output matching the next agent's expected input schema.
+2. **Test failure modes** — Simulate each failure propagation path. Verify the system degrades gracefully, not catastrophically.
+3. **Measure overhead** — Coordination cost should be <20% of total execution time. If higher, simplify the topology.
+4. **Document topology** — Create a diagram showing agent relationships, handoff directions, and failure paths.
+
+## Self-check before task completion
+
+- [ ] Pattern choice is justified by task structure (not over-engineered)
+- [ ] Each agent has clear, non-overlapping responsibilities
+- [ ] Handoff protocol includes task, context, constraints, and acceptance criteria
+- [ ] Failure propagation strategy is defined for every inter-agent connection
+- [ ] State management approach is explicit (shared vs isolated vs hybrid)
+- [ ] Coordination overhead is measured and acceptable (<20% of total time)
+- [ ] All agent-to-agent contracts are typed and validated
+- [ ] System degrades gracefully under partial failure conditions

package/.mindforge/skills/agent-tool-selection/SKILL.md

@@ -0,0 +1,204 @@
+---
+name: agent-tool-selection
+version: 1.0.0
+min_mindforge_version: 10.0.4
+status: stable
+triggers: agent tool selection, capability matching, tool routing, cost-aware tool choice, tool fallback chain, tool composition strategy, function selection, tool description optimization, tool disambiguation, tool preference, tool capability matrix, tool ranking
+compose: tool-design
+---
+
+# Skill — Agent Tool Selection (Intelligent Capability Routing)
+
+## When this skill activates
+When designing tool selection logic for AI agents, optimizing tool descriptions,
+building fallback chains, or analyzing tool usage patterns. Use for any scenario
+where an agent must choose between multiple tools to accomplish a task.
+
+Core principle: **Specificity over generality** — when two tools can both handle a
+task, prefer the more specific one. It will be faster, cheaper, and more reliable.
+A hammer works for screws, but a screwdriver is better.
+
+## Mandatory actions when this skill is active
+
+### Tool Selection Algorithm
+
+1. **Selection pipeline (in order):**
+   ```
+   Input: task description + available tools
+
+   Step 1 — Capability Match:
+   - For each tool: does its capability set cover the task requirements?
+   - Eliminate tools that CANNOT handle the task (hard filter)
+
+   Step 2 — Rank by Specificity:
+   - More specific tool > more general tool
+   - Example: "Read file" > "Bash (cat)" for reading files
+   - Specificity = how narrow is the tool's intended use case?
+
+   Step 3 — Rank by Cost-Efficiency:
+   - Among equally capable tools: prefer cheaper/faster
+   - Read (free, instant) > Bash cat (process spawn) > API call (network)
+
+   Step 4 — Rank by Reliability:
+   - Among equal cost: prefer higher success rate
+   - Check historical success rate per tool per task type
+
+   Step 5 — Verify Preconditions:
+   - Does the selected tool's preconditions hold?
+   - Example: Edit requires file was previously Read
+   - If preconditions not met: add prerequisite steps
+
+   Output: ordered list of tools to try (primary + fallbacks)
+   ```
+
+2. **Decision matrix template:**
+   ```
+   | Task Type          | Primary Tool | Fallback 1   | Fallback 2  | Anti-pattern     |
+   |--------------------|--------------|--------------|-----------  |------------------|
+   | Read file content  | Read         | Bash (cat)   | —           | Grep (wrong use) |
+   | Search for pattern | Grep         | Bash (grep)  | Read + scan | Read all files   |
+   | Edit existing file | Edit         | Write        | —           | Bash (sed)       |
+   | Create new file    | Write        | Bash (echo>) | —           | Edit (no file)   |
+   | Run tests          | Bash         | —            | —           | Read test output |
+   | Check file exists  | Bash (ls)    | Read (error) | —           | Grep for path    |
+   ```
+
+### Tool Description Optimization
+
+3. **Writing effective tool descriptions (they ARE prompts):**
+   ```
+   Good description (specific, with examples):
+   "Read a file from disk. Use when you need to see file contents.
+    Supports text, images, PDFs. Prefer over Bash cat/head/tail.
+    NOT for directories (use Bash ls)."
+
+   Bad description (vague):
+   "Reads things from the filesystem."
+
+   Good description (with when-to-use and when-NOT-to-use):
+   "Edit an existing file by replacing exact string matches.
+    Use when: modifying 1-5 specific locations in a file.
+    Do NOT use when: rewriting >50% of the file (use Write instead).
+    Requires: file must have been Read in this session first."
+
+   Bad description (missing boundaries):
+   "Edits files."
+   ```
+
+   Rules:
+   - Include WHEN to use (positive examples)
+   - Include when NOT to use (negative examples — prevents misuse)
+   - State preconditions explicitly
+   - Keep descriptions under 100 words (concise > comprehensive)
+   - Use concrete examples, not abstract capabilities
+
+### Cost-Aware Selection
+
+4. **Cost hierarchy (prefer cheaper when quality is equal):**
+   ```
+   Tier 1 — Free/Instant (prefer these):
+   - Read (file content)
+   - Edit (modify file)
+   - Write (create file)
+   - Grep (pattern search in known scope)
+
+   Tier 2 — Cheap/Fast (use when Tier 1 can't):
+   - Bash (shell commands — process spawn overhead)
+   - Glob (file path patterns)
+
+   Tier 3 — Moderate (use when necessary):
+   - LSP (language server queries)
+   - Web fetch (network requests)
+
+   Tier 4 — Expensive (use sparingly):
+   - Sub-agent spawn (full agent instantiation)
+   - Multi-file analysis (token-heavy)
+   - External API calls (rate-limited, costly)
+   ```
+
+   Rules:
+   - Always check if a Tier 1 tool can handle the task before reaching for Tier 3-4
+   - Track cumulative cost during a session (don't let tool costs compound silently)
+   - For repeated operations: batch when possible (one Bash with && vs many Bash calls)
+   - Cost includes: token consumption, time, API calls, compute resources
+
+### Fallback Chains
+
+5. **Designing robust fallback sequences:**
+   ```
+   Fallback chain structure:
+   1. Try primary tool (most specific, cheapest)
+   2. If fails (error, timeout, precondition unmet):
+      - Log failure reason
+      - Try fallback 1 (broader capability)
+   3. If fallback 1 fails:
+      - Try fallback 2 (most general/expensive)
+   4. If all fail:
+      - Escalate to user with: what was tried, why each failed, what's needed
+
+   Example:
+   Task: "Find where function X is defined"
+   1. Grep (fast, pattern-based) → found? done
+   2. LSP (semantic, language-aware) → found? done
+   3. Bash find + grep (brute force) → found? done
+   4. Escalate: "I couldn't locate function X. Can you point me to the file?"
+   ```
+
+   Rules:
+   - Fallback chains should be pre-defined per task type (not improvised)
+   - Each fallback should be DIFFERENT in approach (not just retry)
+   - Log which level of the chain succeeded (optimize primary over time)
+   - Max 3 fallback levels before escalation (avoid infinite retry loops)
+
+### Tool Composition
+
+6. **Combining tools for complex tasks:**
+   ```
+   Composition patterns:
+
+   Sequential: Tool A output → Tool B input
+   Example: Grep (find file) → Read (get content) → Edit (modify)
+
+   Parallel: Tool A + Tool B independently → merge results
+   Example: Grep (find usages) + Read (get definition) → understand full context
+
+   Conditional: If Tool A succeeds → Tool B, else → Tool C
+   Example: If Read(file) succeeds → Edit, else → Write (file doesn't exist)
+
+   Iterative: Repeat Tool A until condition met
+   Example: Bash(test) → fails → Edit(fix) → Bash(test) → passes → done
+   ```
+
+   Rules:
+   - Plan composition BEFORE executing (don't improvise mid-chain)
+   - Minimize total tool calls (combine steps where possible)
+   - Never call the same tool twice with identical inputs (cache/reuse results)
+   - If composition exceeds 5 sequential steps: consider if there's a more direct tool
+
+### Tool Disambiguation
+
+7. **When multiple tools seem equally valid:**
+   ```
+   Disambiguation criteria (in priority order):
+   1. Fewer side effects: Read-only > Read-write (prefer observation over action)
+   2. More specific: Narrow tool > broad tool (Edit > Bash sed)
+   3. Cheaper: Less resource consumption > more
+   4. More reversible: Undoable > permanent (Edit > Write for existing files)
+   5. Better error messages: Tools with clear failure modes > opaque failures
+
+   If still tied after all criteria: pick the one that appears first in the
+   tool list (convention-based tie-breaking, prevents analysis paralysis)
+   ```
+
+## Self-check before task completion
+
+Before marking a task done when this skill was active:
+
+- [ ] Did I follow the selection pipeline (capability → specificity → cost → reliability)?
+- [ ] Are tool descriptions specific, with positive AND negative usage examples?
+- [ ] Is cost hierarchy respected (cheaper tools preferred when quality is equal)?
+- [ ] Are fallback chains defined for each critical task type (max 3 levels)?
+- [ ] Is tool composition planned before execution (not improvised)?
+- [ ] Are disambiguations resolved by: side effects → specificity → cost → reversibility?
+- [ ] Are tool calls minimized (no redundant calls, batched where possible)?
+- [ ] Is escalation to user defined as the final fallback (not infinite retry)?

package/.mindforge/skills/ai-agent-deployment/SKILL.md

@@ -0,0 +1,176 @@
+---
+name: ai-agent-deployment
+version: 1.0.0
+min_mindforge_version: 10.1.1
+status: stable
+triggers: ai agent deployment, agent hosting, agent scaling, agent versioning, agent A/B testing, agent monitoring production, agent rollback, agent health check, agent cost production, agent performance monitoring, agent canary, agent shadow testing
+compose: deployment-workflow
+---
+
+# Skill — AI Agent Deployment
+
+## When this skill activates
+Any task involving deploying AI agents to production, versioning agent configurations,
+A/B testing agent variants, monitoring agent quality in production,
+or managing the operational lifecycle of AI agents.
+
+## Mandatory actions when this skill is active
+
+### Before writing any code
+1. Define the agent version tuple: model + prompt + tools + config (all pinned together).
+2. Identify success metrics (quality, latency, cost, user satisfaction).
+3. Plan rollback strategy (instant version pointer switch).
+4. Design monitoring (token usage, error rate, quality signal).
+
+### During implementation
+- Package agent as a versioned, immutable deployment artifact.
+- Implement health check endpoint (synthetic task probe).
+- Add structured logging for every agent action (input, output, tools used, tokens).
+- Build traffic splitting capability for A/B and canary.
+- Instrument cost tracking per-task and per-user.
+- Implement graceful degradation (fallback to simpler model on failure).
+
+### After implementation
+- Verify shadow test shows no regression vs current version.
+- Confirm monitoring dashboards capture all key metrics.
+- Test rollback procedure end-to-end.
+- Validate cost projections against actual usage.
+- Run synthetic probes for health verification.
+
+## Versioning Strategy
+
+### Agent Version = Immutable Tuple
+```json
+{
+  "version": "agent-v2.3.1",
+  "model": "claude-sonnet-4-20250514",
+  "prompt_hash": "sha256:abc123...",
+  "tools": ["search_v2", "code_exec_v1", "web_browse_v3"],
+  "config": {
+    "temperature": 0.3,
+    "max_tokens": 4096,
+    "timeout_ms": 30000
+  }
+}
+```
+
+### Rules
+- Changing ANY component = new version.
+- Never mutate a deployed version in place.
+- Keep previous N versions warm for instant rollback.
+- Version string includes all components for traceability.
+
+## Hosting Patterns
+
+### Containerized (Recommended)
+- Docker container with model client, prompt, tool implementations.
+- Auto-scale on queue depth (not CPU — agents are I/O bound).
+- GPU allocation only if running local inference.
+- Isolate per-tenant for data separation.
+
+### Scaling Signals
+| Signal | Scale Direction | Reason |
+|--------|----------------|--------|
+| Queue depth increasing | Scale up | Work is backing up |
+| P95 latency rising | Scale up | Capacity insufficient |
+| Queue empty for 5min | Scale down | Over-provisioned |
+| Error rate > 5% | Pause scaling | Fix errors first |
+
+## A/B Testing
+
+### Setup
+1. Define hypothesis (e.g., "new prompt reduces hallucination by 20%").
+2. Split traffic (e.g., 90/10 control/experiment).
+3. Run for statistical significance (typically 1000+ samples per variant).
+4. Measure: quality score, latency, cost, user feedback.
+
+### Metrics to Compare
+- Task success rate (did the agent complete the task correctly?).
+- Token usage (cost proxy).
+- Latency p50/p95/p99.
+- Tool failure rate.
+- User satisfaction signal (thumbs up/down, follow-up corrections).
+- Hallucination rate (if measurable via ground truth).
+
+### Graduation Criteria
+- Improvement statistically significant (p < 0.05).
+- No regression in any critical metric.
+- Cost increase acceptable (<20% for same quality).
+
+## Shadow Testing
+
+### Pattern
+```
+User Request → Production Agent (responds to user)
+            → Shadow Agent (runs silently, output logged)
+```
+
+### Purpose
+- Test new version against real traffic without user impact.
+- Compare outputs offline (human eval or automated scoring).
+- Detect regressions before any user sees them.
+
+### Rules
+- Shadow agent output never reaches the user.
+- Shadow uses same input but may have different model/prompt/tools.
+- Compare at scale (1000+ requests) before promoting.
+- Track divergence rate and categorize differences.
+
+## Monitoring
+
+### Key Metrics (Real-Time Dashboard)
+| Metric | Alert Threshold | Action |
+|--------|----------------|--------|
+| Token usage/task | >2x baseline | Check for loops/verbose output |
+| Latency p95 | >30s | Scale up or investigate bottleneck |
+| Tool failure rate | >5% | Check tool availability |
+| Hallucination rate | >3% | Rollback, investigate prompt |
+| User negative feedback | >10% | Investigate, consider rollback |
+| Cost per task | >$0.50 | Check for inefficiency |
+
+### Structured Logging
+Every agent invocation must log:
+- Request ID, user ID, agent version.
+- Input (sanitized of PII).
+- Output summary.
+- Tools invoked and their results.
+- Token counts (input, output, total).
+- Latency breakdown (thinking, tool calls, generation).
+- Success/failure determination.
+
+## Rollback
+
+### Instant Rollback
+- Version pointer in config store (not redeployment).
+- Switch pointer → immediate traffic to previous version.
+- Keep N previous versions warm (containers running, ready).
+- Rollback decision within 5 minutes of detecting regression.
+
+### Rollback Triggers (Automatic)
+- Error rate > 10% for 3 consecutive minutes.
+- P95 latency > 60s for 5 minutes.
+- User negative feedback spike (3x normal rate).
+
+## Health Checks
+
+### Synthetic Probes
+- Run a known-good task against the agent every 5 minutes.
+- Verify output matches expected structure.
+- Check latency within bounds.
+- Alert if probe fails 2 consecutive times.
+
+### Probe Design
+- Task must be deterministic (or have verifiable structure).
+- Must exercise core capabilities (reasoning + at least one tool).
+- Must complete within health check timeout (10s recommended).
+- Results logged for trend analysis.
+
+## Self-check
+- [ ] Agent version tuple defined (model + prompt + tools + config).
+- [ ] Health check probes running every 5 minutes.
+- [ ] Monitoring covers: tokens, latency, errors, quality, cost.
+- [ ] Rollback tested and confirmed instant.
+- [ ] Shadow test shows no regression.
+- [ ] A/B framework ready for future experiments.
+- [ ] Cost per task tracked and within budget.
+- [ ] Graceful degradation implemented for failures.

package/.mindforge/skills/ai-cost-management/SKILL.md

@@ -0,0 +1,57 @@
+---
+name: ai-cost-management
+version: 1.0.0
+min_mindforge_version: 10.5.0
+status: stable
+triggers: AI cost management, token budget optimization, model cost selection, LLM response caching, batch inference processing, AI infrastructure cost, token usage monitoring, cost-aware model routing, inference cost reduction, GPU utilization optimization, AI spend tracking, cost per query optimization
+compose:
+  - llm-cost-optimization
+---
+
+# AI Cost Management & Optimization
+
+## When this skill activates
+
+This skill activates when optimizing AI inference costs, implementing token budgeting, designing cost-aware model routing, or tracking AI spending at scale. It applies to production AI systems where compute costs are significant (>$10K/month) and must be controlled without degrading user experience.
+
+## Mandatory actions when this skill is active
+
+### Before writing any code
+
+1. **Establish cost baseline** — Measure current costs per dimension: cost per request, cost per user, cost per model, cost per task type. Identify cost drivers: which models, which tasks, which users consume the most budget. Focus optimization efforts on the top 20% of cost drivers (Pareto principle).
+2. **Set cost budgets and alerts** — Define acceptable cost thresholds per day, per week, per month. Set up alerts when spending exceeds thresholds (50%, 80%, 100%). Alerts prevent runaway costs from unexpected usage spikes or inefficient code.
+3. **Design cost attribution model** — Assign costs to cost centers: per-team, per-product, per-customer tier (free vs. paid). Enables chargeback (internal teams pay for their usage) and informs product decisions (should free tier have lower model quality to reduce costs?).
+4. **Identify optimization opportunities** — Rank opportunities by potential savings: caching (30-70% reduction for repetitive queries), model downgrade (10-50% reduction with minimal quality loss), batching (20-40% reduction via throughput optimization), prompt compression (10-30% reduction by reducing tokens).
+
+### During implementation
+
+- **Implement aggressive response caching** — Cache LLM responses keyed by prompt hash + model + hyperparameters. Cache hit rate >50% is achievable for many use cases. Use Redis or Memcached for low-latency lookups (<1ms). Set TTL based on content freshness requirements (hours for news, days for documentation, indefinite for static content).
+- **Design cost-aware routing** — Route requests to the cheapest model that meets quality requirements. Example: simple classification → Haiku ($0.25/MTok), complex reasoning → Opus ($15/MTok). Measure quality degradation when downgrading models. If accuracy drop is <2%, downgrade is safe.
+- **Compress prompts aggressively** — Remove filler words, use abbreviations, compress whitespace. Test that compressed prompts produce equivalent outputs. Measure compression ratio (original tokens / compressed tokens). Target: 20-50% compression with <1% quality loss.
+- **Batch inference for throughput** — Group requests into batches (10-100 per batch) to maximize GPU utilization. Batching increases throughput (requests/second) and reduces cost per request (amortize fixed overhead). Trade-off: higher latency (wait for batch to fill) for lower cost.
+- **Implement prompt caching (for supported models)** — Use prompt caching for repeated prompt prefixes (system message, context). Claude and GPT-4 support prompt caching. Reduces cost by 90% for cached tokens. Ensure prompts are structured with stable prefixes and variable suffixes.
+- **Track token usage per request** — Log input tokens, output tokens, and total cost per request. Aggregate by model, task type, user, and time. Identify outliers: queries with unusually high token counts (may indicate inefficient prompts or bugs). Set up monitoring dashboards.
+
+### After implementation
+
+- **Measure cache hit rate** — Track % of requests served from cache. Target: >50% hit rate for production systems with repetitive queries. If lower, analyze cache misses: are prompts subtly different (normalize prompts), is TTL too short (increase TTL), or is traffic too diverse (caching won't help)?
+- **Validate quality after cost optimization** — Compare model accuracy, user satisfaction, or business metrics before and after optimization. Acceptable thresholds: <2% accuracy drop, <5% user satisfaction drop. If degradation is higher, roll back optimizations.
+- **Benchmark cost reduction** — Measure cost per request after optimizations vs. baseline. Target: 30-50% cost reduction from caching + routing + compression. Document savings in $/month and ROI (developer time spent vs. cost saved).
+- **Monitor for cost anomalies** — Set up alerts for sudden cost spikes (>2x daily average) or unusual patterns (single user consuming 10x typical usage). Anomalies indicate bugs (infinite loops, retry storms) or abuse (attackers exploiting free tier).
+
+## Self-check before task completion
+
+- [ ] Cost baseline is measured per request, user, model, and task type
+- [ ] Cost budgets are set with alerts at 50%, 80%, 100% thresholds
+- [ ] Cost attribution model assigns costs to teams, products, or customer tiers
+- [ ] Optimization opportunities are ranked by potential savings and effort
+- [ ] Response caching is implemented with cache hit rate tracked (target >50%)
+- [ ] Cost-aware routing selects cheapest model that meets quality requirements
+- [ ] Prompt compression achieves 20-50% token reduction with <1% quality loss
+- [ ] Batch inference is implemented for throughput optimization (10-100 requests per batch)
+- [ ] Prompt caching (if supported) reduces cost by 90% for repeated prompt prefixes
+- [ ] Token usage is logged per request and aggregated by model, task, user, time
+- [ ] Cache hit rate is validated at >50% for production systems with repetitive queries
+- [ ] Model quality is validated post-optimization (<2% accuracy drop, <5% satisfaction drop)
+- [ ] Cost reduction is benchmarked (target 30-50% reduction from baseline)
+- [ ] Cost anomaly alerts are configured for spikes (>2x daily average) and abuse patterns

package/.mindforge/skills/ai-safety-alignment/SKILL.md

@@ -0,0 +1,53 @@
+---
+name: ai-safety-alignment
+version: 1.0.0
+min_mindforge_version: 10.5.0
+status: stable
+triggers: AI safety implementation, AI alignment technique, output filtering AI, content moderation AI, bias detection model, AI guardrail implementation, harmful content prevention, AI red teaming, model safety evaluation, AI ethics implementation, alignment testing, responsible AI deployment
+compose:
+  - guardrails-and-safety
+---
+
+# AI Safety & Alignment
+
+## When this skill activates
+
+This skill activates when implementing output filters, content moderation systems, bias detection, adversarial robustness, or alignment techniques for AI systems. It applies when deploying AI in production environments where safety failures have legal, reputational, or ethical consequences.
+
+## Mandatory actions when this skill is active
+
+### Before writing any code
+
+1. **Define harm taxonomy** — Categorize potential harms specific to your domain: toxic content, misinformation, bias (demographic, cultural), privacy leaks, prompt injection, jailbreaks, copyright violations. Prioritize by severity and likelihood. Not all harms are equal.
+2. **Establish safety thresholds** — Define numeric thresholds for each harm category: toxicity score <0.3, PII detection confidence >0.9, bias parity ratio 0.8-1.2. Thresholds must be tuned on representative data, not arbitrary guesses.
+3. **Select detection models** — Choose specialized classifiers per harm type: Perspective API or custom models for toxicity, named entity recognition for PII, fairness metrics for bias. General-purpose LLMs are too slow and expensive for real-time safety filtering.
+4. **Design layered defense** — Implement multiple safety layers: input validation (reject malicious prompts), model guardrails (constrain model behavior), output filtering (catch harmful completions), monitoring (detect safety failures post-deployment). Single-layer defense is insufficient.
+
+### During implementation
+
+- **Implement input sanitization first** — Validate all user inputs before reaching the model. Reject or sanitize: SQL injection patterns, prompt injection attempts (ignore previous instructions), PII in prompts, excessive length, special characters that break parsing. Log rejected inputs for analysis.
+- **Apply constitutional AI principles** — Constrain model behavior via system prompts: "You are a helpful assistant. You must not generate harmful, biased, or illegal content. If asked to do so, politely refuse and explain why." Test refusal behavior extensively.
+- **Use classifier-guided generation** — For high-risk domains, run a safety classifier on every output before returning to the user. If toxicity/bias/PII is detected above threshold, retry generation with a modified prompt or return a safe default response.
+- **Implement red-teaming as tests** — Create automated adversarial tests: jailbreak attempts, bias triggers, edge cases designed to elicit failures. Run these tests in CI/CD. Safety regressions must block deployment.
+- **Log all safety events** — Record every safety filter activation: input rejected, output filtered, threshold exceeded. Include context: user ID, timestamp, input/output text, classifier scores. This data is critical for tuning thresholds and identifying attack patterns.
+- **Design graceful degradation** — When safety filters trigger, provide user-friendly explanations: "I can't generate that content because [reason]." Do not expose internal classifier scores or filter logic (attackers use this to evade detection).
+
+### After implementation
+
+- **Validate safety coverage** — Test the system with a held-out safety dataset: known toxic examples, bias triggers, jailbreak prompts. Measure recall (% of harmful content caught) and precision (% of flagged content that is truly harmful). Target: recall >95%, precision >90%.
+- **Measure bias across demographics** — Evaluate model outputs for demographic parity, equalized odds, and calibration across protected attributes (race, gender, age). Use fairness toolkits (Fairlearn, AI Fairness 360). Bias gaps >20% require mitigation.
+- **Conduct human red-teaming** — Hire external red-teamers to attempt jailbreaks and adversarial attacks. Automated tests miss creative attack vectors. Budget 20-40 hours of red-teaming per major release.
+- **Monitor safety in production** — Track safety metrics over time: filter activation rate, false positive rate, user complaints about incorrect filtering. Safety degrades as attackers adapt and user behavior shifts.
+
+## Self-check before task completion
+
+- [ ] Harm taxonomy is defined with severity ratings and likelihood estimates
+- [ ] Safety thresholds are set per harm type and validated on representative data
+- [ ] Input sanitization rejects malicious prompts before reaching the model
+- [ ] Output filtering runs on every completion with <100ms latency overhead
+- [ ] Constitutional AI constraints are embedded in system prompts and tested
+- [ ] Automated red-teaming tests run in CI/CD and block deployment on failures
+- [ ] Safety events are logged with full context for post-hoc analysis
+- [ ] Bias is measured across demographics with fairness parity within acceptable bounds
+- [ ] Human red-teaming has been conducted with documented attack attempts and mitigations
+- [ ] Production monitoring tracks filter activation rate and false positive trends