wizz-method 1.3.2 → 1.4.0

package/src/skills-lib/ai-product-design/references/design-agent-orchestration/agent-role-design/SKILL.md

@@ -0,0 +1,37 @@
+---
+name: agent-role-design
+description: Defining what each agent does, knows, and owns in a multi-agent system.
+---
+# Agent Role Design
+In a multi-agent system, each agent needs a clearly defined role — what it does, what it knows, what it's responsible for, and where its authority ends. Without clear roles, agents duplicate work, conflict with each other, or leave gaps.
+## Defining an Agent Role
+For each agent in the system, specify:
+- **Purpose**: What is this agent for? One sentence describing its reason to exist.
+- **Capabilities**: What can this agent do? List specific actions and outputs.
+- **Knowledge scope**: What does this agent know about? What domains, data, and context does it have access to?
+- **Authority**: What decisions can this agent make autonomously? What requires approval?
+- **Boundaries**: What is explicitly outside this agent's scope? Where does it stop and hand off?
+- **Success criteria**: How do you know this agent is doing its job well?
+## Role Design Principles
+- **Single responsibility**: Each agent should have one clear purpose. If you need a paragraph to explain what it does, it's doing too much.
+- **Clear boundaries**: The line between one agent's scope and another's should be unambiguous. No overlapping authority without explicit conflict resolution.
+- **Minimal coupling**: Agents should be able to do their work with minimal dependencies on other agents. Share results, not process.
+- **Appropriate autonomy**: The level of autonomous decision-making should match the stakes and the agent's reliability in that domain.
+## Role Patterns
+- **Specialist**: Deep expertise in one domain. Handles all tasks of a specific type.
+- **Router**: Doesn't do work itself but directs tasks to the right specialist.
+- **Orchestrator**: Manages the overall workflow, coordinates between specialists.
+- **Validator**: Reviews other agents' outputs for quality, safety, or compliance.
+- **Fallback**: Handles cases that other agents can't or won't.
+## Role Conflicts
+When agents' roles overlap or conflict:
+- **Priority rules**: When two agents could handle a task, which one gets it?
+- **Escalation paths**: When agents disagree, who decides?
+- **Shared resources**: When agents need the same data or tools, how is access managed?
+- **Feedback loops**: How do agents inform each other about what they've done?
+## Design Artefacts
+- Agent role cards (one per agent with all specifications)
+- Role boundary map showing where each agent's authority starts and stops
+- Interaction matrix showing which agents communicate with which
+- Authority hierarchy diagram
+- Role conflict resolution rules

package/src/skills-lib/ai-product-design/references/design-agent-orchestration/failure-recovery/SKILL.md

@@ -0,0 +1,37 @@
+---
+name: failure-recovery
+description: What happens when an agent fails — retry, fallback, escalate, or graceful degradation.
+---
+# Failure Recovery
+Agents fail. Networks time out, models hallucinate, tools error, and edge cases surprise. Failure recovery design determines whether a failure becomes a dead end or a graceful detour.
+## Failure Types in Multi-Agent Systems
+- **Agent failure**: A single agent crashes, times out, or produces invalid output
+- **Handoff failure**: Context is lost or corrupted during transfer between agents
+- **Coordination failure**: Agents conflict, deadlock, or produce inconsistent results
+- **Resource failure**: External tools, APIs, or data sources are unavailable
+- **Cascading failure**: One agent's failure causes downstream agents to fail
+## Recovery Strategies
+- **Retry**: Try the same operation again. Works for transient errors (network timeouts, rate limits). Set a retry limit to avoid infinite loops.
+- **Fallback**: Switch to an alternative approach. A different agent, a simpler method, or a cached result.
+- **Escalation**: Pass the problem to a more capable agent or to a human. Used when the failure is beyond the current agent's ability to resolve.
+- **Graceful degradation**: Deliver a partial result rather than nothing. Tell the user what worked and what didn't.
+- **Compensation**: Undo the effects of a partially completed workflow before retrying or escalating.
+## Designing Recovery Paths
+For each point in the workflow where failure is possible:
+- **What could fail?** List the failure modes
+- **What's the first recovery strategy?** Usually retry for transient errors
+- **What's the fallback?** If retry fails, what's the alternative?
+- **When do you escalate?** After how many retries or what type of failure?
+- **What does the user see?** Transparent about the failure or silently recovered?
+- **What's the worst case?** If all recovery fails, what's the graceful degradation?
+## User Experience of Failures
+- **Invisible recovery**: The system retries or falls back without the user noticing. Best for minor, quickly resolved failures.
+- **Transparent recovery**: The system tells the user something went wrong and how it's handling it. "This is taking longer than usual — trying an alternative approach."
+- **Participatory recovery**: The system asks the user to help. "I couldn't access your calendar. Can you check the connection?"
+- **Honest failure**: The system tells the user it can't complete the task and explains why. Offers alternatives.
+## Design Artefacts
+- Failure mode inventory per agent and per handoff
+- Recovery strategy specifications (retry limits, fallback paths, escalation triggers)
+- Cascading failure analysis
+- User experience specifications for each failure scenario
+- Recovery testing protocols

package/src/skills-lib/ai-product-design/references/design-agent-orchestration/handoff-protocols/SKILL.md

@@ -0,0 +1,45 @@
+---
+name: handoff-protocols
+description: Designing smooth transitions between agents and between AI and humans.
+---
+# Handoff Protocols
+A handoff is the moment one agent passes work to another — or to a human. It's where multi-agent systems most commonly fail. A dropped handoff means lost context, repeated work, or abandoned tasks.
+## Anatomy of a Handoff
+Every handoff has:
+- **Trigger**: What causes the handoff? (task completion, scope boundary, failure, user request)
+- **Source**: Who is handing off?
+- **Destination**: Who is receiving?
+- **Payload**: What information transfers? (context, partial results, user state, instructions)
+- **Acknowledgment**: How does the source know the destination received the handoff?
+- **User experience**: What does the user see during the handoff?
+## Handoff Types
+- **Sequential**: Agent A finishes, passes results to Agent B who continues. Like a relay race.
+- **Parallel fan-out**: One agent distributes subtasks to multiple agents simultaneously.
+- **Parallel fan-in**: Multiple agents' results converge back to one agent for synthesis.
+- **Escalation**: An agent can't handle the task and passes up to a more capable agent or human.
+- **Fallback**: The primary agent fails and a backup takes over.
+- **Human handoff**: AI passes work to a human for review, decision, or completion.
+## Context Transfer
+The most common handoff failure is context loss. Design what transfers:
+- **Full context**: Everything the source agent knew. Safe but potentially overwhelming.
+- **Summarised context**: Key information distilled. Efficient but risks losing important nuance.
+- **Structured context**: Predefined fields that must be populated. Consistent but rigid.
+- **Incremental context**: Only what's new since the last handoff. Efficient for ongoing collaborations.
+## Designing for the User
+The user's experience of handoffs matters:
+- **Invisible handoff**: The user doesn't know agents changed. The experience feels seamless.
+- **Transparent handoff**: The user is told a new agent is taking over and why.
+- **Participatory handoff**: The user confirms the handoff or provides additional context.
+- **User-initiated handoff**: The user explicitly requests a different agent or a human.
+## Handoff Anti-Patterns
+- **The black hole**: Work is handed off but never picked up
+- **The echo chamber**: Agents hand work back and forth without progress
+- **The context cliff**: Critical information is lost in the handoff
+- **The jarring transition**: The user's experience changes dramatically at the handoff point
+- **The silent redirect**: The user doesn't know they've been handed off and gets confused by changes
+## Design Artefacts
+- Handoff protocol specifications (trigger, source, destination, payload, acknowledgment)
+- Context transfer templates
+- Handoff sequence diagrams
+- User experience specifications for each handoff type
+- Handoff failure mode analysis

package/src/skills-lib/ai-product-design/references/design-agent-orchestration/human-in-the-loop/SKILL.md

@@ -0,0 +1,39 @@
+---
+name: human-in-the-loop
+description: Designing intervention points where humans review, approve, or redirect agent work.
+---
+# Human-in-the-Loop
+Human-in-the-loop design defines when, where, and how humans intervene in automated workflows. Too little human involvement and the system makes dangerous mistakes. Too much and you've just built an expensive notification system.
+## Intervention Types
+- **Approval gates**: The system pauses and waits for human approval before proceeding
+- **Review checkpoints**: The system presents results for human review but can continue if no objection
+- **Correction opportunities**: The system shows its work and the human can edit before it's finalised
+- **Override controls**: The human can stop, redirect, or undo the system's actions at any time
+- **Monitoring dashboards**: The human passively observes the system and intervenes only when needed
+## When to Require Human Intervention
+- **High stakes**: Actions that are expensive, irreversible, or affect many people
+- **Low confidence**: The system is uncertain about the right action
+- **Novel situations**: The input or context is outside the system's training distribution
+- **Ethical judgments**: Decisions that require moral reasoning or value trade-offs
+- **Legal requirements**: Regulatory or compliance requirements mandate human review
+- **User request**: The user explicitly asks for human involvement
+## Designing Intervention Points
+For each intervention point:
+- **Trigger**: What causes the intervention? (confidence threshold, stakes level, user request, policy requirement)
+- **Presentation**: What does the human see? (summary, full context, recommendations, options)
+- **Time constraint**: How quickly must the human respond? What happens if they don't?
+- **Decision options**: What can the human do? (approve, reject, edit, escalate, defer)
+- **Feedback integration**: How does the human's decision feed back into the system?
+## Avoiding Human Bottlenecks
+Human intervention is expensive and slow. Design to minimise unnecessary intervention:
+- **Graduated autonomy**: Start with more human oversight, reduce as the system proves reliable
+- **Batch review**: Group similar decisions for efficient human processing
+- **Smart routing**: Send interventions to the right human based on expertise and availability
+- **Default actions**: If the human doesn't respond within a time window, take a safe default action
+- **Learning from interventions**: Use human decisions to improve the system so it needs less intervention over time
+## Design Artefacts
+- Intervention point map (where in the workflow, what triggers each)
+- Intervention interface designs (what the human sees and can do)
+- Time constraint specifications
+- Graduated autonomy plans
+- Intervention analytics specifications (tracking how often, why, and what humans decide)

package/src/skills-lib/ai-product-design/references/design-agent-orchestration/observability-design/SKILL.md

@@ -0,0 +1,48 @@
+---
+name: observability-design
+description: Making multi-agent workflows visible and debuggable for designers and developers.
+---
+# Observability Design
+You can't improve what you can't see. Observability design makes the internal workings of multi-agent systems visible — so designers can understand user experience problems, developers can debug failures, and teams can improve the system over time.
+## What to Make Observable
+- **Workflow execution**: Which agents were involved, in what order, with what results
+- **Decision points**: What decisions were made, what alternatives were considered, why one was chosen
+- **Handoff details**: What context transferred between agents, was anything lost
+- **Timing**: How long each agent took, where bottlenecks occur
+- **Failures**: What failed, how it was recovered, what the user experienced
+- **Quality signals**: Output quality scores, user satisfaction signals, task success markers
+## Observability for Different Audiences
+**For designers:**
+- User journey view: What did the user experience across the whole workflow?
+- Pain point identification: Where did users struggle, abandon, or express frustration?
+- Quality patterns: Which outputs are high and low quality, and why?
+**For developers:**
+- Execution traces: Step-by-step log of agent actions
+- Error logs: What failed and where
+- Performance metrics: Latency, throughput, resource usage
+**For product managers:**
+- Usage patterns: Which workflows are used most, which are abandoned
+- Success metrics: Task completion rates, user satisfaction trends
+- Cost analysis: Resource consumption per workflow
+**For users (optional):**
+- Progress indicators: Where is the system in the workflow?
+- Agent transparency: Which agent is handling their request?
+- Audit trails: What the system did on their behalf
+## Designing Observability Interfaces
+- **Dashboards**: Real-time and historical views of system health and performance
+- **Trace viewers**: Detailed step-by-step views of individual workflow executions
+- **Alert systems**: Notifications when metrics exceed thresholds
+- **Search and filter**: Ability to find specific executions by criteria
+- **Comparison tools**: Compare performance across time periods, versions, or cohorts
+## Observability Without Overload
+Too much data is as bad as too little:
+- **Layered detail**: Start with high-level summary, drill down on demand
+- **Smart defaults**: Show the most important information first
+- **Anomaly highlighting**: Surface unusual patterns automatically
+- **Contextual views**: Different views for different questions
+## Design Artefacts
+- Observability architecture diagrams
+- Dashboard specifications per audience
+- Trace schema definitions
+- Alert threshold configurations
+- Observability tool requirements

package/src/skills-lib/ai-product-design/references/design-agent-orchestration/state-management/SKILL.md

@@ -0,0 +1,95 @@
+---
+name: state-management
+description: Managing shared context, memory, and state across multiple agents.
+---
+# State Management
+
+In a multi-agent system, state is the shared truth about what's happened, what's in progress, and what's been decided. Without state management, agents work with stale or conflicting information — and the user pays the cost in repeated questions, contradictory answers, and lost progress.
+
+State management is the *plumbing* skill of multi-agent design. Get it wrong and every other skill in this plugin gets harder.
+
+## Types of state
+
+- **Task state**: where the overall task is in its lifecycle. Which subtasks are complete, in progress, or pending.
+- **Context state**: what each agent knows. What has been shared, summarised, or dropped.
+- **User state**: preferences, history, and current emotional state.
+- **Decision state**: decisions made, options considered, options rejected (and why).
+- **Error state**: what has failed, been retried, been escalated.
+
+## State architecture patterns
+
+- **Centralised state**: one shared store all agents read from and write to. Simple, debuggable. Bottleneck risk at scale.
+- **Distributed state**: each agent maintains its own state and syncs with others. Flexible. Consistency risk.
+- **Event-sourced state**: state is built from a log of events. Every change is recorded. Auditable. Complex.
+- **Blackboard pattern**: shared workspace where agents post results and read others' contributions. Good for collaborative problem-solving.
+
+## Designing state for users
+
+Users have expectations about what the system remembers:
+
+- **Within-session state**: everything said in this conversation should persist consistently
+- **Cross-session state**: preferences, decisions, and context from past sessions should carry forward
+- **Cross-agent state**: if one agent learned something, other agents should know
+- **User-controlled state**: users should be able to see, edit, and clear what the system remembers
+
+## Decision rules
+
+- **Default to centralised state.** Reach for distributed only when measured cross-agent latency is genuinely the bottleneck. Most teams choose distributed prematurely and pay in consistency bugs forever.
+- **If a piece of state lives in a single agent's working memory, treat it as lost.** Memory across model invocations is unreliable; promote anything that needs to persist to the shared store.
+- **Cross-session state requires explicit consent per category.** "Remember preferences" ≠ "remember what we discussed". Granularity is the design constraint, not a nice-to-have.
+- **For state conflicts, prefer detection over silent merging.** A surfaced conflict the user resolves is recoverable; a silently merged inconsistency is invisible damage.
+- **State a user can't see, they can't trust.** Any state used to personalise behaviour must be visible somewhere the user can find within ~30 s of UI navigation.
+
+## Anti-patterns
+
+- **The stale-context bug**: an agent acts on state that's been superseded by another agent. Hard to detect because the agent looks correct in isolation. Mitigated by versioning every read.
+- **The one-source-of-truth fight**: two agents both claim authority over the same state slice. Resolution rules are vague. Last write wins, intermittently.
+- **Implicit state**: state lives in an agent's prompt context rather than the shared store. Lost on restart, untransferable, undebuggable.
+- **State sprawl**: schema grows unbounded as features accrete. After six months nobody knows what's used, what's dead, or what's load-bearing.
+- **Sync deferral**: writes are batched for efficiency, then dropped on failure. The user sees "saved" but the state never made it.
+- **Invisible personalisation**: state shapes behaviour the user can't see or override — drift toward manipulation.
+
+## When not to use this
+
+- **Single-agent products** — the model's context window *is* the state. Reach for `context-window-design` instead.
+- **Stateless transactional features** (one-shot completions, image generation calls) — explicit state architecture adds overhead without benefit.
+- **Prototype phase** — premature state architecture freezes choices that should still be loose. Use the simplest possible store, document decisions in `decision-state` only.
+
+## See also
+
+- `handoff-protocols` — context transfer between agents *uses* the state architecture; design them together.
+- `observability-design` — state mutations are the most useful traces. Design what's logged at the same time as what's stored.
+- `consent-and-agency` — cross-session and cross-agent state is consent-laden by default.
+- `task-decomposition` — task-state schema is determined by the decomposition shape; co-design.
+
+## Design Artefacts
+
+- State architecture diagrams (centralised / distributed / event-sourced / blackboard)
+- State schema definitions (what's stored, where, by whom, with what TTL)
+- State lifecycle specifications (creation, update, archival, deletion)
+- Conflict resolution rules per state slice
+- User-facing state visibility and control designs
+
+Worked example — task state for a multi-agent customer support flow:
+
+```
+{
+  "task_id": "tk_8b2",
+  "owner_agent": "router",                  // who currently holds the task
+  "status": "in_progress | escalated | done",
+  "subtasks": [
+    {"id": "verify_account", "status": "done", "result_ref": "ctx_a91"},
+    {"id": "check_refund_eligibility", "status": "in_progress", "owner": "billing_agent"}
+  ],
+  "user_state_ref": "us_482",               // pointer, not embedded copy
+  "decisions": [
+    {"at": "...", "by": "router", "choice": "billing_agent", "rejected": ["faq_agent"], "why": "intent: refund"}
+  ],
+  "errors": [],
+  "version": 7                              // increments on every write; readers check before acting
+}
+```
+
+The version field is the simplest defence against the stale-context bug. The `decisions` log with `rejected` makes the routing legible to humans during incident review.
+
+Adapted from work on shared mental models in multi-agent systems and distributed-systems consistency literature (Lamport on logical clocks; the actor model on isolated state).

package/src/skills-lib/ai-product-design/references/design-agent-orchestration/task-decomposition/SKILL.md

@@ -0,0 +1,36 @@
+---
+name: task-decomposition
+description: Breaking complex user goals into subtasks that agents can handle.
+---
+# Task Decomposition
+Users come with goals, not subtasks. Task decomposition is how a multi-agent system breaks a complex user goal into pieces that individual agents can handle — and then reassembles the results into something coherent.
+## Decomposition Strategies
+- **Sequential decomposition**: Break the goal into ordered steps. Step 1 must complete before Step 2 starts.
+- **Parallel decomposition**: Break the goal into independent parts that can be worked on simultaneously.
+- **Hierarchical decomposition**: Break the goal into sub-goals, then break each sub-goal into tasks.
+- **Conditional decomposition**: The next step depends on the result of the current step. Different results lead to different paths.
+- **Iterative decomposition**: Start with a rough version, then refine through multiple passes.
+## Designing Decomposition Rules
+For each type of user goal the system handles:
+- **What's the entry point?** How does the system receive the goal?
+- **What are the subtasks?** List all possible subtasks for this goal type.
+- **What are the dependencies?** Which subtasks depend on others' outputs?
+- **What's the critical path?** Which sequence of subtasks determines the minimum completion time?
+- **What can be parallelised?** Which subtasks can run simultaneously?
+- **What's the reassembly logic?** How do subtask results combine into the final output?
+## Granularity
+How finely to decompose matters:
+- **Too coarse**: Single agents get tasks that are too complex, leading to lower quality
+- **Too fine**: Overhead from handoffs exceeds the benefit of specialisation
+- **Just right**: Each subtask matches one agent's sweet spot in terms of scope and complexity
+## Handling Ambiguity
+User goals are often ambiguous. The system needs to:
+- **Clarify before decomposing**: Ask the user to specify when the goal is unclear
+- **Decompose tentatively**: Start with a plan and adjust as information emerges
+- **Recompose when needed**: If decomposition was wrong, restructure without starting over
+## Design Artefacts
+- Task decomposition trees for each goal type
+- Dependency maps showing subtask relationships
+- Parallelisation specifications
+- Reassembly logic definitions
+- Ambiguity handling protocols

package/src/skills-lib/ai-product-design/references/evaluation/comparative-evaluation/SKILL.md

@@ -0,0 +1,39 @@
+---
+name: comparative-evaluation
+description: A/B testing, side-by-side comparison, and preference ranking for AI outputs.
+---
+# Comparative Evaluation
+Absolute quality scores are useful but limited. Comparative evaluation — putting outputs side by side and asking which is better — often reveals quality differences that rubrics miss.
+## Comparison Methods
+- **A/B testing**: Show different users different versions and compare outcomes
+- **Side-by-side evaluation**: Show evaluators two outputs for the same input and ask which is better
+- **Preference ranking**: Show evaluators multiple outputs and rank them from best to worst
+- **Paired comparison**: Compare every pair of options to build a complete ranking
+- **Elo rating**: Use tournament-style comparisons to develop continuous quality scores
+## Designing A/B Tests for AI
+A/B testing AI is different from A/B testing UI:
+- **Variance is high**: The same prompt can produce different outputs, so you need more samples
+- **Context matters**: The same change might help for one task and hurt for another
+- **Metrics lag**: AI quality changes may take time to show up in user behavior
+- **Interaction effects**: A change to one part of the conversation affects all subsequent parts
+Design A/B tests with:
+- Sufficient sample sizes to account for output variance
+- Segmentation by task type and user experience level
+- Multiple metrics (don't optimise for one at the expense of others)
+- Guardrails to catch severe quality regressions quickly
+## Side-by-Side Evaluation Design
+For human evaluation of AI outputs:
+- **Blind evaluation**: Evaluators shouldn't know which version is which
+- **Consistent inputs**: Compare outputs generated from the same input
+- **Structured criteria**: Give evaluators specific dimensions to compare on, not just "which is better"
+- **Multiple evaluators**: Use at least 3 evaluators per comparison for reliability
+- **Diverse inputs**: Test across a representative sample of real user inputs
+## When to Use Comparative vs. Absolute Evaluation
+- **Comparative**: Best for choosing between alternatives, detecting subtle quality differences, and model selection
+- **Absolute**: Best for measuring against a standard, tracking progress over time, and certification
+## Design Artefacts
+- A/B test design templates
+- Side-by-side evaluation protocols
+- Evaluator instructions and rubrics
+- Sample size calculators for AI experiments
+- Comparison result analysis frameworks

package/src/skills-lib/ai-product-design/references/evaluation/failure-taxonomy/SKILL.md

@@ -0,0 +1,47 @@
+---
+name: failure-taxonomy
+description: Classifying AI failures — hallucination, refusal, irrelevance, tone mismatch, latency.
+---
+# Failure Taxonomy
+Not all AI failures are the same. A hallucination is different from a refusal, which is different from a tone mismatch. A failure taxonomy classifies failure types so teams can track, prioritise, and address them systematically.
+## Failure Categories
+**Content Failures:**
+- **Hallucination**: The AI presents false information as fact
+- **Inaccuracy**: The AI gets details wrong (dates, numbers, names)
+- **Incompleteness**: The AI misses important information
+- **Irrelevance**: The AI's response doesn't address the user's actual question
+- **Contradiction**: The AI contradicts itself within or across responses
+**Behavioral Failures:**
+- **Inappropriate refusal**: The AI refuses a reasonable request
+- **Missing refusal**: The AI fulfils a request it should have declined
+- **Tone mismatch**: The AI's tone is wrong for the context
+- **Persona break**: The AI drops out of its defined persona
+- **Over-generation**: The AI produces far more than needed
+**Technical Failures:**
+- **Latency**: Response takes too long
+- **Truncation**: Response is cut off
+- **Format errors**: Output is in the wrong format or structure
+- **Tool failures**: The AI attempts to use a tool and fails
+- **Context loss**: The AI loses track of conversation history
+**Safety Failures:**
+- **Harmful content**: The AI generates content that could cause harm
+- **Privacy violation**: The AI reveals sensitive information
+- **Bias manifestation**: The AI's output shows bias against a group
+- **Manipulation**: The AI's output could be used to deceive or manipulate
+## Severity Levels
+- **Critical**: Causes harm or creates serious trust violation. Requires immediate fix.
+- **High**: Significantly degrades user experience or task success. Fix within days.
+- **Medium**: Noticeable quality issue that users can work around. Fix within weeks.
+- **Low**: Minor quality issue. Track and batch with other fixes.
+## Using the Taxonomy
+- **Logging**: Classify every detected failure by type and severity
+- **Trending**: Track failure type frequency over time
+- **Prioritisation**: Address highest-severity, highest-frequency failures first
+- **Root cause analysis**: Group failures by type to identify systemic causes
+- **Prevention**: Use failure patterns to inform guardrail design and prompt improvements
+## Design Artefacts
+- Failure taxonomy reference document
+- Failure logging templates
+- Severity classification rubric
+- Failure trend dashboards
+- Root cause analysis protocols

package/src/skills-lib/ai-product-design/references/evaluation/heuristic-evaluation-ai/SKILL.md

@@ -0,0 +1,46 @@
+---
+name: heuristic-evaluation-ai
+description: Adapting Nielsen's heuristics and new AI-specific heuristics for AI interfaces.
+---
+# Heuristic Evaluation for AI
+Nielsen's 10 usability heuristics were designed for traditional software. AI products need adapted heuristics that address the unique challenges of probabilistic, generative, and conversational systems.
+## Classic Heuristics, Adapted for AI
+**1. Visibility of system status**
+AI adaptation: The user should always know what the AI is doing, what it's working with, and how confident it is. Progress indicators for generation. Transparency about data sources.
+**2. Match between system and real world**
+AI adaptation: The AI should use language and concepts the user understands. Don't expose model internals. Frame capabilities in terms of user tasks, not technical features.
+**3. User control and freedom**
+AI adaptation: Users must be able to stop generation, undo AI actions, edit outputs, and override suggestions. AI autonomy should always have an exit.
+**4. Consistency and standards**
+AI adaptation: The AI should behave consistently across similar requests. Same input type should produce same output format. Persona should be stable.
+**5. Error prevention**
+AI adaptation: Design prompts and interfaces that guide users toward effective interactions. Suggest clarifications before producing low-quality output.
+**6. Recognition rather than recall**
+AI adaptation: Show users what the AI can do rather than requiring them to discover commands. Surface relevant capabilities contextually.
+**7. Flexibility and efficiency of use**
+AI adaptation: Support both novice (guided) and expert (shortcut) interaction modes. Power users should be able to customise AI behavior.
+**8. Aesthetic and minimalist design**
+AI adaptation: AI outputs should be concise and well-structured. Don't pad responses with unnecessary caveats or filler.
+**9. Help users recognise, diagnose, and recover from errors**
+AI adaptation: When the AI fails, explain what went wrong in user terms, not technical terms. Offer clear recovery paths.
+**10. Help and documentation**
+AI adaptation: Provide contextual guidance on how to interact with the AI effectively. Teach prompting skills through the interface.
+## AI-Specific Heuristics
+Beyond the classic 10, AI products need evaluation against:
+- **Calibrated trust**: Does the interface help users trust the AI appropriately — neither too much nor too little?
+- **Graceful degradation**: When the AI can't fully help, does it partially help rather than failing completely?
+- **Feedback effectiveness**: Can users correct the AI easily, and does the AI adapt?
+- **Transparency of limitations**: Are the AI's boundaries clear before the user hits them?
+- **Appropriate autonomy**: Does the AI take the right amount of initiative for the task and context?
+## Running an AI Heuristic Evaluation
+1. Select 3-5 evaluators with AI product experience
+2. Define the scope (which features, which user tasks)
+3. Each evaluator independently works through the heuristics
+4. Capture issues with severity ratings
+5. Consolidate findings and prioritise
+## Design Artefacts
+- AI heuristic checklist (adapted classics + AI-specific)
+- Evaluation protocol and scoring rubric
+- Issue severity classification guide
+- Heuristic evaluation report template
+- Prioritised findings matrix

package/src/skills-lib/ai-product-design/references/evaluation/longitudinal-measurement/SKILL.md

@@ -0,0 +1,38 @@
+---
+name: longitudinal-measurement
+description: Tracking AI product quality over time — drift, degradation, and improvement.
+---
+# Longitudinal Measurement
+AI products change over time — models get updated, usage patterns shift, and quality can drift without anyone noticing. Longitudinal measurement is how you track quality across time and catch degradation before users do.
+## What Changes Over Time
+- **Model updates**: New model versions may improve some capabilities and regress others
+- **Prompt drift**: System prompts accumulate edits that may interact in unexpected ways
+- **Usage evolution**: Users discover new use cases that weren't tested for
+- **Data drift**: The real-world inputs diverge from what was tested
+- **Expectation drift**: Users' expectations change as they become more experienced
+## What to Measure Longitudinally
+- **Quality scores**: Track rubric scores on a consistent test set over time
+- **Task success rates**: Monitor whether users are completing tasks at the same rate
+- **Satisfaction signals**: Track trends in explicit and implicit satisfaction
+- **Error rates**: Monitor failure frequency and type distribution
+- **Latency**: Response time changes can indicate degradation
+- **Engagement patterns**: Changes in usage frequency, depth, and breadth
+## Measurement Infrastructure
+- **Golden test sets**: A fixed set of inputs evaluated regularly to detect quality changes
+- **Automated evaluation**: Run golden test sets automatically on a schedule
+- **Dashboards**: Visualise trends and set alerts for significant changes
+- **Regression detection**: Statistical methods to distinguish real changes from noise
+- **User cohort tracking**: Follow specific user groups over time
+## Responding to Drift
+When measurements show drift:
+1. **Detect**: Automated alerts flag significant changes
+2. **Diagnose**: Was it a model update, prompt change, data shift, or usage change?
+3. **Assess**: Is the drift harmful, neutral, or actually an improvement?
+4. **Act**: Adjust prompts, revert changes, update guardrails, or accept the new baseline
+5. **Verify**: Confirm the fix worked and set the new baseline
+## Design Artefacts
+- Longitudinal measurement plan
+- Golden test set specifications
+- Quality trend dashboards
+- Drift detection alert configurations
+- Response protocols for detected drift

package/src/skills-lib/ai-product-design/references/evaluation/output-quality-rubrics/SKILL.md

@@ -0,0 +1,41 @@
+---
+name: output-quality-rubrics
+description: Defining what "good" looks like for AI outputs — accuracy, relevance, helpfulness.
+---
+# Output Quality Rubrics
+Without a rubric, quality evaluation is subjective and inconsistent. A rubric defines what "good" means in concrete, measurable terms — so different evaluators reach the same conclusions.
+## Core Quality Dimensions
+- **Accuracy**: Is the information correct? Are claims verifiable? Are there hallucinations?
+- **Relevance**: Does the output address what the user actually asked? Is everything included necessary?
+- **Completeness**: Does the output cover everything needed? Are there gaps?
+- **Helpfulness**: Can the user actually use this output to accomplish their goal?
+- **Clarity**: Is the output easy to understand? Is it well-structured?
+- **Tone appropriateness**: Does the output match the expected tone for the context?
+- **Safety**: Is the output free from harmful, biased, or inappropriate content?
+## Building a Rubric
+For each dimension, define a scale:
+**Example — Accuracy (1-5):**
+- 5: All claims are verifiable and correct. No hallucinations.
+- 4: Minor inaccuracies that don't affect usefulness. No hallucinations.
+- 3: Some inaccuracies that could mislead if not caught. No dangerous hallucinations.
+- 2: Significant inaccuracies. User would need to verify most claims.
+- 1: Major hallucinations or factually wrong information presented confidently.
+## Weighting Dimensions
+Not all dimensions matter equally for every use case:
+- A medical AI weights accuracy and safety highest
+- A creative writing AI weights helpfulness and tone highest
+- A coding AI weights accuracy and completeness highest
+- A customer service AI weights tone and helpfulness highest
+Define weights when creating the rubric. Make the priorities explicit.
+## Rubric Calibration
+A rubric is only useful if evaluators use it consistently:
+- **Anchor examples**: Provide sample outputs at each score level
+- **Calibration sessions**: Have multiple evaluators score the same outputs and discuss disagreements
+- **Inter-rater reliability**: Measure agreement between evaluators and refine the rubric until agreement is high
+- **Edge case guidance**: Document how to score ambiguous cases
+## Design Artefacts
+- Scoring rubric with dimension definitions and scales
+- Anchor examples at each score level
+- Dimension weighting specifications per use case
+- Calibration session protocols
+- Scoring templates and checklists

package/src/skills-lib/ai-product-design/references/evaluation/task-success-metrics/SKILL.md

@@ -0,0 +1,35 @@
+---
+name: task-success-metrics
+description: Measuring whether the AI actually helped users accomplish their goals.
+---
+# Task Success Metrics
+Output quality doesn't guarantee task success. The AI might produce a beautiful response that doesn't actually help the user do what they came to do. Task success metrics measure the end-to-end outcome.
+## Defining Task Success
+For each user task, define:
+- **What does success look like?** The user completed their goal (sent the email, found the information, finished the design)
+- **What are the success criteria?** Specific, observable conditions that indicate the task is done
+- **What's the time expectation?** How long should this task take with AI assistance vs. without?
+- **What's the quality bar?** Not just done, but done well enough
+## Task Success Metrics
+- **Task completion rate**: Percentage of users who complete the task (not just get a response)
+- **Time to completion**: How long from first input to task done
+- **Turns to completion**: How many back-and-forth exchanges needed
+- **First-attempt success rate**: Did the AI's first response accomplish the task, or did it require iteration?
+- **Intervention rate**: How often did the user need to correct, redirect, or override the AI?
+- **Abandonment rate**: How often did users give up before completing the task?
+## Measuring Task Success
+- **Direct measurement**: Track task completion through product analytics (user clicked "done", saved the output, moved to next step)
+- **Inferred measurement**: Infer success from proxy signals (session length, return rate, output edits)
+- **Self-reported measurement**: Ask users whether the AI helped them accomplish their goal
+- **Comparative measurement**: Compare task success with AI vs. without AI, or with version A vs. version B
+## Task Success vs. Output Quality
+These can diverge:
+- **High output quality, low task success**: The AI's answer is well-written but doesn't address the real need
+- **Low output quality, high task success**: The AI's answer is rough but gives the user exactly what they needed
+- **Both matter**: Track both and investigate when they diverge
+## Design Artefacts
+- Task success definitions per key user task
+- Metrics framework with measurement methods
+- Success criteria specifications
+- Baseline measurements (before AI, or current version)
+- Task success dashboard specifications