loki-mode 5.1.3 → 5.2.0

package/skills/model-selection.md

@@ -176,3 +176,234 @@ Task(model="haiku", description="Fix import in utils.py", prompt="...")
 # Complex tasks -> Supervisor orchestration
 Task(description="Implement user authentication with OAuth", prompt="...")
 ```
+
+## Tiered Agent Escalation Triggers
+
+*Inspired by [oh-my-claudecode](https://github.com/Yeachan-Heo/oh-my-claudecode) tiered architecture.*
+
+**Tier Mapping:** LOW=fast=Haiku, MEDIUM=development=Sonnet, HIGH=planning=Opus. This section uses LOW/MEDIUM/HIGH for clarity in escalation context.
+
+Explicit signals that determine when to escalate from one tier to another. These triggers enable automatic tier selection based on task characteristics and runtime conditions.
+
+### Tier Definitions
+
+| Tier | Model | Cost | Speed | Use Case |
+|------|-------|------|-------|----------|
+| **LOW** | Haiku | Lowest | Fastest | Simple, repetitive, well-defined tasks |
+| **MEDIUM** | Sonnet | Medium | Balanced | Complex implementation, testing, debugging |
+| **HIGH** | Opus | Highest | Slowest | Critical decisions, architecture, security |
+
+### Automatic Escalation Triggers
+
+#### LOW -> MEDIUM Escalation
+
+Escalate from Haiku to Sonnet when:
+
+| Trigger | Threshold | Rationale |
+|---------|-----------|-----------|
+| Error count | > 2 consecutive failures | Haiku struggling with complexity |
+| File count | > 5 files modified | Cross-cutting changes need context |
+| Lines changed | > 200 lines | Large changes need careful review |
+| Test failures | > 3 failing tests | Need deeper debugging |
+| Dependency changes | Any package.json/requirements.txt | Dependency resolution is complex |
+| Retry attempts | > 1 retry on same task | Task too complex for current tier |
+
+```python
+# Example: Auto-escalate on repeated failures
+if task.error_count > 2:
+    task.escalate(to="sonnet", reason="Multiple failures indicate complexity")
+```
+
+#### MEDIUM -> HIGH Escalation
+
+Escalate from Sonnet to Opus when:
+
+| Trigger | Threshold | Rationale |
+|---------|-----------|-----------|
+| Error count | > 3 consecutive failures | Even Sonnet struggling |
+| Complexity score | > 15 cyclomatic | Highly complex logic needs planning |
+| Architecture files | Any changes to core/* | Architecture decisions are critical |
+| Breaking changes | API contract modifications | Need careful impact analysis |
+| Performance issues | > 2x baseline regression | Need optimization strategy |
+| Integration scope | > 3 services affected | Cross-service coordination |
+
+```python
+# Example: Auto-escalate for architecture changes
+if "core/" in modified_files or "architecture" in task.tags:
+    task.escalate(to="opus", reason="Architecture changes require planning tier")
+```
+
+#### HIGH -> HUMAN Escalation (Terminal)
+
+When even Opus fails, escalate to human intervention:
+
+| Trigger | Threshold | Action |
+|---------|-----------|--------|
+| Error count | > 5 consecutive at HIGH tier | Create `.loki/signals/HUMAN_REVIEW_NEEDED` |
+| Ambiguous requirements | Cannot determine correct behavior | Create signal with specific questions |
+| External dependencies | Blocked on third-party API/service | Document blocker, pause task |
+| Ethical concerns | Task may violate principles | Halt immediately, document concern |
+
+```python
+# Terminal escalation - no automated recovery
+if task.tier == "opus" and task.error_count > 5:
+    create_signal(".loki/signals/HUMAN_REVIEW_NEEDED", {
+        "task_id": task.id,
+        "reason": "5+ failures at planning tier",
+        "attempts": task.attempt_log,
+        "recommendation": "Manual investigation required"
+    })
+    task.status = "blocked_on_human"
+```
+
+### Threshold Rationale
+
+Why these specific thresholds? Each value is grounded in research or proven heuristics:
+
+| Threshold | Value | Justification |
+|-----------|-------|---------------|
+| **Error count > 2** (LOW->MEDIUM) | 3 attempts | "Two strikes" principle: first failure may be transient (network, timeout), second suggests genuine complexity. Third attempt warrants escalation to capable tier. |
+| **Error count > 3** (MEDIUM->HIGH) | 4 attempts | Sonnet has significantly more capability than Haiku, so allow one additional attempt before expensive Opus escalation. Balances cost vs. success rate. |
+| **Error count > 5** (HIGH->HUMAN) | 6 attempts | Planning tier (Opus) exhausted all automated reasoning options. Further attempts unlikely to succeed; human judgment required. |
+| **File count > 5** | 6+ files | Cross-cutting changes affecting 5+ files require holistic understanding of system interactions. Research on code review effectiveness shows reviewer accuracy drops with multi-file changes. |
+| **Lines changed > 200** | 200 LOC | Studies on code review effectiveness (Cisco, SmartBear) show review quality degrades significantly above 200-400 LOC. Microsoft's internal research suggests 200 LOC as optimal review size. |
+| **Cyclomatic complexity > 15** | McCabe threshold | Industry standard since McCabe (1976). NIST considers >15 "high risk." Many static analysis tools default to this threshold. |
+| **Test failures > 3** | 4+ failures | Distinguishes isolated flakiness from systemic issues. Single test failure may be flaky; 3+ indicates deeper problems requiring debugging capability. |
+| **Retry attempts > 1** | 2+ retries | First retry accounts for transient issues. Second retry at same tier signals fundamental mismatch between task complexity and model capability. |
+| **5+ successful tasks** (de-escalation) | Success streak | Sustained success indicates task complexity has reduced or model has adapted. Safe to try lower-cost tier with quick re-escalation if needed. |
+
+**References:**
+- McCabe, T.J. (1976). "A Complexity Measure." IEEE Transactions on Software Engineering.
+- Cisco Code Review Study: Optimal review size 200-400 LOC for defect detection.
+- SmartBear "Best Kept Secrets of Peer Code Review": Review effectiveness drops 50% above 400 LOC.
+
+### Always-HIGH Triggers (No Escalation Path)
+
+These tasks ALWAYS start at HIGH tier (Opus):
+
+| Category | Examples | Rationale |
+|----------|----------|-----------|
+| **Security** | Auth, encryption, secrets, RBAC | Security cannot be compromised |
+| **Architecture** | System design, service boundaries, data models | Foundation decisions |
+| **Breaking Changes** | API versioning, schema migrations, deprecations | High blast radius |
+| **Production Incidents** | Outage response, data corruption, rollback | Critical impact |
+| **Compliance** | GDPR, HIPAA, SOC2 implementations | Regulatory requirements |
+| **Cost Decisions** | Infrastructure scaling, vendor selection | Financial impact |
+
+```python
+# Example: Security tasks always use Opus
+ALWAYS_HIGH_PATTERNS = [
+    r"(auth|security|encrypt|secret|credential|token|password)",
+    r"(architecture|system.design|schema.migration)",
+    r"(production|incident|outage|rollback)",
+    r"(compliance|gdpr|hipaa|soc2|pci)",
+]
+
+if any(re.search(p, task.description, re.I) for p in ALWAYS_HIGH_PATTERNS):
+    task.tier = "HIGH"  # No escalation, start at Opus
+```
+
+### De-escalation Triggers (Cost Optimization)
+
+De-escalate to lower tier when conditions improve:
+
+| Trigger | Action | Rationale |
+|---------|--------|-----------|
+| 5+ successful tasks at tier | Consider de-escalation | Complexity resolved |
+| Single-file changes | Use LOW for isolated fixes | Simple scope |
+| Test-only changes | Use LOW for unit tests | Well-defined output |
+| Documentation | Use LOW for docs/comments | Low risk |
+
+```python
+# Example: De-escalate when task becomes routine
+if task.success_streak >= 5 and task.scope == "single_file":
+    task.deescalate(to="haiku", reason="Task scope is simple and stable")
+```
+
+### Escalation Flow Diagram
+
+```
+                    +------------------+
+                    |   Task Arrives   |
+                    +--------+---------+
+                             |
+                    +--------v---------+
+                    | Check ALWAYS_HIGH |
+                    +--------+---------+
+                             |
+              +--------------+--------------+
+              |                             |
+        [matches]                     [no match]
+              |                             |
+     +--------v--------+           +--------v--------+
+     |   START: HIGH   |           |   START: LOW    |
+     |    (Opus)       |           |    (Haiku)      |
+     +-----------------+           +--------+--------+
+                                            |
+                                   +--------v--------+
+                                   |  Execute Task   |
+                                   +--------+--------+
+                                            |
+                              +-------------+-------------+
+                              |                           |
+                        [success]                   [failure]
+                              |                           |
+                    +---------v---------+       +---------v---------+
+                    | Continue at tier  |       | Check thresholds  |
+                    +-------------------+       +---------+---------+
+                                                          |
+                                              +-----------+-----------+
+                                              |                       |
+                                        [under limit]           [over limit]
+                                              |                       |
+                                     +--------v--------+     +--------v--------+
+                                     |  Retry at tier  |     | ESCALATE tier   |
+                                     +-----------------+     +-----------------+
+```
+
+### Implementation in Provider Context
+
+For Claude (full features):
+```python
+# Task tool with tier awareness
+Task(
+    model=determine_tier(task),  # Returns "opus", "sonnet", or "haiku"
+    description=task.description,
+    prompt=task.prompt,
+    metadata={"escalation_count": task.escalation_count}
+)
+```
+
+For Codex/Gemini (degraded mode):
+```python
+# Map tiers to effort/thinking levels
+TIER_MAPPING = {
+    "codex": {"HIGH": "xhigh", "MEDIUM": "high", "LOW": "low"},
+    "gemini": {"HIGH": "high", "MEDIUM": "medium", "LOW": "low"},
+}
+effort_level = TIER_MAPPING[provider][determine_tier(task)]
+```
+
+### Metrics for Tier Optimization
+
+Track these metrics to tune escalation thresholds:
+
+| Metric | Purpose | Target |
+|--------|---------|--------|
+| Escalation rate | How often tasks escalate | < 20% |
+| First-tier success | Tasks completed without escalation | > 80% |
+| Cost per task | Average token cost by tier | Minimize |
+| Time to completion | Including escalation delays | Minimize |
+| Quality score | Post-completion review score | > 4.0/5.0 |
+
+```python
+# Log escalation events for analysis
+log_escalation(
+    task_id=task.id,
+    from_tier=current_tier,
+    to_tier=new_tier,
+    trigger=trigger_reason,
+    error_count=task.error_count,
+    timestamp=now()
+)
+```

package/skills/quality-gates.md

@@ -21,6 +21,190 @@
 
 ---
 
+## Chain-of-Verification (CoVe) Protocol
+
+**Research:** arXiv 2309.11495 - "Chain-of-Verification Reduces Hallucination in Large Language Models"
+
+### Core Insight
+
+Factored, decoupled verification mitigates error propagation. Each verification is computed independently without access to the original response, preventing the model from rationalizing its initial mistakes.
+
+### The 4-Step CoVe Process
+
+```
+Step 1: DRAFT          Step 2: PLAN           Step 3: EXECUTE        Step 4: REVISE
++-------------+        +---------------+      +-----------------+    +----------------+
+| Generate    |  --->  | Self-generate |  --> | Answer each     | -> | Incorporate    |
+| initial     |        | verification  |      | question        |    | corrections    |
+| response    |        | questions     |      | INDEPENDENTLY   |    | into final     |
++-------------+        +---------------+      +-----------------+    +----------------+
+                       "What claims     |      (factored exec)
+                        did I make?     |      No access to
+                        What could be   |      original response
+                        wrong?"
+```
+
+### Step-by-Step Implementation
+
+**Step 1: Draft Initial Response**
+```yaml
+draft_phase:
+  action: "Generate initial code/response"
+  model: "sonnet"  # Fast drafting
+  output: "baseline_response"
+```
+
+**Step 2: Plan Verification Questions**
+```yaml
+verification_planning:
+  prompt: |
+    Review the response above. Generate verification questions:
+    1. What factual claims did I make?
+    2. What assumptions did I rely on?
+    3. What could be incorrect or incomplete?
+    4. What edge cases did I miss?
+  output: "verification_questions[]"
+```
+
+**Step 3: Execute Verifications INDEPENDENTLY (Critical)**
+```yaml
+factored_execution:
+  critical: "Each verification runs in isolation"
+  rule: "Verifier has NO access to original response"
+
+  # Launch in parallel - each is independent
+  verifications:
+    - question: "Does the function handle null inputs?"
+      context: "Function signature and spec only"  # NOT the implementation
+      verifier: "sonnet"
+    - question: "Is the SQL query injection-safe?"
+      context: "Query requirements only"
+      verifier: "sonnet"
+    - question: "Does the API match the documented spec?"
+      context: "API spec only"
+      verifier: "sonnet"
+```
+
+**Step 4: Generate Final Verified Response**
+```yaml
+revision_phase:
+  inputs:
+    - original_response
+    - verification_results[]
+  action: "Revise response incorporating all corrections"
+  output: "verified_response"
+```
+
+### Factor+Revise Variant (Longform Code Generation)
+
+For complex code generation, use the enhanced Factor+Revise pattern. The key difference from basic Factored execution is an **explicit cross-check step** where the model compares original claims against verification results before revision.
+
+```yaml
+factor_revise_pattern:
+  step_1_draft:
+    action: "Generate complete implementation"
+    output: "draft_code"
+
+  step_2_factor:
+    action: "Decompose into verifiable claims"
+    outputs:
+      - "Function X handles error case Y"
+      - "Loop invariant: Z holds at each iteration"
+      - "API call returns type T"
+      - "Memory is freed in all paths"
+
+  step_3_independent_verify:
+    # CRITICAL: Each runs with ONLY the claim + minimal context
+    # No access to full draft code
+    parallel_tasks:
+      - verify: "Function X handles error case Y"
+        context: "Function signature + error spec"
+        result: "PASS|FAIL + evidence"
+      - verify: "Loop invariant holds"
+        context: "Loop structure only"
+        result: "PASS|FAIL + evidence"
+
+  step_3b_cross_check:
+    # KEY DIFFERENCE: Explicit consistency check before revision
+    action: "Compare original claims against verification results"
+    prompt: "Identify which facts from the draft are CONSISTENT vs INCONSISTENT with verifications"
+    output: "consistency_report"
+
+  step_4_revise:
+    inputs: [draft_code, verification_results, consistency_report]
+    action: "Discard inconsistent facts, use consistent facts to regenerate"
+    output: "verified_code"
+```
+
+### Why Factored Execution Matters
+
+The paper tested 4 execution variants:
+- **Joint**: Questions and answers in one prompt (worst - repeats hallucinations)
+- **2-Step**: Separate prompts for questions vs answers (better)
+- **Factored**: Each question answered separately (recommended)
+- **Factor+Revise**: Factored + explicit cross-check step (best for longform)
+
+Without factoring (naive verification):
+```
+Model: "Here's the code"
+Model: "Let me check my code... looks correct!"  # Confirmation bias
+```
+
+With factored verification:
+```
+Model: "Here's the code"
+Model: "Question: Does function handle nulls?"
+[New context, no code visible]
+Model: "Given a function that takes X, null handling requires..."  # Independent reasoning
+```
+
+**Key principle from the paper:** The verifier cannot see the original response, only the verification question and minimal context. This prevents rationalization of errors and breaks the chain of hallucination propagation.
+
+### CoVe Integration with Blind Review
+
+CoVe operates BEFORE blind review as a self-correction step:
+
+```
+Developer Code --> CoVe (self-verification) --> Blind Review (3 parallel)
+                          |                            |
+                   Catches errors early         Catches remaining
+                   via factored checking        issues independently
+```
+
+**Combined workflow:**
+```yaml
+quality_pipeline:
+  phase_1_cove:
+    # Developer runs CoVe on their own code
+    draft: "Initial implementation"
+    verify: "Self-generated questions, factored execution"
+    revise: "Corrected implementation"
+
+  phase_2_blind_review:
+    # 3 independent reviewers (no access to CoVe results)
+    reviewers:
+      - focus: "correctness"
+      - focus: "security"
+      - focus: "performance"
+    # Reviewers see verified code but don't know what was corrected
+
+  phase_3_aggregate:
+    if: "unanimous approval"
+    then: "Devil's Advocate review"
+```
+
+### Metrics
+
+Track CoVe effectiveness:
+```
+.loki/metrics/cove/
++-- corrections.json     # Issues caught by CoVe before review
++-- false_positives.json # CoVe flags that were actually correct
++-- review_reduction.json # Reviewer findings before/after CoVe adoption
+```
+
+---
+
 ## Velocity-Quality Feedback Loop (CRITICAL)
 
 **Research from arXiv 2511.04427v2 - empirical study of 807 repositories.**
@@ -98,6 +282,105 @@ Task(model="sonnet", description="Code review: performance", prompt="Review for
 
 ---
 
+## Two-Stage Review Protocol
+
+**Source:** Superpowers (obra) - 35K+ stars GitHub project
+
+**CRITICAL: Never mix spec compliance and code quality review. They are separate stages.**
+
+### Why Separate Stages Matter
+
+Mixing stages causes these problems:
+- **"Technically correct but wrong feature"** - Code is clean, well-tested, maintainable, but doesn't implement what the spec requires
+- **Spec drift goes undetected** - Quality reviewers approve beautiful code that solves the wrong problem
+- **False confidence** - "3 reviewers approved" means nothing if none checked spec compliance
+
+### Stage 1: Spec Compliance Review
+
+**Question:** "Does this code implement what the spec requires?"
+
+```
+Review this implementation against the specification.
+
+Specification:
+{paste_spec_or_requirements}
+
+Implementation:
+{paste_code_or_diff}
+
+Check ONLY the following:
+1. Does the code implement ALL required features from the spec?
+2. Does the code implement ONLY what the spec requires (no scope creep)?
+3. Are edge cases from the spec handled?
+4. Do the tests verify spec requirements?
+
+DO NOT review code quality, style, or maintainability.
+Output: PASS/FAIL with specific spec violations listed.
+```
+
+**Stage 1 must PASS before proceeding to Stage 2.**
+
+### Stage 2: Code Quality Review
+
+**Question:** "Is this code well-written, maintainable, secure?"
+
+```
+Review this code for quality. Spec compliance has already been verified.
+
+Code:
+{paste_code_or_diff}
+
+Check the following:
+1. Is the code readable and maintainable?
+2. Are there security vulnerabilities?
+3. Is error handling appropriate?
+4. Are there performance concerns?
+5. Does it follow project conventions?
+
+DO NOT verify spec compliance (already done).
+Output: PASS/FAIL with specific issues listed by severity.
+```
+
+### Implementation in Loki Mode
+
+```yaml
+two_stage_review:
+  stage_1_spec:
+    reviewer_count: 1  # Spec compliance is objective
+    model: "sonnet"
+    must_pass: true
+    blocks: "stage_2"
+
+  stage_2_quality:
+    reviewer_count: 3  # Quality is subjective, use blind review
+    model: "sonnet"
+    must_pass: true
+    follows: "stage_1"
+    anti_sycophancy: true  # Devil's advocate on unanimous
+
+  on_stage_1_fail:
+    action: "Return to implementation, DO NOT proceed to Stage 2"
+    reason: "Quality review of wrong feature wastes resources"
+
+  on_stage_2_fail:
+    action: "Fix quality issues, re-run Stage 2 only"
+    reason: "Spec compliance already verified"
+```
+
+### Common Anti-Pattern
+
+```
+# WRONG - Mixed review
+Task(prompt="Review for correctness, security, performance, and spec compliance...")
+
+# RIGHT - Separate stages
+Task(prompt="Stage 1: Check spec compliance ONLY...")
+# Wait for pass
+Task(prompt="Stage 2: Check code quality ONLY...")
+```
+
+---
+
 ## Severity-Based Blocking
 
 | Severity | Action |